Kimball Forum
Would you like to react to this message? Create an account in a few clicks or log in to continue.

Best Practice for ETL Incremental File loading

2 posters

Go down

Best Practice for ETL Incremental File loading Empty Best Practice for ETL Incremental File loading

Post  chime101 Mon Apr 14, 2014 11:09 am

Hi - Newbie to the forum, so please excuse any indiscretions …..

I am looking for any advice for a best practice guide for loading incremental files to STAGE Data Warehouse layer.

Essentially looking for the mechanism -

Source files arrive on server -  what filename - use of a sequence ?
How do we know that files have arrived fully ?
What if files don't arrive.
What happens after a file is loaded ?
How can we rollback - what if a file is missed completely?

Many thanks

Last edited by chime101 on Mon Apr 14, 2014 11:18 am; edited 1 time in total (Reason for editing : spelling :-()


Posts : 1
Join date : 2014-04-14

Back to top Go down

Best Practice for ETL Incremental File loading Empty Re: Best Practice for ETL Incremental File loading

Post  nick_white Wed Apr 23, 2014 2:51 am

The specific answers to your questions are going to be completely dependent on your environment and your requirements e.g. most of what you have asked could be achieved through scripting and/or an ETL tool - but without knowing what environment you have these questions couldn't be answered.
Also, questions such as "what do we do if a file doesn't arrive" depend on your business and the answer could be anything ranging from "do nothing, it doesn't matter" to "stop everything, alert everyone as our business is about to collapse" - there is no generic answer.

Also, effectively what you are asking is how to manage an ETL process. To cover this would take many chapters in a book on ETL (Kimball covers it in a number of his books) so you are unlikely to get an answer in a forum like this - you are more likely to get an answer if you ask specific questions that can be answered in a couple of paragraphs.


Posts : 364
Join date : 2014-01-06
Location : London

Back to top Go down

Back to top

- Similar topics

Permissions in this forum:
You cannot reply to topics in this forum