Best Practice for ETL Incremental File loading
2 posters
Page 1 of 1
Best Practice for ETL Incremental File loading
Hi - Newbie to the forum, so please excuse any indiscretions …..
I am looking for any advice for a best practice guide for loading incremental files to STAGE Data Warehouse layer.
Essentially looking for the mechanism -
e.g.
Source files arrive on server - what filename - use of a sequence ?
How do we know that files have arrived fully ?
What if files don't arrive.
What happens after a file is loaded ?
How can we rollback - what if a file is missed completely?
Many thanks
chime101
I am looking for any advice for a best practice guide for loading incremental files to STAGE Data Warehouse layer.
Essentially looking for the mechanism -
e.g.
Source files arrive on server - what filename - use of a sequence ?
How do we know that files have arrived fully ?
What if files don't arrive.
What happens after a file is loaded ?
How can we rollback - what if a file is missed completely?
Many thanks
chime101
Last edited by chime101 on Mon Apr 14, 2014 11:18 am; edited 1 time in total (Reason for editing : spelling :-()
chime101- Posts : 1
Join date : 2014-04-14
Re: Best Practice for ETL Incremental File loading
The specific answers to your questions are going to be completely dependent on your environment and your requirements e.g. most of what you have asked could be achieved through scripting and/or an ETL tool - but without knowing what environment you have these questions couldn't be answered.
Also, questions such as "what do we do if a file doesn't arrive" depend on your business and the answer could be anything ranging from "do nothing, it doesn't matter" to "stop everything, alert everyone as our business is about to collapse" - there is no generic answer.
Also, effectively what you are asking is how to manage an ETL process. To cover this would take many chapters in a book on ETL (Kimball covers it in a number of his books) so you are unlikely to get an answer in a forum like this - you are more likely to get an answer if you ask specific questions that can be answered in a couple of paragraphs.
Also, questions such as "what do we do if a file doesn't arrive" depend on your business and the answer could be anything ranging from "do nothing, it doesn't matter" to "stop everything, alert everyone as our business is about to collapse" - there is no generic answer.
Also, effectively what you are asking is how to manage an ETL process. To cover this would take many chapters in a book on ETL (Kimball covers it in a number of his books) so you are unlikely to get an answer in a forum like this - you are more likely to get an answer if you ask specific questions that can be answered in a couple of paragraphs.
nick_white- Posts : 364
Join date : 2014-01-06
Location : London
Similar topics
» ETL Architecture and Control Flow
» Raw File Destination for errors?
» File content in dimension
» Multiple source file with different measures
» Flat File Importing via SSIS - Best approach?
» Raw File Destination for errors?
» File content in dimension
» Multiple source file with different measures
» Flat File Importing via SSIS - Best approach?
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum