Kimball Forum
Would you like to react to this message? Create an account in a few clicks or log in to continue.

design for data load failures

2 posters

Go down

design for data load failures  Empty design for data load failures

Post  Ishwar Tue May 31, 2011 10:00 am

how to design for data load failures? I have seen a system where if (days) data load has issues, ETL stops all days data to go into the destination tables. Any reading material for the same will help. I am more concerned about database model. I have following questions,

1) With which data I need to check the transformed data?

2) database model for example

3) Stop few data or the entire data load?


Ishwar

Posts : 6
Join date : 2011-05-31

Back to top Go down

design for data load failures  Empty Re: design for data load failures

Post  ngalemmo Tue May 31, 2011 11:46 am

This is something particular to an installation. It all depends on the business and what the user expects.

With that said, I have never implemented a process that, when doing database to database load processes, would ever reject data. The only situations would be when loading from flat files, and it was determined the file was corrupt in some manner. In which case the entire file is rejected.
ngalemmo
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

http://aginity.com

Back to top Go down

design for data load failures  Empty Re: design for data load failures

Post  Ishwar Tue May 31, 2011 2:56 pm

In my current work (Retail data warehouse development), after studying underlying data sources I have picked up tables and columns to work on. Data validation will decide whether I have picked up the correct fields and tables from data sources.

What if some of setting in underlying data source changes, (for example addition of new functionality or addition of new promotion which I didn't consider while designing data warehouse)?
At which step this data should be checked?

What if user identifies issue in data? I need to check and reload the data for the data source?

Any thoughts!!!!

Ishwar

Posts : 6
Join date : 2011-05-31

Back to top Go down

design for data load failures  Empty Re: design for data load failures

Post  ngalemmo Tue May 31, 2011 4:18 pm

If there is a material change in the structure of the source system (new tables, columns etc...) then effecting such changes is a matter of changing the DW model and load processes. That is part of the system lifecycle and would hardly be considered a 'load error'.

If simply adding a new promotion (or any new values to an existing structure) causes problems, then you need to reconsider how you are loading things in the first place. Any load process should gracefully handle new data values... such as adding a new row to the promotion dimension if a new promotion code is encountered.
ngalemmo
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

http://aginity.com

Back to top Go down

design for data load failures  Empty Re: design for data load failures

Post  Ishwar Tue May 31, 2011 4:28 pm

I agree, but does that mean after implementing data warehouse there will not be any data validation?

Ishwar

Posts : 6
Join date : 2011-05-31

Back to top Go down

design for data load failures  Empty Re: design for data load failures

Post  Sponsored content


Sponsored content


Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum