Datawarehouse Load Best Practices
3 posters
Page 1 of 1
Datawarehouse Load Best Practices
In other Datawarehousing projects that I have worked on, I have always presumed that we load all data that we receive from source systems as long as the data model and supports it. However, on my current project, they want to restrict what data we load based on reporting requirements. So in my case, we ask certain users what project codes they want to see on monthly reports and only load those project codes and related data.
My thought is to load all the data and then limit the users selection to only their project codes in the report. This way, if the user mistakenly gave us the wrong codes, or changed their minds, the effort would quite less then trying to go back and reload all the data. We do not have a storage issue, so that is not the reasoning. Are there any "Best Practices" that you can refer me to in order to make the decision? I welcome your feedback as well. thanks
My thought is to load all the data and then limit the users selection to only their project codes in the report. This way, if the user mistakenly gave us the wrong codes, or changed their minds, the effort would quite less then trying to go back and reload all the data. We do not have a storage issue, so that is not the reasoning. Are there any "Best Practices" that you can refer me to in order to make the decision? I welcome your feedback as well. thanks
mr_neal- Posts : 17
Join date : 2012-01-26
Re: Datawarehouse Load Best Practices
Your instincts are correct - load all the data, then gracefully filter in the BI/Reporting layer.
Ideally there can be an attribute associated with the project code that can be used as a filter, allowing for a changing project code list (because reporting requirements ALWAYS change) with no ETL or BI changes.
Ideally there can be an attribute associated with the project code that can be used as a filter, allowing for a changing project code list (because reporting requirements ALWAYS change) with no ETL or BI changes.
LAndrews- Posts : 132
Join date : 2010-05-13
Location : British Columbia, Canada
Re: Datawarehouse Load Best Practices
Some concepts are so elementary that to consider this a best practice is silly. I mean I interview folks all the time and ask about ETL best practices and I would never expect to hear this. As such, I don't believe you will find any papers or blogs that identify what your current project is doing is bad practice.
BoxesAndLines- Posts : 1212
Join date : 2009-02-03
Location : USA
Similar topics
» Sharing best practices
» What are some Best Practices to ETL from Excel to SQL Server 2008?
» Fact Design and best practices
» Daily Snapshots Best Practices
» What are some Best Practices to ETL from Excel to SQL Server 2008?
» Fact Design and best practices
» Daily Snapshots Best Practices
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum