Kimball Forum
Would you like to react to this message? Create an account in a few clicks or log in to continue.

How to test deployment of ETL jobs (dev-test-production)?

4 posters

Go down

How to test deployment of ETL jobs (dev-test-production)? Empty How to test deployment of ETL jobs (dev-test-production)?

Post  larus Wed Aug 17, 2011 2:23 am

We have been developing a Data Warehouse and bunch ETL jobs for loading data from source systems into the DW. Now we are on the egde of deploying these ETL jobs into production environment. ETL jobs should be heavily tested and I would like to know if you guys could give some hints and guidence how to do this testing? What should be tested (data validation, job function, database performance, indexing)? What kind of reports should be written etc.?

larus

Posts : 5
Join date : 2011-03-01

Back to top Go down

How to test deployment of ETL jobs (dev-test-production)? Empty Re: How to test deployment of ETL jobs (dev-test-production)?

Post  BoxesAndLines Thu Aug 18, 2011 9:45 am

Did you use any documentation to build the ETL logic? Source to target mappings, design documents, etc? These documents are the source of the rules to test your ETL.
BoxesAndLines
BoxesAndLines

Posts : 1212
Join date : 2009-02-03
Location : USA

Back to top Go down

How to test deployment of ETL jobs (dev-test-production)? Empty Re: How to test deployment of ETL jobs (dev-test-production)?

Post  larus Thu Aug 18, 2011 2:18 pm

Well, yes of course, but I think that there is much more than just testing ETL jobs (logic, function). I would be delighted if someone could give some advice what kind of testing should be done at minimum.

larus

Posts : 5
Join date : 2011-03-01

Back to top Go down

How to test deployment of ETL jobs (dev-test-production)? Empty Re: How to test deployment of ETL jobs (dev-test-production)?

Post  BrentGreenwood Mon Jan 02, 2012 12:42 pm

Here are some additional ideas for testing. The list is really endless. Just depends on how much budget/time you are able to spend on setting up the tests and framework.

- Failure / Restart testing (at each logical break point - by landing zone - raw, cleansed, transformed, integrated, staged, final load)
- Unit Testing (isolated logic and function tests - ideally, these would be created in a way that they can be automated and used for regression on later builds)
- Performance testing (hardware)
- Load testing (ensure load process will scale to handle peak loads within processing window)
- Security testing (ensure users don't have access to the kitchen)
- Integration testing (over multiple units and/or entire process - test functions as well as data integrity from source to target)

A good resource in this space is Ralph Hughes' book Agile Date Warehousing. He also teaches classes specifically in Agile DW Testing that are quite informative, and go into great detail on a testing framework he built for a massive DW at a telecom client recently.

Hope that helps.
Brent Greenwood

BrentGreenwood

Posts : 6
Join date : 2011-12-25
Location : San Diego, CA

http://brentgreenwood.blogspot.com

Back to top Go down

How to test deployment of ETL jobs (dev-test-production)? Empty Re: How to test deployment of ETL jobs (dev-test-production)?

Post  burgerboy123 Thu Jan 22, 2015 1:24 am

So... is it the dropping and reloading that is causing me the problem? or should i generate a surrogate key a different way ensuring it gets the same key after any necessary rebuild?



________
adil

burgerboy123

Posts : 1
Join date : 2015-01-22

Back to top Go down

How to test deployment of ETL jobs (dev-test-production)? Empty Re: How to test deployment of ETL jobs (dev-test-production)?

Post  Sponsored content


Sponsored content


Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum