Kimball Forum
Would you like to react to this message? Create an account in a few clicks or log in to continue.

ETL Execution & Scheduling

3 posters

Go down

ETL Execution & Scheduling Empty ETL Execution & Scheduling

Post  ETLDev Wed Feb 03, 2016 9:12 am

I had a question about the ETL design. Given that ETL would involve tools/scripts/programs written in different programming languages and platforms, what platforms/tools are used to manage the execution and scheduling of these? How do we make sure that these tools/scripts/programs run at the same time and pull data into the data warehouse?

Thanks,
Amit

ETLDev

Posts : 1
Join date : 2016-02-03

Back to top Go down

ETL Execution & Scheduling Empty Re: ETL Execution & Scheduling

Post  nick_white Wed Feb 03, 2016 10:41 am

I would use a dedicated ETL tool (Informatica, SSIS, Talend, Datastage, ODI, etc.) otherwise, as you say, you have to hand code not only the actual data transformations but also the control framework to make sure everything runs with the correct precedents, errors are handled appropriately, etc.
Except for the simplest of processes I think you would have to be a bit of a masochist these days (or very confident in you abilities) to hand code your ETL system

nick_white

Posts : 364
Join date : 2014-01-06
Location : London

Back to top Go down

ETL Execution & Scheduling Empty Re: ETL Execution & Scheduling

Post  ngalemmo Wed Feb 03, 2016 2:18 pm

Some etl tools have built in scheduling and execution capabilities, but generally, they are pretty limited in functionality. Most sites use an enterprise scheduling tool and execute etl processes using scripts calling the etl tool's CLI (command line interface). The CLI will usually return a status code that the scheduling tool would use to check progress of the load.
ngalemmo
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

http://aginity.com

Back to top Go down

ETL Execution & Scheduling Empty Re: ETL Execution & Scheduling

Post  Sponsored content


Sponsored content


Back to top Go down

Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum