Kimball Forum
Would you like to react to this message? Create an account in a few clicks or log in to continue.

Overnight Load Failure Frequency - Benchmark?

4 posters

Go down

Overnight Load Failure Frequency - Benchmark? Empty Overnight Load Failure Frequency - Benchmark?

Post  djschiff Fri Oct 23, 2015 2:44 pm

I'm wondering if there are any benchmarks as to the percentage of time that the overnight load process for a data warehouse is successful (on-time, untouched by developers or support staff) vs late or encountering a problem requiring some sort of intervention.  Obviously there's no "one size fits all" answer; there are many factors including the number and types of source systems, data volume, data type, system/data maturity, etc.  But are there rough benchmarks, such as problems once a month is "normal" vs. "above/below avg," etc.

djschiff

Posts : 2
Join date : 2010-01-18

Back to top Go down

Overnight Load Failure Frequency - Benchmark? Empty Re: Overnight Load Failure Frequency - Benchmark?

Post  ngalemmo Fri Oct 23, 2015 3:46 pm

If you are dealing with a well defined source with decent data management, such as an ERP system, load failures (those not caused by external issues such as hardware or network failure) should be near zero. Even uncontrolled sources should have sufficient processes in place to separate bad data from good to prevent a full failure.

In other words, if you regularly encounter load issues, your process is broken.
ngalemmo
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

http://aginity.com

Back to top Go down

Overnight Load Failure Frequency - Benchmark? Empty Re: Overnight Load Failure Frequency - Benchmark?

Post  djschiff Sat Oct 24, 2015 9:27 am

Well "external issues," I guess that's the rub, isn't it? A network issue here, a database problem or server glitch there. Suddenly there are three mornings that appear only as "data warehouse load issues" to the users, but there's little the data warehouse staff can do.

djschiff

Posts : 2
Join date : 2010-01-18

Back to top Go down

Overnight Load Failure Frequency - Benchmark? Empty Re: Overnight Load Failure Frequency - Benchmark?

Post  ngalemmo Sat Oct 24, 2015 2:54 pm

That's the nature of the beast. There are a lot of moving parts and all IT has to do their bit to keep things running. The best you can do is grin an bear it, and keep documentation.
ngalemmo
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

http://aginity.com

Back to top Go down

Overnight Load Failure Frequency - Benchmark? Empty Re: Overnight Load Failure Frequency - Benchmark?

Post  nick_white Mon Oct 26, 2015 3:11 pm

If another system has caused the problem then don't report them as "data warehouse load issues"

nick_white

Posts : 364
Join date : 2014-01-06
Location : London

Back to top Go down

Overnight Load Failure Frequency - Benchmark? Empty Re: Overnight Load Failure Frequency - Benchmark?

Post  ron.dunn Tue Oct 27, 2015 3:17 am

Nick, you must never have been a DBA (Default Blame Acceptor), otherwise you'd know that any system problem is automatically the fault of the biggest database in the chain, until incontrovertibly proven otherwise by the over-worked, under-valued database team

ron.dunn

Posts : 55
Join date : 2015-01-06
Location : Australia

http://ajilius.com

Back to top Go down

Overnight Load Failure Frequency - Benchmark? Empty Re: Overnight Load Failure Frequency - Benchmark?

Post  Sponsored content


Sponsored content


Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum