Implementation and Timelines

Hello DW Gurus -

I've been tasked with coming up with a total WAG on how much it will take in time, dollars, and humans to create a data warehouse environment. Like most organizations, we have data in a variety of source systems. We are a municipal utility in a medium sized city. Unfortunately, my timeline for getting something to the corner offices is early next week. There is a vision for a data driven decision making, but little recognition of the complexity of the task they've asked for. If I ask management things like 'relational model or nosql' I get blank stares.

I'm a guy that knows how to code, understands enough about databases to be dangerous and can craft pretty nice solutions with BI tools against existing relational structures, but as far as implementing a large scale data warehouse, I just don't have that experience set. Yet, the deadline for some kind of answer remains; I won't have my feet held to the fire per se, but I'd like to give an answer that is horseshoes close.

Any insight is *greatly appreciated*.

Thanks in advance.
brian

If data driven decision making is the goal, then your time should be spent trying to estimate the effort in designing, building, and populating good dimensional models. Hadoop is a great way to prototype your data's ability to drive decisions, but not a long term production level solution.

That being said, I too am with a utility with a data driven vision. I would SWAG 3-5 years for a complete system, but built using the business process dimensional modeling lifecycle ideas set forth by the Kimball Group. You could, with a properly built team and easy enough first project, kick something in a 3 months. Of course, the value from the first BP modeled will be minimal. But it will set you up for the next one, and next one, and pretty soon, those join together to allow for some pretty good analytics.

It's really hard to give dollar estimates, as they depend on the toolsets and platforms chosen. I'd stick to whatever the standard database you currently use, and open source allows for minimal to no costs to ETL functionality. And don't let them push you into developing the Meter Data Management System as the first project. We've been doing ours for 6 months now with nothing spectacular to show because without the supporting dimensional data in the many other systems, their isn't much value to it.

Hi TheNJDevil -

Thanks so much for the rapid response! And to get someone who is in the utility business as well seems incredibly lucky. I was evaluating Kimball training and their toolkit books. Will move forward with them.

And don't let them push you into developing the Meter Data Management System as the first project.

LOL. Don't they have to be working the way the white paper said they would before we bother to analyze their data? Oops, did I type that?

Well data warehouses aren't really projects, they're more like groups. Once you decide to build one, you're always building, updating, enhancing. It never ends. I'd pick the common roles needed, and say we need to hire X people. If you want it a little faster, you can supplement the FTE staff with some contractors. Typical roles would be a PM, data analyst, data modeler, ETL, QA, DBA, Report developers.