Real time datawarehousing

Hey,

There is a lot of buzz going on regarding real time datawarehousing. But how do you actually do this ? How can you implement conformed dimensions and conformed facts in real time ? How can you consolidate data real time ? Vendors promise applications that can make this happen but how is this possible ? Does anybody do this and if yes, how ?

Best regards,

Rik

With smoke and mirrors.

I think true real-time data warehousing is impossible. Transaction systems are constantly changing - 1 second after you query data it's changed. I think that it's possible to have close to real time DW on a a few key pieces of data as long as the volume is fairly small. You could probably pick up only data that has changed, but I would think it would be as strain on the transactional system.

I've heard of companies that load twice a day.

We had someone that was talking about realtime data - until some told him that are systems run in batch overnight.

You can't even get real time quotes on mutual funds, which is one of the only things that I can think of in which changes in a short time can have a huge impact.

LOL. I always get a chuckle when I read about RDW. Isn't real time just the application database? It would probably be easier to consolidate the front end applications into a unified database than try to implement a real time DW. Fortunately, my users are happy with business plus 1 day.

We have something updated every 3 hours or 2-3 times a day. But it's a small portion of data and there are not a complicated metric calculations.

Everyone throwing "real time data warehousing" terms around - is just "me too" in many cases.
We should really use "near time", unless it IS the OLTP system.

But think about a manufacturing process. Perhaps when the variance on a Widget length goes from 1 inch to 1.1 inch.
This may mean that a machine cutting blade is dulling, and could break and take a production line down [very costly]. If the information were to be reported in near time, it may mean that the machine can be down for a few minutes [planned] and the blade replaced and minimal work stoppage down the line.

Keep this in mind - if you can describe a business process improvement that can be obtained by "near time" DW, then you need it.
Otherwise it really is just "me too".

Dano Smith

Can anyone tell me how to handle loading data to a fact table several times a day when dimensions are only loaded nightly? We run into date logic problems with this. The fact table has a dimension ID that relates to the current record when the fact was loaded. A batch job that run nightly then updates the dimension with new records that are effective dated for the same day. Now my fact table dimensions are tied to the wrong dimension record actually the day before.

clgentr