Populating order fact table incrementally,
3 posters
Page 1 of 1
Populating order fact table incrementally,
I have several dimensions (DimProduct, DimCustomer, DimCustomerAddress, DimOrderDate, DimOrderStatus) and a fact table that tracks orders at the order line/product level.
The fact table also has OrderNumber as a degenerate dimension and this along with the FKs from the dimensions makes up the primary key in the fact table. The measures included in the fact table are ProductOriginalPrice, ProductPriceAfterDiscount, ProductQuantity, ProductDiscountPercentage, COGS, Profit, Margin, and ShippingAmount (this comes from the source system on the order level but is divided by quantity amount to bring it down to item/product level).
I have SCD set up on most of my dimensions and that works fine. I also populated the fact table initially and it has almost 50 million records.
My question has to do with loading the fact table incrementally. How do I do this? Btw, this is a transactional fact table.
I don't want to pull all 50 mill records every time I update DW so do I just check if the DD/FKs combination (PK) exists and if not insert the record into the fact table?
What happens if, even though this should not happen, one of the measures changes in the source system - say ProductQuantity? I can't insert a new row into the fact table as it'll have the same PK and as such it won't let me do it. Do I check if any of the measures for any of the records in the fact table has changed and if so I update it? Or do I just ignore it because at the time of the load this is what the source system had? What is the typical way of doing this? Again, this is suppose to be a transactional fact table.
Thanks!
The fact table also has OrderNumber as a degenerate dimension and this along with the FKs from the dimensions makes up the primary key in the fact table. The measures included in the fact table are ProductOriginalPrice, ProductPriceAfterDiscount, ProductQuantity, ProductDiscountPercentage, COGS, Profit, Margin, and ShippingAmount (this comes from the source system on the order level but is divided by quantity amount to bring it down to item/product level).
I have SCD set up on most of my dimensions and that works fine. I also populated the fact table initially and it has almost 50 million records.
My question has to do with loading the fact table incrementally. How do I do this? Btw, this is a transactional fact table.
I don't want to pull all 50 mill records every time I update DW so do I just check if the DD/FKs combination (PK) exists and if not insert the record into the fact table?
What happens if, even though this should not happen, one of the measures changes in the source system - say ProductQuantity? I can't insert a new row into the fact table as it'll have the same PK and as such it won't let me do it. Do I check if any of the measures for any of the records in the fact table has changed and if so I update it? Or do I just ignore it because at the time of the load this is what the source system had? What is the typical way of doing this? Again, this is suppose to be a transactional fact table.
Thanks!
dk2014- Posts : 15
Join date : 2014-11-10
Re: Reply with quote Populating order fact table incrementally,
Hi,
are your business users interested in capturing the changes/updates in the fact table, if not interested then you can update the fact table.
thanks
are your business users interested in capturing the changes/updates in the fact table, if not interested then you can update the fact table.
thanks
hkandpal- Posts : 113
Join date : 2010-08-16
Re: Populating order fact table incrementally,
If you want to keep change history, one way to do it is insert the change amount and get rid of the PK declaration. This would be a transactional fact table (as opposed to a table that stores information about transactions) in that rows are insert only and each row represents a net change.
Alternately, you can build an accumulating snapshot fact which, in a sense, is a lot like a type 2 dimension. You store versions of rows that are restatements of a previous state. Rows would be date bounded. If you declare a PK you would include the effective date in that key.
Alternately, you can build an accumulating snapshot fact which, in a sense, is a lot like a type 2 dimension. You store versions of rows that are restatements of a previous state. Rows would be date bounded. If you declare a PK you would include the effective date in that key.
Re: Populating order fact table incrementally,
Thank you hkandpal and ngalemmo!
dk2014- Posts : 15
Join date : 2014-11-10
Similar topics
» Order dimension vs. order degenerate dimensions in the fact table!?
» Need to merge fact tables
» Combining master and history tables to fact
» combine order, invoice, and backlog detail in one fact table
» Is it possible to get a distinct order count with a transaction line sales fact table?
» Need to merge fact tables
» Combining master and history tables to fact
» combine order, invoice, and backlog detail in one fact table
» Is it possible to get a distinct order count with a transaction line sales fact table?
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum