Miscellaneous Fields
3 posters
Page 1 of 1
Miscellaneous Fields
I have a fairly large account dimension ~ 6.5 million rows and 35 columns. It is a type1 SCD.
In the source table there are Miscellaneous fields (1-14) that can contain just about any piece of information varchar(50). I don't have any control over these fields or what goes in them.
I need to start pulling these misc fields and keeping a history on them (type2).
I am not sure what the best approach would be.
Any help is appreciated.
In the source table there are Miscellaneous fields (1-14) that can contain just about any piece of information varchar(50). I don't have any control over these fields or what goes in them.
I need to start pulling these misc fields and keeping a history on them (type2).
I am not sure what the best approach would be.
Any help is appreciated.
jayz- Posts : 3
Join date : 2014-09-16
Re: Miscellaneous Fields
Putting fields that can contain anything about anything into a data warehouse is a disaster waiting to happen. Once you have them loaded, how do you expect anyone to use them? To do any useful query they would need to place a context around the value (what does it represent, what do the different values mean…), meaning the end user will need to do some (if not a lot of) research to figure out what they are looking at.
For that reason, it makes more sense to do the work once, in the beginning, to figure out what the data is before you load it.
As far as storing the data goes, if this is truly a changing set of attributes, you may want to consider a factless fact table. Dimensions such as account, attribute type, attribute value, would provide the necessary context and allow the data to be integrated with the account. But such a table can be tricky to use. If the context turns out to be a lot more stable, placing them as properly named and documented attributes in the dimension is also an option.
For that reason, it makes more sense to do the work once, in the beginning, to figure out what the data is before you load it.
As far as storing the data goes, if this is truly a changing set of attributes, you may want to consider a factless fact table. Dimensions such as account, attribute type, attribute value, would provide the necessary context and allow the data to be integrated with the account. But such a table can be tricky to use. If the context turns out to be a lot more stable, placing them as properly named and documented attributes in the dimension is also an option.
Re: Miscellaneous Fields
Thanks for the response, I really appreciate your time.
To clarify the misc fields usage a little. We have about 100 clients that use this data. The Misc fields are for each of these clients to use, so while I say the data can be anything each client standardizes and uses each misc field for their own purpose.
So for example client 1 may have client number in misc 9 field where client 2 may have it in misc 1 field.
Client 1 may have campaign information in misc 1 where client 2 has no campaign information at all.
My point is that they will know what is in each misc field (and most of them aren't used).
To clarify the misc fields usage a little. We have about 100 clients that use this data. The Misc fields are for each of these clients to use, so while I say the data can be anything each client standardizes and uses each misc field for their own purpose.
So for example client 1 may have client number in misc 9 field where client 2 may have it in misc 1 field.
Client 1 may have campaign information in misc 1 where client 2 has no campaign information at all.
My point is that they will know what is in each misc field (and most of them aren't used).
jayz- Posts : 3
Join date : 2014-09-16
Re: Miscellaneous Fields
I would probably just load the data as-is into the dimension, then have some customer based metadata the drives the query interface.
Re: Miscellaneous Fields
I'm concerned that having the misc fields (type 2) in the Account Dimension (type 1) will make the account dimension too large. I was thinking of putting each misc field in its own dimension, but that creates so many keys. Is there a different option I should consider?
jayz- Posts : 3
Join date : 2014-09-16
Re: Miscellaneous Fields
Use a junk dimension.
BoxesAndLines- Posts : 1212
Join date : 2009-02-03
Location : USA
Similar topics
» Dimension fields depend on other fields
» Naming of ID and code fields
» Y/N fields in dimension or as measure with 0 and 1 (or both)?
» Sharing fields among dimensions
» extra custom fields
» Naming of ID and code fields
» Y/N fields in dimension or as measure with 0 and 1 (or both)?
» Sharing fields among dimensions
» extra custom fields
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum