http://kimballgroup.forumotion.net/etl-and-data-quality-f9/-t1.htm Mon, 16 Nov 2009 05:13:22 GMT 10 http://kimballgroup.com/images/KGlogoBasic.gif http://kimballgroup.forumotion.net/etl-and-data-quality-f9/-t1.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/hand-coded-etl-revisited-t35.htm Nigel Nichols
I read Gary Nissen's article, 'Is Hand-Coded ETL the Way to Go?', with interest.

Given that this was written nearly six years ago, I wonder whether the position has changed i.e. whather ETL tools are now more strongly recommended ober hand-coding.

Nigel]]> ETL and Data Quality Fri, 13 Feb 2009 15:41:10 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/hand-coded-etl-revisited-t35.htm#143 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/hand-coded-etl-revisited-t35.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/business-intelligence-tool-t345.htm Jaswanth How to work on repository ETL and Data Quality Mon, 16 Nov 2009 05:13:22 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/business-intelligence-tool-t345.htm#1404 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/business-intelligence-tool-t345.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-data-without-key-t338.htm hennie7863 For a customer of mine i'm loading messages in a datawarehouse. The messages don't have an id(?!). With this message i want to load some dimensions and the fact. Are there best/Good practices of doing this? Currently i'm thinking of giving these messages a self generated key. Load the data and compare afterwards if the load went ok. So the dimensions are using this key and the fact. I'm not very happy with this solution. So i hope that someone gives me a better solution. O and we're talking ... ETL and Data Quality Thu, 12 Nov 2009 12:56:49 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-data-without-key-t338.htm#1384 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-data-without-key-t338.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/scd-type2-etl-design-t331.htm KK_ETL Hi, We are trying to build a new data warehouse. Planning to capture the data as SCD Type2 in the Data Warehouse. However, the source system doesn't has any date fields for extraction. Let me explain the scenario: Table Name : DWS_CUST Scenario 1 Columns : Customer No (PK) Cust_Name Cust_address1 Cust_address2 Cust_address3 Cust_address4 Start_date End_date Data 1 XYZ 34 ... ETL and Data Quality Thu, 05 Nov 2009 17:02:46 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/scd-type2-etl-design-t331.htm#1369 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/scd-type2-etl-design-t331.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/database-kdb-t322.htm bi_at_nj
If so, do u use informatica with it ?


- Thanks,
bi_at_nj]]> ETL and Data Quality Sat, 31 Oct 2009 06:14:08 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/database-kdb-t322.htm#1337 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/database-kdb-t322.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-informatica-32bit-vs-64bit-t321.htm bi_at_nj
If so what kind of problems are you facing in the 64 bit version.

Is there anything you like the most in the 64bit version?

- Thanks in advance,
bi_at_nj]]> ETL and Data Quality Sat, 31 Oct 2009 04:55:58 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-informatica-32bit-vs-64bit-t321.htm#1336 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-informatica-32bit-vs-64bit-t321.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-table-loads-t320.htm bi_at_nj Here is a scenario: * Fact table contains 100 million records * Monthly Load of 10million is done at the end of the month to the same fact table In this scenario, how do you handle the indexes in fact table at the time of load? If indexes are made unusable before load, then the rebuild index is time consuming after the load. So, what strategy do you follow in such scenarios? How do we ensure that the data is made available to the users in the shortest possible time. (Assume ... ETL and Data Quality Sat, 31 Oct 2009 04:54:03 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-table-loads-t320.htm#1335 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-table-loads-t320.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/how-to-load-a-slowly-changing-dimension-type-2-with-one-sql-merge-statement-in-oracle-t11.htm ubethke
You can access the solution at http://www.business-intelligence-quotient.com/?p=66]]> ETL and Data Quality Tue, 03 Feb 2009 15:26:28 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/how-to-load-a-slowly-changing-dimension-type-2-with-one-sql-merge-statement-in-oracle-t11.htm#16 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/how-to-load-a-slowly-changing-dimension-type-2-with-one-sql-merge-statement-in-oracle-t11.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/source-for-accumulating-snapshot-fact-table-t223.htm kbarrett I have a question about accumulating snapshots and I was hoping someone could shed some light on the subject. Does Kimball give any guidance anywhere (books, online, etc.) as to whether one should build the supporting transactional fact tables that relate to the accumulating snapshot before building the snapshot? We are looking at building a snapshot fact table for the entire insurance policy lifecycle (from initial submission to quote to binding/issuing the policy to first claim (if ... ETL and Data Quality Tue, 21 Jul 2009 16:23:57 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/source-for-accumulating-snapshot-fact-table-t223.htm#957 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/source-for-accumulating-snapshot-fact-table-t223.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/open-source-etl-t115.htm AzeemFarooqui
Our client is keen on using java code to perform ETL. I don't feel this is a viable option and am looking into the option of using an open source ETL tool. Does any one have any useful information on open source ETL and the pros/cons of these against standard ETL tools?

I appreciate your help.

Regards
Azeem]]> ETL and Data Quality Tue, 21 Apr 2009 09:52:04 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/open-source-etl-t115.htm#509 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/open-source-etl-t115.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/open-source-etl-vs-commercial-etl-t217.htm dellsters Anybody have experience with open source etl like Talend or Pentaho? They are becoming more popular, and I was wondering what advantages/ disadvantages of open source vs commercial etl tools. Any downsides of using open source for SOX or HIPAA? ETL and Data Quality Thu, 16 Jul 2009 04:56:32 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/open-source-etl-vs-commercial-etl-t217.htm#931 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/open-source-etl-vs-commercial-etl-t217.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/techniques-for-updating-existing-fact-records-t275.htm johnpaulmurphy I wanted to get peoples opinion on what techniques work best when you have to do updates to Fact rows i.e. overwrite, adjust but keep audit etc...

Thanks.]]> ETL and Data Quality Wed, 16 Sep 2009 18:57:59 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/techniques-for-updating-existing-fact-records-t275.htm#1164 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/techniques-for-updating-existing-fact-records-t275.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/maintaining-reference-tables-t282.htm ajviolette I'm looking for some feedback regarding what tools others are using to maintain data in Data Warehouse specific reference tables. These tables are typically used to provide cross reference or hierarchical definitions for the source data during ETL processing. My previous company developed web pages for maintaining these special purpose tables that were stored in the staging area of the Data Warehouse. My current company has been using Excel spreadsheets to store and maintain ... ETL and Data Quality Thu, 01 Oct 2009 20:32:32 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/maintaining-reference-tables-t282.htm#1211 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/maintaining-reference-tables-t282.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/conforming-dimensions-standardising-de-duplicating-and-suvivorship-t278.htm johnryan Hi, I'm currently reading the DW ETL toolkit, which seems to have some excellent ideas. However, as it doesn't come with any downloads (e.g example SSIS packages) - I'm struggling to understand a few things. If anyone can answer the following it would be most appreciated: 1)2 sources for a dimenion interests me. Am I right in understanding that the issues here are essentially that we could have an attribute (e.g. customer location for the Customer Dim) in both data sources. Therefore, ... ETL and Data Quality Wed, 23 Sep 2009 15:02:19 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/conforming-dimensions-standardising-de-duplicating-and-suvivorship-t278.htm#1186 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/conforming-dimensions-standardising-de-duplicating-and-suvivorship-t278.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/financail-calendarseed-data-t232.htm GBS74 I am tring to write pl/sql procedure to create financial calendar seed date. I have confusion about ... how to code to get financial week /period/start and end date for calendar if financial year and quarter start at first monday of April. any idea or script ?
regards]]> ETL and Data Quality Wed, 29 Jul 2009 21:29:42 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/financail-calendarseed-data-t232.htm#999 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/financail-calendarseed-data-t232.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/business-logic-dwh-vs-source-system-t37.htm inglev Hello everyone, We are currently implementing a Data Warehouse (consolidating data from several source systems) and we had an argument on where the business logic should reside. Simplified example: The source contains the fields “amount”, a “discount” and a “total amount”. The “total amount” is supposed to be the “amount”*(1-“discount”), but for us, the DWH team, it is already available as a readily loadable fixed ... ETL and Data Quality Tue, 17 Feb 2009 08:07:12 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/business-logic-dwh-vs-source-system-t37.htm#154 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/business-logic-dwh-vs-source-system-t37.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/does-it-belong-in-the-stage-tables-or-fact-tables-t270.htm kskistad If I have a fact table with a "counter" fact, for example a customer places an order, but then goes back and changes parts of that order any number of times, I want to store how many times the customer changed his/her order. The grain of the fact table is the orderID. The changes are captured at the source in a change history table, but that table only stores the previous 5 days changes. I see two ways to do this: create another table, a factless fact table, and capture the history ... ETL and Data Quality Thu, 10 Sep 2009 16:37:01 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/does-it-belong-in-the-stage-tables-or-fact-tables-t270.htm#1150 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/does-it-belong-in-the-stage-tables-or-fact-tables-t270.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/transposing-from-columns-to-rows-t268.htm nxlefrancois I have data extracts from transaction system containing multiple measures (columns) all being of the same type of indicator, for which I would like to transform into multiple rows in my fact table. E.g. Extract contains (for each row): date location nbr_visitors_can_bc nbr_visitors_can_on nbr_visitors_can_qc ... (one for each of the 10 Canadian provinces) nbr_visitors_us_ma nbr_visitors_us_wi nbr_visitors_us_ok ... (one for each of the 50 US states) I'd like my FACT_VISITATION ... ETL and Data Quality Wed, 09 Sep 2009 18:25:46 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/transposing-from-columns-to-rows-t268.htm#1145 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/transposing-from-columns-to-rows-t268.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-architecture-and-control-flow-t116.htm monsieur_arrie Hi Folks, I am new to the forums, they look like an interesting place to discuss BI issues and troubles. So, I am not new to BI, but we have done a lot of re-architecting of the etl systems with servers being moved, wan considerations etc. Currently, our source system extracts, compresses and ftps data to our etl area. File names are representative of the 'day' the data refers to. This allows data o be queued in case of ftp failure etc. My etl system is entirely file based. ... ETL and Data Quality Tue, 21 Apr 2009 15:32:52 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-architecture-and-control-flow-t116.htm#510 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-architecture-and-control-flow-t116.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-server-sizing-t266.htm sjain Hi, Could help me to suggest what should be the possible considerations for sizing an ETL Server? Like platform( UNIX, Linux, Windows), Volume of data(in GBs), type of source(relational, application, files), type of transformation/transforms(type of operartion such as SCD) ,Batch - window( Daily, weekly, monthly) sampling rate, commit size How it affect the sizing (directly or indirectly) Or is it depend upon the tool you are using like SAP BO Data Services, Informatica, Datastage Any ... ETL and Data Quality Mon, 07 Sep 2009 08:04:22 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-server-sizing-t266.htm#1141 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-server-sizing-t266.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-server-hardware-configuration-t255.htm Devendra Naik Example Server Type(Intel/Itaninum etc ) , Number of CPU/cores / Memory / Disk configuration and size .
We are planning to upgrade our env. and to start a new project, I would like]]> ETL and Data Quality Fri, 21 Aug 2009 15:12:17 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-server-hardware-configuration-t255.htm#1093 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-server-hardware-configuration-t255.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/incremetal-load-from-1-fact-to-another-fact-t241.htm vibhutidevatraj Hi i have a situation. There is a fact A and fact B which acts as a source for another fact C. The Fact A and B contains detail values which are loaded daily and Fact C contains aggregated values which is loaded weekly. Fact A and B contains Source_file_name, source_file_date information in them but fact C has no file informations. My questions are Is it a good practice to have multiple facts as source for another fact? If yes, then how the data sould be loaded incrementally in ... ETL and Data Quality Tue, 04 Aug 2009 09:13:58 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/incremetal-load-from-1-fact-to-another-fact-t241.htm#1025 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/incremetal-load-from-1-fact-to-another-fact-t241.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/perfect-design-vs-time-to-implement-t237.htm DanColbert
I have a situation where the time I have to launch a business process is shorter than the time I need to get a new dimension designed and pushed. I'm thinking about implementing the process without that new dimension and adding it later.

What are the pitfalls I can expect if I choose this rout?

Thanks in advance!

Dan]]> ETL and Data Quality Fri, 31 Jul 2009 14:41:36 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/perfect-design-vs-time-to-implement-t237.htm#1009 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/perfect-design-vs-time-to-implement-t237.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/using-3rd-party-sort-packages-in-etl-stream-t216.htm juz_b I was wondering if anyone can share your experience with using a 3rd party Sort package (CoSort, SyncSort etc) in your ETL Stream. I have been using Business Objects Data Integrator for the last 5 years and never had a chance to integrate a 3rd party Sort package into the ETL stream. My understanding is that it is always faster to process Flat file (source and target), compared to a database table. My dilemma is that by processing using a flat file, you lose the ability to query in the ... ETL and Data Quality Wed, 15 Jul 2009 21:13:47 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/using-3rd-party-sort-packages-in-etl-stream-t216.htm#926 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/using-3rd-party-sort-packages-in-etl-stream-t216.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/audit-dimension-help-t198.htm mugen_kanosei Hello all. I'm at the stage now where im building the audit dimension for my warehouse. I am having a little trouble figuring out a few things though. How do you aggregate the data from the error event fact table into the audit dimension? Some records will have 5 screens fail, some 3, but they may not all be the same screens. How do you summarize all these into a few unique audit records? The best I have come up with so far is to write a select statement to make audit dimension columns for ... ETL and Data Quality Mon, 29 Jun 2009 23:43:58 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/audit-dimension-help-t198.htm#874 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/audit-dimension-help-t198.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/staging-activities-t196.htm kskistad What are the common staging activities that the Kimball method recognizes? Traditionally I have used staging databases to 1) Decouple from the source-to-Data Mart for restartability 2) Consolidate multiple sources and source formats into a single homogeneous environment 3) Data cleansing, such as populate missing data, scrubbing fields, etc. But some articles I've seen talk about surrogate key handling and conforming within the staging database. Most ETL tools I've used will read ... ETL and Data Quality Mon, 29 Jun 2009 17:30:24 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/staging-activities-t196.htm#868 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/staging-activities-t196.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/large-fact-table-and-maintaining-periodic-snapshot-practice-t182.htm buzzer75 I would like some opinions with my approach here. I am trying to replace an overkill lift and load ETL process that basically replicated the entire universe of dataset every period instead of doing just Delta Load. In this delta approach, I have a stage table with new fact rows and I merge it to the target base table to load delta. I also a current flag to the new and old records. For reporting purposes, the current picture is all we need and I use the flag. For weekly and monthly view if needed, ... ETL and Data Quality Fri, 19 Jun 2009 00:30:25 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/large-fact-table-and-maintaining-periodic-snapshot-practice-t182.htm#830 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/large-fact-table-and-maintaining-periodic-snapshot-practice-t182.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/interval-time-sum-column-t173.htm Enrico Hello, In my data warehouse I have a fact table that joins with calendar table on BETWEEN clause ( calendar.date between myfact.startdate and myfact.enddate ). In calendar date I have a record for each day. I need to sum a fact table column in my query only one time for each fact table records. Now my fact table column is sum for each records of the query. select sum(myfact.value) from calendar, myfact where calendar.date between myfact.startdate and myfact.enddate How ... ETL and Data Quality Mon, 15 Jun 2009 11:10:36 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/interval-time-sum-column-t173.htm#801 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/interval-time-sum-column-t173.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/updating-historic-transactions-and-snapshots-t176.htm cal.sneds Hi, I have a project where we will be building Transactional Fact Tables along with daily Periodic Snapshot Fact tables based on the same data. Now, I'll try to be as brief as possible, but this needs some explaining, this is the dilemma; The task seems straight forward, but the business is able to go back in time and update some of the transactional records and they want to reflect this in the DW, while keeping the old record along with the update as a new record and an audit ... ETL and Data Quality Tue, 16 Jun 2009 10:23:10 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/updating-historic-transactions-and-snapshots-t176.htm#810 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/updating-historic-transactions-and-snapshots-t176.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/enormous-data-size-t143.htm jaiveeru This all started from my query 2 weeks back on solving data modeling problem in recursive hierarchical data table. I posted a query and got replies, almost instantly. All replies pointing to one solution. I did exactly as suggested i.e. I resolved the tree structure into a flat table so that there is no many to many items left. This was done in addition to resolving all one to many data other tables as well. I happily added all redundant fields into table and my fact table size now is ... ETL and Data Quality Thu, 14 May 2009 15:04:30 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/enormous-data-size-t143.htm#622 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/enormous-data-size-t143.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-fact-table-t163.htm rpcasey001 In reading a previous thread, I understand that when a Dimension is changed by a SCD type 2, a new record is created for the new key and no update is needed on the previously existing facts. However, what happens in the loading of the fact table that lets the process know that it has to load a fact again? Does this happen since the combination of keys has changed and it is recognized as a new row? Should a fact table load check for existing rows, if so, how? -- RPC ETL and Data Quality Mon, 01 Jun 2009 16:03:54 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-fact-table-t163.htm#741 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-fact-table-t163.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/stage-table-for-fact-data-t165.htm rpcasey001
--- RPC]]> ETL and Data Quality Tue, 02 Jun 2009 14:20:10 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/stage-table-for-fact-data-t165.htm#755 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/stage-table-for-fact-data-t165.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-fact-table-t38.htm bakunian Hi, I have following tables FACT, A_DIM, B_DIM. How do I update relationship between a_key and b_key in the FACT table when new record arrives in TYPE2 a_dim dimension? Below is simple create scrip to illustrate what I mean. create table fact (a_key integer, b_key integer); create table a_dim (a_key integer, a_id integer, a_string varchar2(20)); create table b_dim (b_key integer, b_id integer, b_string varchar2(20)); insert into a_dim values (1, 100, 'value1'); insert into ... ETL and Data Quality Tue, 17 Feb 2009 20:56:00 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-fact-table-t38.htm#160 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-fact-table-t38.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-data-aggregated-to-date-into-fact-table-t152.htm grahan007 I have following source table: Source Table ID App_Version Os_Version Date 9123305 2.5.2.60 Windows NT 5.1 2/15/2009 9123306 2.5.2.60 Windows NT 5.1 2/15/2009 9123307 2.5.2.60 Windows NT 5.1 2/15/2009 9123308 2.5.2.60 Windows NT 5.1 2/15/2009 9123309 2.5.2.60 Windows NT 6.0 2/15/2009 9123310 2.5.2.60 Windows NT 5.1 2/15/2009 9123311 2.5.2.60 Windows NT 5.1 2/15/2009 9123312 2.5.2.60 ... ETL and Data Quality Tue, 26 May 2009 13:12:06 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-data-aggregated-to-date-into-fact-table-t152.htm#706 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/loading-data-aggregated-to-date-into-fact-table-t152.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/hand-coding-etl-questions-t141.htm mugen_kanosei I know the numerous pros of using an ETL tool, but due to circumstances outside my control I have to hand code the ETL. My questions are in regards to actual coding practices. I am currently loading a couple of dimensions using perl. So far the entire load is in one perl script that is set to run every night. I'm wanting to code something more like what the books suggest, metadata driven, batch scheduling, etc. But im unsure how this is getting handled in general. At first I thought metadata ... ETL and Data Quality Thu, 14 May 2009 00:40:10 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/hand-coding-etl-questions-t141.htm#610 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/hand-coding-etl-questions-t141.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/sql-or-pl-sql-for-hand-coding-etl-t146.htm tropically Wanted to get a general idea, as to what others have used when hand coding ETL for loading data into data marts.
My thoughts : Straight inserts , updates, merges are faster, however can't capture errors. Pl/SQL is more flexible allowing to log errors if any.

Any thoughts would be appreciated.]]> ETL and Data Quality Mon, 18 May 2009 16:20:11 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/sql-or-pl-sql-for-hand-coding-etl-t146.htm#635 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/sql-or-pl-sql-for-hand-coding-etl-t146.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/date-dimension-in-oracle-with-one-sql-statement-t52.htm ubethke CREATE TABLE d_date AS SELECT n AS Date_ID, TO_DATE('31/12/2007','DD/MM/YYYY') + NUMTODSINTERVAL(n,'day') AS Full_Date, TO_CHAR(TO_DATE('31/12/2007','DD/MM/YYYY') + NUMTODSINTERVAL(n,'day'),'DD') AS Days, TO_CHAR(TO_DATE('31/12/2007','DD/MM/YYYY') + NUMTODSINTERVAL(n,'day'),'Mon') AS Month_Short, TO_CHAR(TO_DATE('31/12/2007','DD/MM/YYYY') + NUMTODSINTERVAL(n,'day'),'MM') AS Month_Num, TO_CHAR(TO_DATE('31/12/2007','DD/MM/YYYY') + NUMTODSINTERVAL(n,'day'),'Month') ... ETL and Data Quality Thu, 26 Feb 2009 17:54:07 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/date-dimension-in-oracle-with-one-sql-statement-t52.htm#245 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/date-dimension-in-oracle-with-one-sql-statement-t52.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-begin-and-end-dates-t139.htm tropically deleted ETL and Data Quality Wed, 13 May 2009 22:29:29 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-begin-and-end-dates-t139.htm#607 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-begin-and-end-dates-t139.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-load-dropping-indexes-and-constraints-t60.htm AzeemFarooqui
I am currently working on an ETL solution using BODI and SQL Server 2005. Our data warehouse is very small (no more than 5mb) currently and expected growth over the next year is not going to exceed 15mb.

Based on the above volume estimates does it make sense to drop existing indexes/constraints when performing the ETL load into the fact table?

I'd appreciate other peoples comments and views on this.

Regards
Azeem]]> ETL and Data Quality Tue, 03 Mar 2009 12:32:56 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-load-dropping-indexes-and-constraints-t60.htm#290 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-load-dropping-indexes-and-constraints-t60.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/cost-of-an-etl-tool-t43.htm Rik Declercq We are considering bying an ETL tool to replace our hand-coded scripts. We have no idea about the cost of such a tool. Can somebody give an idea ? ETL and Data Quality Thu, 19 Feb 2009 14:33:10 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/cost-of-an-etl-tool-t43.htm#188 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/cost-of-an-etl-tool-t43.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/test-data-generation-t86.htm Udankar Hi, I am a part of a data warehousing project which has now entered into testing phase. Our client or if I have to generalize every Company is reluctant to share the production data with the software vendors for testing purpose - For security reasons. For testing a data warehouse, we need to use good test data without compromising on data security and privacy concerns. So how to generate almost real test data which will improve the test quality. The test data must give a feeling of ... ETL and Data Quality Sat, 21 Mar 2009 11:12:02 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/test-data-generation-t86.htm#385 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/test-data-generation-t86.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-optimization-scenario-t101.htm Mohsin Currently size of our fact table is in millions which will continue to expand and pretty soon our ETL processes will fail, so need to optimize them now before it gets out of hand.

So we need to create a scenario which will allow was to test our ETL processes.

Can anyone tell me, how can I cook up such a scenario.]]> ETL and Data Quality Thu, 02 Apr 2009 09:36:35 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-optimization-scenario-t101.htm#437 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-optimization-scenario-t101.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-row-dilemma-t105.htm DanColbert I have a FactOrders table. When we have a return, I record it as a new row on the original order, but with negative values in the Quantity and ExtendedPrice columns. There are times when we credit a customer a dollar amount, but no units are returned. I'm not sure how to record that in the fact row. If I record the dollar amount, but no units - any calculated "average price" measure will break because of a divide by zero. If I record a (-1) for the quantity, then I overstate ... ETL and Data Quality Mon, 06 Apr 2009 14:04:14 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-row-dilemma-t105.htm#458 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/fact-row-dilemma-t105.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-tool-choice-t85.htm Rik Declercq We are currently investigating if we want to replace our hand-coded ETL-system by an ETL tool. If we go for a tool we of course have to choose a tool. Is the following [url=etltool]http://www.etltool.com/[/url] worth the money ? ETL and Data Quality Fri, 20 Mar 2009 10:04:02 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-tool-choice-t85.htm#384 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/etl-tool-choice-t85.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/oracle-warehouse-builder-what-does-the-future-hold-t56.htm robber I have a client implementing an analytic solution which uses OWB for ETL. I have concerns over OWB's future given the Sunopsis acquisition which they now call ODI (I think). Part of the appeal of OWB is it's low cost, I'm assuming that advantage will diminish as Oracle pushes ODI. Anyone else using OWB and concerned about it's future? ETL and Data Quality Sat, 28 Feb 2009 19:58:04 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/oracle-warehouse-builder-what-does-the-future-hold-t56.htm#259 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/oracle-warehouse-builder-what-does-the-future-hold-t56.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/sap-netweaver-and-bocd-enterprise-server-t58.htm DilMustafa I saw an awesome thread on the Future of Oracle Datawarehouse Builder and Oracle Data Integrator. I will appreciate people to comment on SAP BI technologies as well. SAP bought BO, and BO came with an ETL,DQ toolset. A toolset well respected in the market. Now what is SAP going to do with existing ETL suite they had under SAP BI/BW thing. Also please, expand this thread to comment on SAP BI reporting and analytics as well. I mean BO and SAP have competitive products in all areas of BI here. Your ... ETL and Data Quality Sun, 01 Mar 2009 19:48:45 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/sap-netweaver-and-bocd-enterprise-server-t58.htm#272 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/sap-netweaver-and-bocd-enterprise-server-t58.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/how-long-should-1-dummy-records-exist-in-fact-tables-t29.htm Scoop Dear Kimball Group, we are using the Kimball BUS Architecture method with SQL Server 2005, SSIS, SSAS, ProClarity, PPS, and MOSS. I am at a clients site that is using Full Refresh process for building 3 data marts so far every night. We are trying to get incremental going using replication and ODS but their parent company is not allowing us to go forward with replication on production due to them thinking this a huge strain and IO issue on the production OLTP servers. We are working on this ... ETL and Data Quality Tue, 10 Feb 2009 14:57:09 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/how-long-should-1-dummy-records-exist-in-fact-tables-t29.htm#99 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/how-long-should-1-dummy-records-exist-in-fact-tables-t29.htm http://kimballgroup.forumotion.net/etl-and-data-quality-f9/random-musings-group-by-t41.htm Edwin Kurian Hoping to get your thoughts.

Edwin]]> ETL and Data Quality Wed, 18 Feb 2009 15:22:20 GMT http://kimballgroup.forumotion.net/etl-and-data-quality-f9/random-musings-group-by-t41.htm#171 http://kimballgroup.forumotion.net/etl-and-data-quality-f9/random-musings-group-by-t41.htm