Data Integration and ETL is Shifting to the End-User

As highlighted in recent news from Data and Application Integration provider Adeptia, end-user independence will be a large factor in the space through 2014. As integration solutions continue to incorporate the cloud into their services, end-users will be interested in integrating the data they need for their specific business processes without the assistance or reliance of internal IT departments.

In a recent programmable web post Adeptia refers to this as the “democratization of integration” saying that it is being driven by the very nature of today’s business. Simply, this shift of ownership and management of data boils down to time. Today’s business is moving faster than ever. Situation and questions need to be addressed in such a timely fashion that if business managers need to wait on results from IT, their opportunity may have already passed.

Follow the link to read the full article. http://solutions-review.com/data-integration/data-integration-is-shifting-to-the-end-user-according-to-adeptia/

So, instead of experienced analysts figuring out the complexities of integrating data correctly, we will now have a multitude of independent end user groups doing whatever.  It sounds like an efficient use of resources to me… certainly allowing a business to move very quickly in some direction with questionable information.

Yep, my days are numbered.

mattAda wrote:As highlighted in recent news from Data and Application Integration provider Adeptia, end-user independence will be a large factor in the space through 2014. As integration solutions continue to incorporate the cloud into their services, end-users will  be interested in integrating the data they need for their specific business processes without the assistance or reliance of internal IT departments.

In a recent programmable web post Adeptia refers to this as the “democratization of integration” saying that it is being driven by the very nature of today’s business. Simply, this shift of ownership and management of data boils down to time. Today’s business is moving faster than ever. Situation and questions need to be addressed in such a timely fashion that if business managers need to wait on results from IT, their opportunity may have already passed.

Follow the link to read the full article. http://solutions-review.com/data-integration/data-integration-is-shifting-to-the-end-user-according-to-adeptia/

Matt,
not going to happen. Ever. The whole "justification" for moving things out of IT is that doing things inside IT is too slow. The development of ETL is one of the slowest and most prone to error issues in all of data warehousing. This is why I have built tools in this area for nearly 20 years now. My first ETL tool was developed in 1995. Ever since then I have had a 2x advantage in ETL development costs. I did Manulife in Hong Kong in 1997 where our total consulting budget was less than the license cost of Prism from Bills company.

In 2002 I was bitching and complaining to Ralph about how Informatica was forcing me to change my data models and how the ETL tools were still slowing us down. Ralph said to me "if you are so smart, write me a paper on what the ETL vendors should do and I will publish it." So I took up Ralphs challenge. Alas I was really busy and I did not get the paper finished before Ralph moved on from DBMS magazine. But a few months later I had some time off and I picked up the paper and thought to myself "you know, if I could build a tool with all these features I would have the best ETL tool on the planet". So in late 2002 I decided to try and build such a tool.

I was successful in that I figured out how to build a "mappingless" ETL tool in C++. The first release went out the door in 2003 and I got a few buyers from my old days in Australia. But not much more. I was also working on Saudi Telecom so we used the tool there and I introduced memory mapped IO so that it could handle the 20M customers and 60M CDRs per day. We built the prototype ETL with it and then migrated that to DataStage. We cut more than 50% off the previous best effort for an ETL system on DataStage of that size. I was PSM for Ardent in AP so I know the timeframes for building with DataStage.

The next project I did was Orange in Romania. This is where we invented the idea of autogeneration of compenents out of the mapping workbook. Usin this idea we were able to cut another 50% off the Saudi Telecom development effort. That meant we were 75% down on what we had done just three years earlier.

This tool has further developed. When I did Carphone Warehouse in the UK on Netezza we decided to test generating SQL as ETL. I was convinced this would not work...I was staggered when it did. We could attribute 80M CDRs in 20 minutes. A process that was taking 4 hours on Informatica. That was 2009. Since then we have piled the development effort in to generating ETL in SQL.

I did another client in 2010. A telco. On day1 we demoed the tool in the morning and built ETL for three dimension tables and a fact table in the afternoon. The next morning the project manager asked "why only 4 mappings on the first day?".

We can now manage the entire DW development and docmentation process via excel workbooks. A unique idea that is going to revolutionise BI as much as star schemas did. Many people here already know I was the guy who handed out Ralph first book in Australia. I was known as "Mr. Star Schema" in Australia LONG before it was fashionable. I was trained at Metaphor and owe a great deal to Ralph and his colleagues.

With what we have done ETL development is so fast that there is no excuse not to do it properly and for anyone but IT to do it. The days of "us end users want to do ETL because IT are too slow" are over.

You are welcome. This is one of my two big contributions to the BI world. With this tool, SeETL, anyone can afford to build a data warehouse.

Anyone who wants to take a glimpse of the future of ETL development? You can get it on this link. 51MB.

http://www.instantbi.com/LinkClick.aspx?fileticket=cPsrkXN-POs%3d&tabid=59&mid=443

I've been following the development of the Power Query Excel Add-In in this space with some interest. The proposition of a free Add-In for Excel, developed by Microsoft is quite compelling and it has proven quite well balanced between functionality and ease-of-use.

As a consultant/contractor I look at it as an ice-breaker. I can use it to quickly build a crude "datamart" to show the potential value of analysing or integrating some new data source, often blending it with existing DW dimensions and facts.

I'm very clear throughout that this is an interim solution that wont scale or be particularly robust or dependable. If it is successful, I have another piece of work to build it with a "serious" ETL tool. At that point you already have a working prototype producing expected results, so risk is greatly reduced. These are often pieces of work that would never get up if someone had to build a business case for a full ETL solution at the outset.

So my take is, these tools are an opportunity, not a threat.

Mike