- #PENTAHO DATA INTEGRATION IN PENTAHO CDWE HOW TO#
- #PENTAHO DATA INTEGRATION IN PENTAHO CDWE UPDATE#
- #PENTAHO DATA INTEGRATION IN PENTAHO CDWE CODE#
I would have thought that there would be a higher percentage of new users, at least from the forums.
Or maybe I’m just not aware of these resources.Īgain, it is interesting to me that most of the people (over 75%) who responded to my survey have over two years of PDI experience. It was interesting to me that so many folks were developers, but that the resources for testing PDI (and version control and other kinds of developer type tasks) are so sparse. The ‘other’ category ranged from “I do everything” to “Data Warehouse Architect”. So these results are of mostly anecdotal interest. I received almost 20 responses, but I know that this is only a tiny portion of the PDI userbase (after all, in the last week, kettle 4.4.0 has been downloaded over 110,000 times since it was released last November).
#PENTAHO DATA INTEGRATION IN PENTAHO CDWE HOW TO#
I marketed this on the pentaho forums and twitter (got a nice retweet from I wasn’t sure how to reach more of the PDI audience. I just wanted to post results from the PDI/Kettle user survey I ran a few weeks ago. Here’s a list of open source ETL tools to evaluate. When you are confronted with a data movement problem, take a look at an ETL tool like Kettle and see if you can stand on the shoulders of giants.
#PENTAHO DATA INTEGRATION IN PENTAHO CDWE CODE#
Every line of code you write is another you have to maintain. I did.ĭon’t re-invent the wheel! Your data movement problem may very well be a super special snowflake, but chances are it isn’t. You have to roll your own testing framework.Some components (at least for Kettle) are not open source.Limits of components mean you sometimes have to contort your data flows, or drop down to write your own component.Possible to version control, impossible to merge.You can write your own components that leverage existing libraries.There is support for other performance features like parallel jobs that you can configure.You operate at a higher level of abstraction.It’s hard to over emphasize how much time this can save, allowing you to focus on business logic. There are pre-built components for common data tasks (connecting to a database, parsing a flat file) that have been tested and debugged by many many people.This is more true now than in the past because there is a lot more data flowing everywhere, and there are several viable open source ETL tools, so you don’t have to spend thousands or tens of thousands of dollars to get started. While ETL systems are not without their flaws, I think their strengths are such that everyone who is moving data around should consider them.
#PENTAHO DATA INTEGRATION IN PENTAHO CDWE UPDATE#
When I was working for 8z, we used it heavily to pull data from other systems, process it, and update our databases. The one with which I am most familiar is Kettle, aka Pentaho Data Integration.
0 kommentar(er)
