Wednesday, July 30, 2008 4:08 PM
If you consider that moving data around via messages is still ETL, then I want to talk a bit about why I think message based ETL or ETL:M is better than batch based ETL or ETL:B (batch based is typified by tools like SSIS).
I believe that one of the biggest reasons we have ETL:B is because Applications don't have a way to talk to each other. So the solution in the past has been to extract some logic from the source and destination applications and bury it in a ETL:B package that runs at regular intervals. Here I see two problems. One the regular interval could cause some very interesting consistency issues, two the extraction, duplication, and obsfucation of business logic in a tool that was most likely not built to provide the same level of explanatory power that well written code can.
ETL:B is like trying to transfer knowledge by brain transplant rather than the ETL:M approach which I believe is more akin to a conversation between two interested parties.
As we move to adopt a more ETL:M approach I think we will find that integrating our application becomes not only an easier process, but a more responsive process as well.