SAP, Integration and Star Trek: the future is now

I commented ranted on an SDN post yesterday. Submitting it failed, and I lost the +/- 500 words. A bit more miffed after that, I wrote the comment anew in Notepad, and copy/pasted that - it worked.
I got a few reactions, some of which inviting me to post on the topic on SDN via a blog post in stead of just a (lengthy) comment. While I appreciate the invite, I'll just do it here for now

Perfect Integration - the eBook

Perfect Integration

by Martijn Linssen

What started with Perfect Integration 1 - Architectural Approach and ended with Perfect Integration 13 - the do's has become a lot of words, more than 10,000 actually

Perfect Integration 13 - the do's

Final post in the series, this is the summary and conclusion, to be used as some sort of checklist if you like

When conducting enterprise business application integration, within the enterprise IT landscape among applications and systems, or from there to others at another company or even directed towards the customer, here are the pragmatic rules:

Start at the business level, and write down which process it concerns. What is the business event, which its trigger, how often does it occur, how sensitive is the data? Why should this be automated, in other words, what is the manual alternative and the benefits and concerns of both?
Write out the functionality it concerns. List the entities, their cardinalities, and all attributes concerned, and describe them as complete as possible. Here's an example

Perfect Integration 12 - the dont's

I changed my mind and decided to end this series with positive do's, so this is the dont's one. Then again reserving no. 13 for the dont's was a superstitious move anyway, and as I'm neither religious nor superstitious (they usually travel in pairs), it's better this way

This post is about debunking TLA's and FLA's. XML, SOAP and REST primarily, but Webservices and other concepts whose added value is flimsy or even absent, will get proper attention

Perfect Integration 11 - Orchestration

I've compared the diversity of an IT application landscape and managing its information exchange in a uniform way to translation, with the European Parliament as a perfect example of translating dozens of languages via three intermediate languages. In IT, we only need one, as languages (syntaxes) there are far less complex than in the linguistic world

I've used a similar metaphor to explain how different transport protocols can be handled, by referring to James Bond's opening scenes, where he takes all kinds of transportation in order to escape death. He simply manages by getting off one kind of transportation, and getting on another one. Clever hey?

Perfect Integration 10 - the missing link: envelope

With a common language, a common transport protocol, and the need to exercise the necessary translation and transformation on both levels in between, there is a growing need to be able to identify all "service requests" on a generic level too

Numerous and various requests will be made, in different formats, via different transport protocols. This certainly is not Utopia, but a pragmatic observation that is maintained in almost an evolutionary way: within successful companies IT systems, like people, are selected based on professional excellence (meaning specialist instead of generic properties).
Excelling in one area usually means being moderate in a few others, and being capable of good and generic integration is usually one of those. Intellectual supermodels, and good-looking scientists: they are either scarce or non-existent

When all those requests are being made in different ways too, it becomes nearly impossible, but at least very time-consuming and costly, to collect and streamline all the information. Whatever the flexibility on the messaging and transportation level may be, addressing all those variations must be made possible in a generic and structured way

Perfect Integration 9 - history with hindsight

In the previous post, the history of Integration passed: point-to-point, EAI and ESB. For those who read and grasped post 1 through 7, it'll be clear why I favour which one - but let me explain it in more detail

What are the differences between the different historical approaches?

The crucial difference is that EAI describes a hub and spoke architecture, where proprietary messages to and from applications are translated by a central integration broker. Yes, that is the European Parliament post, among others.

As described, applications can plug in and out of the landscape this way, and the necessary transformation of information is taken care of by the central hub

Perfect Integration 8 - history of the last decades

In the first seven posts, the approaches to Integration have been shown. The architectural top-down approach, the common subset theme, and the central integration methods: messaging, transformation and transportation.
Now the approach to successful integration has been set out, a brief story about integration in the real IT world, over the last decades, is in its place

Point to point interfacing
Point to point connections, is a term that describes how applications can be connected. Point to point interfacing was commonly used to link applications together "in the early days", and was a reasonable solution as long as the scale of integration stayed small

Perfect Integration 7 - information exchange: transportation

[Image courtesy of Ferdinand Reus]

After creating and or choosing a common or generic format to exchange the information, there is one other field to explore: the facilitation of various communication protocols through which this information can be transported

What applies to messages, also applies to transport: a common language is to be advised as "main artery" for all the traffic. Besides that, each application must speak its own language here as well, and use its own infrastructure as much as possible. Adjusting applications for transport protocols can be even harder than doing so for messages: transport protocols are very, very static and rely on worldwide standards. Making your own version of it is highly unappreciated

Perfect Integration 6 - Common language: syntaxes

In the previous posts I explained semantics, syntax, and the fastest, cheapest and easiest way to get from diverse IT applications to one uniform business language. This post will take a deep dive into message formats such as Flat file, EDIFACT, XML and JSON

Ever wondered about the pros and cons of XML? JSON? What it is really? What other possible message syntaxes are?
When semantics and a logical structure has been defined, and functional groups and fields have been identified, just about any message format can be chosen.
Choosing message formats (or rather, predefining them) is a rather pragmatic measure that narrows down the margin for discussion when it comes to choosing a form for your common language

Perfect Integration 5 - Common language: indirect translation

Number 5 in the series, this post is about indirect translation, in contrast with the direct translation shown in the previous post, which came with costly, exponential dependencies

When looking towards large-scale use of translators, e.g. the European Committee in Brussels, it is easily observed how these dependencies can be greatly reduced: all languages are first translated into English, French or German, after which either one of those is translated into the final destination language

Perfect Integration 4 - Common language: direct translation

Update March 22 19:07 CET: extended this paragraph along the lines sketched in my comment below. Thanks Stuart

This post in the series is about my favourite subject: translation. Long, long, long ago I aspired to become an interpreter - but changed my mind. Still, I consider myself a linguist and a good one at that, speaking 6 languages of which 3 fluent. The existence of extreme diversity in languages of the world, and the same existence in diversity in an average enterprise IT landscape have always hooked my attention

Just like people speak different dialects and languages, so do applications: not one of them can perfectly understand the other without changing a thing. Even when interfacing between SAP systems, there will be differences on the technical level that have to be resolved.
So, how do you solve these misunderstandings at the technical level? How do you "translate" between the different languages and dialects spoken? One of the sides has to adapt, or maybe both? When, and how?
To answer that, let's look at the everyday world of natural languages. A completely different world, with an identical problem - and a working solution

Learning a language can be very hard and time-consuming, and very few will ever master a non-native language perfectly. Most of us know someone within the same country that has an accent and just can’t seem to get rid of it; those that have connections across the border might have heard other people speak their language and have difficulties understanding them. Or one has heard a colleague or neighbour speak a foreign language and, while not being able to speak that language fluently themselves, has good reason to suspect that there will be people on the other side having difficulties understanding that “version of the language”

Perfect Integration 3 - Common language: semantics first

This post will elaborate on messaging and transformation (part I), and explain how information exchange works in the daily world, considering simple or complex information exchanges. That will then be related to IT, and the basic ways of “writing down” information in IT will be explained.

If you want to have a chat with someone else, what do you do?
First, a common theme to discuss has to be found - usually not that hard. The favourite ones in e.g. IT usually are new technologies, cars, and the occasional sports.
The topics then have to be found, which is a bit harder. Do we discuss hardware technologies, or software technologies, or is it architecture or open source we want to discuss?
If the topics have been identified, last but not least the definitions have to be checked: are we talking about the same subjects?

The Greek philosopher Socrates filled entire conversations, and books, with merely trying to establish that he and his counterpart were talking about the same thing. There are numerous occasions on which he talked half a dozen times or more to someone, only to mutually agree on the fact that they didn’t mean the same thing while using the same word, and then end the discussion.

Perfect Integration 2 - Common subset and transformation

Number two in the series, this post deals with the common subset found on all levels in the previous post:

• what is the shared interest (Business)
• which information do you want to share (Information)
• which definitions are mutually exchangeable (Information Systems)
• how do you want to exchange ideas (Infrastructure)

Information exchange form: messaging
After having agreed on mutual business and information to be exchanged, transferring information in a certain message layout from and to systems is the logical next step.

Perfect Integration 1 - Architectural Approach

First post in a series of 5-10, I will release all my views and opinions on the Art of Integration. I challenge you to disagree, and bash me with arguments and reasoning. Feel free to shoot from the hip and aim at the heart, anything goes really. I am absolutely convinced that I am right and spot-on on every single statement, sentence, and word

Itching already? Great. Let the games begin...

Standardisation, Alas, poor standardisation!

Well, Shakespeare may be dead but not his play on transience

What are standards? Nothing else but what is accepted by "majority vote". Being polite is a good standard, but we all know there are exceptions to that rule - to say the least. In fact, standards are dynamic. They change from time to time, adapting to current times, knowledge gained, and knowledge lost

The Roman and Greek gods were standard a few thousand years ago, but, last time I checked, they're just a thing of the past now
Bryan Larkin has a good blog on how standards can deteriorate