Enterprise Integration interview by Richard Seroter

I got the chance to participate in Richard's Interview Series - I was number 40 and you might know what that number means to me
Richard is a principal architect and Microsoft MVP, and well-versed in integration on a whole, especially Microsoft / BizTalk. He's been blogging since 2007, authored / contributed to three books and written an extensive series on BizTalk

You can read the interview on Richard's site, or right here:

Q: You've been writing a series of provocative articles that take a bit of a contrarian view of REST as a viable enterprise (integration) mechanism. You seem pretty sceptical that REST/JSON is a practical service strategy for most enterprises. Given that an earlier post of yours also expresses doubt that XML/SOAP/WSDL is the answer, What types of services SHOULD enterprises be embracing and investing in so that they have a maintainable and usable ecosystem?

A: Tools and techniques aren't the answer to the Integration issue, and certainly not one single tool and technique. But first you'd have to know what the Integration issue actually is, before trying to formulate an answer to it

The Integration issue is that in IT there's an evolutionary, ever-changing diversity in platforms, operating systems, programming languages, applications - and now also devices and locations. Will there ever be a one-size-fits-all for even any of those? No.
I compare this diversity to human languages: they are extremely diverse, and then you have dialects and accents, and those also evolve, and the persons that speak them also get better or sometimes even worse at speaking them

So, we have to tackle that diversity - we can do that in two ways

1) We can make everyone speak the same language, e.g. English.
What's the ROI of that? It takes years, and the majority of people will never get fluent at any language. A huge investment in time and money, and what is the result?
Take American English, English English, Dutch English, but especially German English, French English and (my favourite) Indian English: very hard to understand.
What's the spin-off of that, the result? Well nothing really, given the bare fact that people speak the same language: you need to understand each other. Does you and your partner speaking the same language prevent arguments, misunderstandings? No.

You first need to find a common ground in the actual topics you want to discuss. You ask me a question, I give you an answer, and / or vice versa: we hold entire conversations by firing off requests and responses. I myself usually switch languages when I speak to e.g. Germans; when it gets hard, I switch back from German to English which is neither my native tongue but still a lot more often used than German.
Does that change the conversation? No - it just serves me better. For me there's no difference between speaking English or Dutch, but for a lot of people it would be a whole lot easier to speak just their native tongue

Take this back to Enterprise IT: you bought, built or made all those applications exactly because they play their role so very well. Each of them are Olympic athletes, perfectly apt to do what you want them to do, specialised in one thing only, well maybe 1.5. Now spend the time and money to teach them a different language - ouch! that will cost you dearly, and probably give you Frenglish or Indienglish at best.
[On a side-note, I am not making any statement about nationality or race here, I am just taking an example everyone can relate to. To me, all people are equal regardless of their physical attributes]
Now, let's see how this can be handled in a professional, business-efficient way: the European Parliament. With currently 23 languages in the EP, there are 506 (23 x 22) possible combinations of spoken languages. 750 members serve for 5 years, which means that on average 12.5 people per month get replaced

How much time and money would it cost to teach each of those e.g. English? Could that even be worthwhile? Of course not, and it would seriously hamper the content of messages sent and received across. So, they don't make all these people speak one and the same language, because the diversity and dynamics are so great, that it is simply not an option.
Remember that these 12.5 people per month getting replaced represents 1.5% of total: could you handle 1.5% of your IT landscape being replaced every month?

2) We can hire interpreters. People specialised in translating languages on the fly in mid-air, face-to-face, real-time. That exactly is what happens at the European Parliament.
Now, we run into another problem: you'd need at least 506 interpreters to handle all the diversity (= variations in language combinations). This is commonly known as the N2 (N to the power of 2) problem where (back to IT!) N2 possible combinations arise for N applications / languages.
The solution to that? Still using one common language, but this time it's used by the translators / interpreters to translate any language into, and from. The result? One fluid, fluent common language hanging in mid-air above all the awesome diversity of all languages spoken. The effort for the participants? Null, zilch. Nada. Niente. Niks. Nichts. Rien
[On a side note, the EP uses three middle languages: English, French and German. That's linguistically but also politically determined]

So, I believe in one common language so that the business is not bothered with the evolutionary IT diversity - after all, that diversity is not a goal, nor even a means; it's an unwanted side-effect that will never go away and has to be dealt with.
Do I think the business should be burdened with that diversity? Absolutely not.
Do I think the participants in the Enterprise conversations should be burdened with it? Most certainly not either


Back to your question, the answer to which will now be easy to understand. Did SOAP solve the Integration issue? No. XML? No. WSDL? No. Will REST? No. Will JSON? No. All those imposed, and all these will impose, the Integration issue onto the participants in the conversation, and the Business.
But let's turn that around: where do I see good application for either? In some places, mainly B2C. Not in A2A, and certainly not in B2B. If your customers or service consumers demand any of the above, or if you can profitably maintain or extend market share by translating from your common business language into those, and back again, please be my guest - you'd be a fool if you wouldn't.
But hold a knife to everyone's throat and force them to change their existing SOAP/XML/WSDL to REST/JSON? Good luck with that

Why do you think Google, Twitter and Facebook never used SOAP? It's too undefined a standard, even after more than a decade - and no one asks for it. I've witnessed its use and implementation in Enterprises, and it only resulted in long, heated debates about whose perception of it was right, ending up in yet another bilateral agreement that didn't result in any interoperability whatsoever.
Why do you think they booted or even refrained from using XML? It's too bloated of a syntax, doesn't add anything but overhead. I've witnessed the use and implementation of it in Enterprises, and it only resulted in long, heated debates about whose perception of it was right, ending up in yet another bilateral agreement that didn't result in any interoperability whatsoever. (sic)
Why do Twitter and Facebook now support JSON? Easy, it dramatically decreases overhead compared to XML. You'll notice that the implementation of JavaScript Object Notation has come to be extremely loosely coupled from Javascript (pun intended) and that it is only used as a flat-file syntax for exchanging information regardless of platform, operating system, etc etc etc. To no surprise, as it's ye good old fashioned CSV with a twist

So, what type of services should Enterprises embrace? Simply extending their existing back-office functionality outside the Enterprise is all.
In what form? Whichever form is best suited. Speak Chinese in China, Greek in Greece, and certainly not vice versa.
The location (= bandwidth) impacts the form because the services need to be exposed and thus transported from the back-end to somewhere else on this earth, and vice versa: the further away from the office and civilised world you get, the smaller the bandwidth.
Fit impacts the form, because most programming languages and platforms have a predefined taste, and even ready-built building blocks or components. The older the platforms and programming languages, the more old-fashioned that taste is and the higher the chance that building blocks are present, and fixed. The older the platforms and programming languages, the smaller the variety as well as the chance that building blocks are present: old will tell you: "Listen we only support format XYZ" whereas new will ask you "Well what do you have to choose from and we'll just pick one" - this is presuming that old is on the supply side, and new on the demand side

It all is a question of supply and demand. If you have ample of supply but little demand, you'll be inclined to adopt your consumers' format and transport protocols. If vice versa, you'll wave your existing format(s) across the consumers' faces and say "my way or the highway". It is as simple as that

Q: What are some the positive trends you see in enterprise integration? What are integrators doing now that they weren't doing 5 or 10 years ago?

A: Well, if my answer to the previous question was long, this one might be even longer - but it ain't. To be concise: we have to travel back to the previous century to answer this.
Back in the 80's Integration was confined to database point-to-point connections. All was batch, mostly focused at database replication when there weren't any tools for that, and the database market was still very diverse and far from mature / settled.
A decade later (I'm being very rough with regards to timelines here), Enterprise Integration moved up the stack and targeted applications itself, directly addressing the business logic layer. It was at that point that the canonical model was invented because diversity dramatically increased

In fact, the invention of the canonical model was the solution to the Integration issue

Yes it added overhead because messages had to be translated more than once, but with the batch schedule and low-frequency near-time Integration back then it was heaven on earth. It also enabled BIM and BAM although those two acronyms never made it out into the world because of the fact that the Integration filed got extremely disrupted by Web.
Then, 10 years plus a few years ago, B2C entered the arena, along with Web. Client-server happened along, and along with all that was the cheapification (some poetic freedom here) of servers and clients. Microsoft invaded the Enterprise and pushed aside the costly main- and midframes. Along with that, VB and Javascript put themselves on the stage

The result? Anyone who was handy could sit next to the business and script them through their solution - it was the point where we as an IT industry went from the old ways to the new ways. The old ways? 80% of code was meant to prevent the system from doing what it was not supposed to do. The new ways? 80% of code was directed at having the system do what it was supposed to do.
Anyone with even a faint memory can tell you that this resulted in unintelligible error messages and program dumps - yet that was beyond the scope of the initial key user.
The effects for Enterprise Integration? It put the profession back for a decade and more, reintroducing siloed point-to-point integrations

And here we are now. Over the last decade, we've tried ESB and SOA, focusing on XML and WSDL to make those happen, forcing all consumers to speak that one single language. And it failed, as I have been saying since last century that it would. W3C has become an authority, Oasis has, and countless others try to become yet another purely technical institution that is sponsored by vendors. It resulted in "standards" that are compromised to death: the standards support what their constituents support.
Will REST make up for that? Absolutely not, it is as undefined a "standard" as SOAP was, and will be. 5 Years from now a new tech discovery (no, not invention) will see the light or some old paradigm will get hijacked the way REST currently is, and the world will try to force it onto Enterprise Integration in exactly the same way. Will I stand at the front lines then? Yes, just like now

So, what are the positive trends I see? Well, not much really. I really like how XSLT enables vendor-independent XML-based mappings, yet every vendor has their own implementation of it, so there goes that win. The vendors have to uphold their lock-in and they do it very well, alas.
Yet I see some positive spin-off from SOAP with companies thinking about an envelope to accompany their messages - they're getting closer to the proven concept of old-fashioned snail mail for routing information exchange.
Gateways are still there, functioning as good old post offices, whether they are VANs or not. It depends on industry really, the financial world has remained almost untouched by the craze of the last decade (they can't afford experimenting) as have most if not all logistic and retail platforms. It is governments and semi-governments (e.g. insurance companies) that still hold the deep pockets of Mickey Mouse money with which they can finance early adoption of a tech solution to a business issue (with the likely outcome) - although that will be changed in the future too, given the current crisis

What are integrators doing now that they weren't doing 5 or 10 years ago? They just try to offer New Blacks as much as they can, regardless of their business value. Integration has become a predominantly tech-ruled field, and I despise that.
System integrators are still partnering with vendors and get a cut of the pie for every vendor product they sell to the customer. On the other hand, there are new kids on the block like tibbr, who handle Integration from a customer-friendly and even neutral perspective.
Apart from that, there are Social Integration tools flooding the world, all of them lightweight and inside-out focused, providing their customers with a few basic Integrations. All these will have to learn the hard way that there is no Integration but any-to-any, and who ever learns that quickest and best will lead that pack. But it will be 2-5 years.
A positive side-effect is that Integration has been put onto the agenda of the Social world - I can't complain about that nor would I want to

Q: What, if any, new challenges arise from integrating off-premises/SaaS applications with on-premises systems? Have you seen what decisions makes these scenarios successful, and unsuccessful?

A: Ah. Now that deserves a really long answer (just kidding). Off-premise poses exciting problems to real-time Integration - bandwidth is the new bottle-neck. Regarding successful or not scenarios, there is no choice really. Salesforce.com does a very nice job integrating real-time and batch, limiting each of those with regards to message size depending on what you pay for. So pay-per-Integration is the new mind boggling topic for Enterprises, and speaking of which, yes JSON in stead of XML will absolutely make a difference there - I bet some sweet money on compressing data before it gets interchanged, and back again, at least for the batch variant

The big question of on-premise versus off-premise is out of the question for Integration there, as a fun side-effect: whether you Cloud your Integration solution or keep it on-premise has become irrelevant from a single CIO decision-point, as performance latency is a given now. Having your own Integration solution and hauling in off-premise data or information versus hosting it in the Cloud (right next to your SaaS) is becoming a very interesting decision matrix, highly dependent on what you SaaS where.
The speed of light doesn't help much either, although any request-response still remains sub-second in theory. A round-trip request-reply over 20,000 km will take at least 0.3 seconds, and I predict that Cloud will follow the same pattern that physical distribution of logistics warehouses whave: some centralised, some decentralised.
I expect SSD to be a best solution for making up the increased latency as Integration is all about I/O, as it always has been. Of course it won't overcome the physical barriers of speed, and if it does, let's excavate Einstein please - he wouldn't want to miss that.

The real issue, however, will be that SaaS will just tell you "hey, here's my integration syntax and transport protocol, happy now?" and eliminate the option of customising-to-death, and lest not forget, the practice of pure ESB: forcing all applications to speak the language of the Bus, reducing the Bus to an architect's wet dream that doesn't add any value whatsoever to the Business.
Of course you will be offered a choice between one or two, maybe even three, but that's it. Cloud will greatly drive standardisation, it's even one of my blog post titles I believe

New challenges in a nut shell then, wrapping this one up? Changing the supply-demand paradigm for most Enterprises into demand-supply. I really would like to see how e.g. SAP handles that, but I'm not putting any money on it any time soon. Off-premise SaaS (that's a pleonasm but hey) will confront all Integration participants with the simple fact I described above: the Integration issue is that there's an evolutionary, ever-changing diversity in the IT components that make up or affect your landscape, and the only solution to that is adapt, not adopt

Q: [stupid question]: I don't think I use more than 20% of the features of any single software product. Microsoft Office? Maybe 15%. Sparx Enterprise Architect? 10%, at best. Microsoft Visual Studio? Probably 2%. What software do you use every day, but rarely stray beyond a core set of capabilities? What software do you think you take the MOST advantage of?

A: Not a stupid question really, it's the package paradigm: you pay for 100% and never use more than 10-20%. Then you have to put up with 100% of upgrades and pay even more for functionality you don't use in terms of time and effort.
I use Notepad for the full 100%, primarily to cut and paste between applications, even if those are Microsoft Word and Microsoft Word. I use that, and PowerPoint for fancy forms / images - my world is limited to content and fancy images really.
I use plenty of programming languages to do whatever I need to do, if that gets complicated I prefer using Ultra Edit over Visual Studio. Why? Because I don't like being confronted with change. I prefer growth over change

Thank you Richard for this interview, and keep it up!