Perfect Integration 13 - the do's

Final post in the series, this is the summary and conclusion, to be used as some sort of checklist if you like

When conducting enterprise business application integration, within the enterprise IT landscape among applications and systems, or from there to others at another company or even directed towards the customer, here are the pragmatic rules:

Start at the business level, and write down which process it concerns. What is the business event, which its trigger, how often does it occur, how sensitive is the data? Why should this be automated, in other words, what is the manual alternative and the benefits and concerns of both?
Write out the functionality it concerns. List the entities, their cardinalities, and all attributes concerned, and describe them as complete as possible. Here's an example

There are about 20,000 employees, each one of which has 1 or 2 addresses, and working on 1 to 5 assignments. The employee data has attributes, of which the functional description is shown.
At this level, anyone can read and understand what is to be exchanged: business people, tech people, inside people, outside people, suppliers, customers: everyone

Depending on the business goal, some of these will be required, and some won't: specify that as well. When creating a new employee, First name, last name and personal ID will be required, but not an employee number: that will be handed out by the system. When modifying an employee, that employee number will be required, including the last name, but perhaps not the personal ID.
So, for the intended business goal, write down which attributes are required and when

Then, with this Information list in hand, make the transition to the systems on both sides: do those requirements meet and match on both sides? If so, you can do business, if not, you have an issue. This indeed means that business agreement, functional analysis and information system (gap) analysis is part of the preliminary phase and could result in the conclusion that business simply can't be done

If successful, both sides will add their information system requirements (these are actually limitations) to the Information: Last name will be 30 characters in one system, and 25 in the other - what to do? Is the business agreement still in place or do we have to revise it?
Apart from the messaging level, there will now be limitations on the transportation level as well: what are the existing possibilities to get messages across? Data sensitivity as defined by the business will lead to transport security and encryption and access restriction requirements, and these will meet limitations: again, the common subset has to be found.
Again, is the business agreement still in place or do we have to revise it?

On both sides there will now be a full technical description of the interface, complete with field length, datatype, format. The business frequency and the given cardinalities will give a rough guesstimation of the resulting file size, thus the volume and load, making it possible to calculate storage requirements and capacity planning. Files will have to be kept online for a certain amount of time, and archived as well, depending on compliancy requirements the business has to follow up - all that costs space as well.
For the transformation into an intermediate language, this will have to be repeated of course. Here the IT department will mostly determine how long these have to be available. If used as basis for a full-blown BPM, the underlying processes will drive these requirements too, although files should only be used for resending (on failure) and "evidence" of what was received and sent


At this point, the issue arises: what file format and transportation protocol should we use?


Which ever ones are most practical and pragmatic, will be the answer. For A2A (application-to-application, a TLA used to describe internal integration) the usual pick will be flat-file for the message and shared folders for transport. A flat-file can be a CSV, a fixed flat-file, JSON, whichever makes you feel good. Maybe it's an XML, an iDoc: all that is up to the applications themselves. When you use an intermediate language and a canonical model, you will have to choose a file format yourself.
If volumes and frequencies are high, a more sturdy transport protocol might be required; thing of queueing mechanisms. XML will definitely pose capacity and storage problems there, ask Google, Twitter and Facebook and companies in the traditional B2B area

If one of the A2A interfaces "makes it out there", i.e. is a candidate for exchanging information with external parties and you engage in B2B or B2C, the question of file format and transportation protocol will arise once again - just in a different context. What is most appropriate then?
Again, do in Rome as the Romans do. Speak X12 in US industries, EDIFACT in Europe, Swift in banking, HL7 in health, etcetera. Transport? FTP or FTPS for static solutions, HTTP for more dynamic ones and AS1 or AS2 for the "bigger boys". Treat each of these questions as a simple daily one: should I pick UPS, global or local mail or courier to transport my package? That of course depends on what the package is, and how fast it should get somewhere

Last but absolutely not least, you should define an envelope for each interface: who is the sender, who is the recipient, what does it contain? This will enable you to route its contents freely across the globe, just like SMTP (email) does, without opening the envelopes and looking at the letters. Don't address envelopes to URLs or other silly notations, you're doing business with an organisational entity, not a machine: let the organisation make up where exactly they want to route their stuff internally, but keep that transparent for the outside world

In an effort to keep this post short, I'll just mention that business integrity requirements will lead to business rules and exceptions, which can be translated into error handling and (user) feedback on a functional and technical level. These will lead to functional and technical acknowledgments on the interface level, retry mechanisms when a file can't be sent out, and exception handling and agreements for that.
To really define the interface, the byte-level has to be spelled out as well: character code (ASCII or UTF-8, EBCDIC or UCS-2, any other flavours), hexadecimal form of delimiters, periods or commas, etcetera: the real dirty stuff

This concludes my series on Perfect Integration. I hope I touched all possible questions and answers, and helped unravel the mystery of buzzword Three Letter Acronyms. I think I have reached a satisfactory level of detail without losing your attention - but you'll be the judge of that