What is Integration?
Over the years, organisations have invested heavily in IT and, consequently, accumulated large portfolios of IT systems comprising of hundreds, possibly even thousands, of separate IT applications. A single organization might use separate systems, developed either in-house or licensed from a third party vendor, to manage their supply chain, customer relationships, employee information, and business logic. This modularization is often desirable. In theory, breaking the task of running a business into multiple smaller functionalities allows for easy implementation of the best and newest technological advancements in each area, and quick adaptation to changing business needs.
The appropriateness of the integration architecture largely depends upon the business requirements for integration, and on what can be achieved with the available technology. Each kind of integration architecture has its own capabilities and limitations.
Batch Process: The simplest type of integration architecture, IT applications communicates asynchronously with each other and there is no business requirement for real-time processing. Batch integration architecture is often suitable for organisations that perform high-volume back-end processing that is able to take place overnight, weekly, or on a scheduled basis.
Point-to-Point Integration: In a point-to-point integration model, a unique connector component is implemented for each pair of applications or systems that must communicate. This connector handles all data transformation, integration, and any other messaging related services that must take place between only the specific pair of components it is designed to integrate.
Broker Model: In a broker approach to EAI, a central integration engine, called the broker, resides in the middle of the network, and provides all message transformation, routing, and any other inter-application functionality. All communication between applications must flow through the hub, allowing the hub to maintain data concurrency for the entire network.
SOA: The key principle of SOA is to design and build enterprise IT architecture around services rather than complete applications. In an architecture like this, the idea is to create components called services — small, discrete units of software that provide a specific functionality and, importantly, are reusable in every application. An example of an application created with SOA principles might be a bank loan application; it would be composed of credit status check services, interest rate services, and customer data services.
ESB: A different way to approach the problems SOA was meant to solve is with a standalone enterprise service bus (ESB) or integration platform as a service (iPaaS) instead of a full proprietary stack. ESBs can power the creation and orchestration of services without requiring an application server or other infrastructure component, eliminating the high upfront costs of implementing SOA. This enables developers to build reusable interfaces with APIs while also establishing a core framework for integrating a governance model for the future.
Integration Platform and API management: Some organizations remain unconvinced of the need for an application-programming interface (API) management solution. This is especially true of those that already use an integration platform as a service (iPaaS) to integrate their applications and data. Nevertheless, the fact is API management and iPaaS solutions perform unique functions that are equally critical to succeeding in today’s digital economy. Modern organizations need both.
The Interplay of APIs and an iPaaS
An iPaaS lets you exchange data between different systems. It enables data to flow between cloud applications, data warehouses, IoT devices, data lakes, and other systems in your technology stack.API creation and API management are two different things. If you don’t manage the APIs you create, you run the risk of data breaches, chronic system malfunctions, and operational inefficiencies — the origins of which can be hard to pinpoint. What’s more, you fail to capitalize on the revenue potential of your APIs.
If you want to turn the APIs you create with your iPaaS into a profit-generating digital ecosystem, then you must manage your API.
Today’s technology and business professionals have several options when it comes to integrating their application data to other applications.
IBM, Oracle, TIBCO, WebMethods (Software AG) and Microsoft BizTalk are examples of enterprise integration solutions that have extensive background and experience in the integration market with their ability to scale performance and functionality in complex, heterogeneous environments.
Mulesoft, Snaplogic, Boomi and Jitterbit are examples of newer companies that have offered their products in the last 5 to 8 years. Their premise is to leverage newer technologies and frameworks to offer simpler yet rich-enough and robust-enough products that are best suited for quicker and smaller integration projects.
Pros & Cons of heavy platforms
Core features of ESB:
- Location Transparency: A way of centrally configuring endpoints for messages, so that a consumer application does not require information about a message producer in order to receive messages
- Transformation: The ability of the ESB to convert messages into a format that is usable by the consumer application.
- Protocol Conversion: Similar to the transformation requirement, the ESB must be able to accept messages sent in all major protocols, and convert them to the format required by the end consumer.
- Routing: The ability to determine the appropriate end consumer or consumers based on both pre-configured rules and dynamically created requests.
- Monitoring / Administration: As such, an ESB must provide an easy method of monitoring the performance of the system, the flow of messages through the ESB architecture, and a simple means of managing the system in order to deliver its proposed value to an infrastructure.
- Security: ESB security involves two main components - making sure, the ESB itself handles messages in a fully secure manner, and negotiating between the security assurance systems used by each of the systems that will be integrated.
Some of the challenges of a centralized ESB pattern are:
- Deploying changes could potentially destabilize other unrelated interfaces running on the centralized ESB.
- Servers containing many integrations has to be kept running and patched live wherever possible. Topologies for high availability and disaster recovery are complex and expensive.
- For stability, servers typically run many versions behind the current release of software reducing productivity.
- The integration specialist teams often do not know much about the applications they are trying to integrate with.
- Pooling of specialist integration skilled people resulted in more waterfall style engagement with application teams.
- Service discovery is immature so documentation becomes quickly outdated.
- Expensive ongoing support and maintenance contracts.
New era of Lighter integration
We could break up the enterprise-wide centralized ESB component into smaller, more manageable, dedicated components. Perhaps even down to one integration runtime for each interface we expose. This makes each individual integration easier to change independently, and improves agility, scaling, and resilience.
Agile integration architecture requires that the integration topology be deployed very differently.
- Installing a modern integration runtime takes a handful of minutes at most. Indeed, if using a pre-built Docker image, it could take no more than the few seconds it takes to start up a Docker container.
- Administering a cluster of integration runtimes no longer requires proprietary knowledge, because they would be running in a standard container orchestration platform such as Kubernetes or Docker Swarm.
- Scaling up and down, and setting up resilience policies, routing policies, and secure communication channels would be done in a standardized way, the same as all of your other application components.
- Command-line interfaces and template scripts are provided to enable standard continuous delivery tools that can be used to automate build pipelines.
- Integration tooling has improved to the point where most interfaces can be built by configuration alone, often by individuals with no integration background. With the addition of templates for common integration patterns, integration best practices are burned into the tooling, further simplifying the tasks.
- You are no longer restricted to one language or runtime, which means you can have a polyglot runtime-a collection of different runtimes, each suited to different purposes.
The main three parts that matters most in selecting Lightweight platforms are covered here and should be sufficient to come to conclusion:
- functional requirements, that means requirements for pure integration features,
- non-functional requirements as technical monitoring and deployment, and
- organisational aspects
Any traditional ESB covers lot of features that not everyone would be using at a time. There in, we need get over the pitfalls of complex and heavy platforms:
Connectivity and communication: Any integration initiative starts with a question, what has to be integrated? If standard protocols only are used it is relevant to look towards lightweight ESBs. One can think of combining asynchronous messaging with lightweight ESBs to get best of both the worlds, either it can call a distant broker or can embed in itself.
Messaging patterns: Messaging patterns have to be chosen in view of business flows to implement. Some patterns are overly complex most of them are fundamental to any integration solution. Lightweight ESBs bring simplicity and efficiency by configuring only the required messaging patterns.
Transformation: Lightweight ESBs have the same mapping capabilities as traditional ones. The general notion for years have been ‘No business rule in middleware’, therefore same issues apply here as well: whether to put the mapping logic on applications side or on the middleware side, whether to use Canonical Data Model or not, etc.
Non – Functional requirements
Deployment: Robustness and availability are greatly eased by embedding a lightweight ESB into an application: the integration system is not a bottleneck neither a single point of failure (SPOF) anymore. And over all there is no additional system in the infrastructure. The integration framework as embed in the application then acts as the integration module of a software package. This is typically the positioning of Spring Integration.
One limitation with embedding the middleware into an application could be the impossibility to implement flows with a many-to-many cardinality which in real life are uncommon.
This may not be a problem anymore with lightweight integrations as most of the products we talked in detailed above now offer good capabilities around performance their internal architecture consists of less layers and that these layers are more integrated together, they offer good performances.
Architectural initiative: Rightly or wrongly, the choice of an ESB is often made at Enterprise level, for strong infrastructure rationalization purpose. Even if the ESB is centralized, try at least not to centralize the integration teams too much. Keep teams autonomous and reactive instead.
Technical competences: The cost of access to the technology is lowered as any JEE (Spring ideally) developer can do the job: developers can use their preferred development environment and capitalize on their JEE knowledge, without learning all the specificities of a vendor solution.
Below picture depicts market leaders as per report published by Forrester & Gartner in early 2019.
Case examples of lighter integrations
American Express Global Business Travel
“Developed integrations with Boomi in a quarter of the time it took us before, and those integrations run three or four times faster than they did on our previous platform, handled complex business use case and application silos that needed to exchange data”
International Airlines group (IAG)
- Huge cost & time savings
- Business were able to focus on new functionality without worrying about various data formats
- Well integrated with other IT solution partners
The Wells Fargo API Gateway, powered by WSO2 API Manager, has over 20 APIs in production for various data services and payment functions such as account aggregation, account balance, foreign exchange, and wire payments. Wells Fargo has built up its transactional APIs to be re-used, allowing customers to move from one experience to another with minimal changes to their resources underlying the APIs.