Skip to main content

Reference architecture of Agile Integration

OK. I said it. Integration is still around, but in different form from my last blog post. So what does modern integration looks like? Looking at how agile scrum has taken over traditional waterfall development framework, by enable shorter deliver cycles, faster feedback, and having the flexibility to rapidly adapt changes. I believe it’s time for traditional integration to be agile again. By breaking up traditional ESB into distributed microservices.

A little recap of what should be in Agile integration:
  • Distributed Integration
    • Lightweight, support distributed deployment
    • Pattern Based Integration
    • Reusable Connectors
    • Microservices
  • Containers
    • Cloud native solutions
    • Lean artifacts, individually deployable
    • Container based scaling and high availability
  • API
    • Well defined, re-usable, and well managed end-points
    • Ecosystem leverage

Base on these three principle, I was asked to create a reference architecture for it. After giving some thoughts into it, drawing up numerous boxes and diagram, something clicked when I watched a 3 year old playing. And here, this is my answer…

Yup. That is it! Reference architecture of Agile Integration. Architecture is often needed during the design phase of a software project, it provides a structure and backbone for the software project. Once it’s done, it’s done, any architectural modification will be difficult and can bring disastrous result.  We all knew “one thing will never change in software development is Change”.  Maybe it’s because regulation requirement, market demands or simply learning more about the business domain. So I started to think, can we build an architecture that is flexible for change, and can be shaped into the needs of a project as needed. So base on that concept, this is my reference architecture for Agile Integration, the modern integration application development that enables flexibility from many angles.  

There are the major components in my reference architecture diagram for Agile Integration:
PaaS - Provides a foundation for fast pace software development by allowing developer to self-service, as well as enable automated provision.It will also boost operational efficiencies by making the entire environment DevOps ready.

Security/Identity management(IAM) -  Handles basic authentication and authorization for application interfaces and platforms.

Automation - Automates both the processes of building applications, and rolling deployments strategies for upgrades. Together with continuous integration tools,  it can achieve continuous delivery of the application softwares.
Logging and Tracing - What was suppose to be easy in monolithic world, has now become one of the challenge in distributed microservice architecture for integration. As it has now become harder to have a unified view of how things are doing. It is important to have a way to see logging as a whole. And has tracing capability for different activities.   

Containerized packages - Container technology allows us to build immutable portable lightweight application package, that is language agnostic. The configuration are independent from this builds allowing same package to deploy quickly in multiple environment.

Container management - In large scale system it is essential to have a convenient management platform to manage all the containers running the applications. Taking care of the monitoring, discovery, recovery and failover of the running containers.

Microservices - Large applications/services are broken down into easy maintainable pieces and can be independently develop and deploy in distributed environment. Each of them should be built with failure in mind. Microservices

Load Balance/Service discovery/Network management  - Microservice are built for flexibility, according to current system load, the load should be automatically balance to the each running instance. All services should be registered and can be found in the system without complex setup. Ideally a proxy endpoint for external users to hide away the deployment complexity.  

Layered architecture -  Logically organize microservice into a sets of layers, each has it’s own responsibility to avoid duplication of effort and make it easier to replace later on. Currently there are 4 layers
  • Gateway layer - provides simple gateway routing capability such as versioning, dealing with different platform of devices.
  • Composite layer - important middle tier that handles composition of multiple microservices. The do more complex routing from processing the content data itself and sometimes handle more complex data aggregation or normalization.  
  • Base layer - like the name which is most likely represents the basic components of the system. Handles data retrieval or business logic processing.
  • Anti-corruption layer - This layer handles interface to legacy application or anything that work against microservice quick and flexible principle. This layer is built in protection wall to your system, by doing the transforming job and translate between two very distinct implementation of the system.   

Application domains - This one is more like a pattern then architecture. But since we mention about layers in software I also wanted to touch a little bit on this. Because it’s all about reining the microservice into a more organized way. It’s a good practice to have each application domain defined (using same set of data model). And have other external integration microservice to wire between each domains. One other good thing about microservice is that you have the flexibility to move each small microservice to different domains as which it fits better with.

API management - Managing the APIs by reinforced access policy, collecting statistics around the usage of APIs. Also building a healthy eco-system among the developer and users of APIs. Further to allow direct revenue source from the APIs.

There are certainly some limitation in today’s technology to allow a complete pliable architectural form. But I believe this model can bring flexibility into the system without significant impact to the system when changes occur. I will explain more details in my upcoming posts.


Popular posts from this blog

Red Hat JBoss Fuse - Getting Started with Fuse Integration Service 2.0 Tech preview

I just realized that I did not do a getting started for Fuse Integration Service 2.0 Tech preview before I did the pipeline demo, thanks for those of you who reminded me! :)

To get started with FIS 2.0, for people who has just getting to know the technology, here is how I interpret it. Basically, it's divide into two aspect,

1. Integration development, FIS uses Apache Camel as the core technology that creates, orchestrate, compose microservices into a super lightweight thin integration layer, and become the API provider and service orchestrator through exposing RESTful or messaging service endpoints. And you can choose to either package and run it with Spring-Boot or Karaf.

2. Application Deployment and Management, FIS takes advantages of OpenShift platform, and allows you to separately deploy the micro-integration service among distributed environment, at the same time takes care of the failover, high availability, load balancing and service lookup problem for you.

So, now we know …

Red Hat JBoss Fuse/A-MQ - Fuse and A-MQ Version 6.3 GA is released!

Fuse and A-MQ 6.3 GA has just went out. Maybe, you would think this is just only a minor version release why should I care? Hold your thoughts on that! Because they have done a lot of improvements and also added many new features into this release.

Besides various bug fixes and making sure Fuse Fabric is much more stable. There are two major change in this version update:

New Tooling in JBoss Developer Studio (JBDS) 9.1 GA. Newer Apache Camel version – Camel v2.17. I was really impressed by the work put in to make developing Camel application much simpler. First is the installation of tooling itself. Now it has a all-in-one installer so you don't need to worry about which plugins you need to check. See the videos below to see the new "Getting Started" of Fuse 6.3.

And If you notice from the above video, the presentation of camel route in JBDS has also updated. It fixed some of the miss representation of logic and making it easier to read.

Old Camel Route
New Camel Route
On …

Fuse Integration Service - Setup JBDS and create first quickstart application

Before we go and start creating our first application, I want to show you how to setup your JBoss Developer Studio, create a small application from the quickstart example and then running it on Fuse Integration Service.

I am using JBoss Developer Studio version 9, you can find it here.
After download the

double-click it, and start installing with default values.

After successful installation, we will need install the plugins for Fuse, on JBoss Central view, select software update, select enable early access.

And select JBoss Fuse Development for the plugin,

Click on install, and we are all set to go!

First thing first, we want to create a Fuse project to deploy on the base of Fuse Integration Service, which is OpenShift. If you have not installed it, please go back to my previous post for instructions. So on your JBDS, right click and start creating the project. Select new, maven project, if you have installed the plugin correctly, you should …