Put Integration Layer (ESB) between Frontend and Backend

Question

I am working on a project where the architect decided to put an ESB (integration layer) between the Front end and back end. The driver was to make the services reusable across channels and the enterprise and to utilize the existing infrastructure (ESB).

This was really achieved for a small set of the services where they were used both over web and API gateway channels, but for the majority of services, ESB just introduced an additional layer to maintain (with all the related burden), especially that the services are very domain specific. My question is:

1- Is it a good practice to make ESB between front end and back end generally? Note that this is different from usual EAI where different systems communicate. It's just an abstraction layer between FE and BE whose businesses are tightly coupled.

2- In case some services might be reusable and most are not, what is the best practice?

1. Use ESB for all, as mentioned, for unification and consistency
2. Use Normal BE service technology (Spring Boot Rest services for example) and develop ESB services for the reusable ones only
3. Rely only on Normal BE service technology (Spring Boot Rest services for example)
4. another option ??

3- Is it worth keeping an additional layer with all the burden of its maintenance only because it might be used sometime in the future and to keep all communications consistent?

Kindly support your answer with references if available

Answer 1

Generally speaking, if there is not ESB operating in the enterprise, avoid introducing one unless you have good justification, and mostly, there are few - if any! It might help you a lot in your decision if you have access to Gartner's awesome report Choosing an Architecture for Managing APIs and Services, they detailed the difference between Service-centric and API-centric architectures, and they suggested a No-ESB architecture where using ESBs is totally optional in modern architectures. They said explicitly there:

The majority of Gartner clients that have embarked on [Service-Centric] have failed to achieve measurable success, because they prioritized governance for control rather than productivity.

Stay focused on the future, which is an API-centric world.

Which seems appropriate to your case.

There is some good excerpts from this report in these slides Monolith to serverless: service-based architectures in the enterprise, and if you are an Arabic speaker you can watch the accompanied session with the slides.

And now the details, which are mostly subjective, and depends on my personal experience that might be different than others' experience and answers...

The driver was to make the services reusable across channels and the enterprise and to utilize the existing infrastructure (ESB).

Reusability is overrated, and adds a lot of complexity to the architecture, and the integration will be a bottleneck. But since you already have an operating ESB, and it seems as if you are forced to use it, then, well, use it wisely and try to get the most benefit you can from it.

1- Is it a good practice to make ESB between front end and back end generally?

No. Layers of indirection add complexity, and this complexity should be justified.

Here are some valid justifications:

1. All applications in the enterprise works in the same way. Here consistency wins. And this answers your question:

3- Is it worth keeping an additional layer with all the burden of its maintenance only because it might be used sometime in the future and to keep all communications consistent?

2. The ESB already exists in the enterprise and you need some of its functionality, e.g. you need orchestration, enrichment, transformation, or messaging/asynchronous communication. The most needed of these is messaging and you can simply use a message broker for it, e.g. RabbitMQ if there is no other existing solution in the enterprise, which is not your case since there is an operating ESB.
3. You have a lot of integrations in the enterprise and point-to-point integration became a mess, then adding a layer of indirection will be helpful to add some order to the chaos. This layer of indirection might be a centralized ESB or something else (see below).

It's just an abstraction layer between FE and BE whose businesses are tightly coupled.

Now, it seems you will not get much benefit of it. But of you can if you want. For example, you can get a lot of insights about your services/APIs usage and performance if all requests pass through a centralized thing (why I say thing not ESB? See below).

2- In case some services might be reusable and most are not, what is the best practice?

My preference is to have a centralized thing, but this is subjective...

Use ESB for all, as mentioned, for unification and consistency

The pros: consistency + I can add here cross-cutting concerns like security and logging at request level.

The cons: single-point of failure, vendor-lock in, scalability will be a big thing to consider.

Meanwhile, my endpoints are as smart as much as possible, and the centralized thing is as dumb as possible (see: Smart endpoints and dumb pipes)

Then, what is the thing I mentioned before?

Actually, it is an API gateway, or a full-fledged API platform, and I have a very good success story in a big enterprise where we used API platform for integration as well, but we enriched it with a dumb message broker, and centralized logging. And all the heavy liftings are kept in the APIs (smart endpoints).

My final advise would be: If you can move away from the ESB then do, but if you cannot, then try to get the most out of if.

Answer 2

There is no good general answer to these questions.

You have to weigh the pros and cons in your situation for your product and team and choose the path that looks best. If you dig into the problem a bit and find some more specific issues, you could get better answers.

1. Is it a good practice to make ESB between front end and back end generally?

No. It's a good practice to include an ESB when you need one. Some projects don't need this layer and we generally save time and money by keeping software as simple as possible to meet the requirements, but no simpler.

Reference: "Everything should be made as simple as possible, but no simpler." - Albert Einstein

2. In case some services might be reusable and most are not, what is the best practice?

It Depends. First see the answer to question 1. If you have determined that you need ESB for some services the question becomes whether to use it for all services. The answer depends on the situation.

If your team invested a lot in learning to use and support the ESB approach effectively it's reasonable to keep using it where you don't technically need it, just in the interest of having one way to do things and doing that one way very well.

If your team has already invested the time to learn to build and support direct http interaction between FE and BE and/or the ESB carries a significant burden on development, infrastructure, or support cost or adversely impacts some services, then it's reasonable to use the ESB only where you need it. If having two ways to do things creates no undue burden and has benefits why not do it? It's up to your team and managers to decide the merits of this choice. Even with the same services two different teams might make different reasonable best choices.

Reference 1: "a foolish consistency is the hobgoblin of little minds" - Ralph Waldo Emerson in "Self Reliance"

Emerson says, basically, trust yourself, make your own way. There's really no alternative

Reference 2: "Use the force Luke." - Obi-Wan Kenobi

Sometimes when the objective choices are unclear you have to go with your gut.

3. Is it worth keeping an additional layer with all the burden of its maintenance only because it might be used sometime in the future and to keep all communications consistent?

No. Don't add another layer and make things more complex just because you might need it in the future. Whether it's worth having the additional layer for consistency is addressed by question 2.

This question can be summed up by the maxim YAGNI (You aren't gonna need it.) from Extreme Programming. In general, add complexity when you need it, not before. The future is unknown.

You could reasonably relax this in a case where you have good evidence that you are likely to need the additional layer and the cost of adding the layer later is sufficiently high. So you make a bet that it's better to add the layer at the start.

The origin of YAGNI is the incredible fallibility of predicting the future and the tendency for developers to try to put in a lot of features in case they will be needed. So hold a high standard for evidence that you are likely to need it.

In my experience, for example, we might do a more complex design because we know that new regulations are in the process of being defined and our best analysis is that the more complex design will be needed when the regulations are adopted. There's no telling exactly when they will be adopted. The cost and time to switch our design is significant. We would rather be able to respond quickly when regulations come by adjusting details. There's a chance the regs will never come, say with a change of government and a new direction, but our analysis is that they will. We're making a bet. We might be wrong.

Reference: "Optimism is an occupational hazard of programming. Feedback is the treatment." - Kent Beck, author of Extreme Programming

I hope these answers show that the questions are really too general for specific answers from people who do not know your team and applications. At best we can suggest ways to think about these questions.

Answer 3

As an Architect who used to position ESB for many years to customers, I would say that adding ESB to architecture is not needed for most of the cases, and most businesses have moved away beyond ESB to a more agile approach for service integration, like API gateways.

To understand why ESB was used in the first place, the main motive was not really reusability, this came later on as an added value, but rather ESB evolved from EAI (enterprise application integration) where the idea was to offload service consumers from writing low-level integration (such as FTP, MQ, SOAP, etc) to back-end systems and rely on a unified interface that is exposed to them.

When the SOA era started in the last decade and Web services, XML and SOAP became mainstream, tech vendors augmented the EAI layer with an additional layer (they called it ESB) that supports these new integration patterns and advertised the fact that you could build a canonical business/data model that could be reused across many services (hence the reusability part), but the reality was much more complex than this, most of the services built on the ESB were specific which made it harder to reuse, also using canonical data models proved to be slowing things down drastically, for the minor benefits of reusability, And when microservices took-off and became popular after while, ESB were seen as a legacy that does not cope with the new world of ever-changing IT landscape, and business pragmatically traded agility over reusability driven by the fact that the time-to-legacy for micro-services has decreased to months rather than years, which effectively mean that the "reusability" of ESB will not be effective as much.

In my view, using ESB now is like going backwards in time, ESB would be beneficial ONLY IF your back legacy end-systems are very stable with old interfaces (SOAP, XML, FTP, MQ, etc) that do not change very often, and that have or allow you to build a canonical data model on top that can be reused across your web services.

Use API gateways rather than ESB to do your API integration, especially to microservices, as we see this pattern used in many businesses and delivered a much higher success rate than traditional ESBs. finally, refer to thought works tech radar, they placed ESB on hold for many years now. https://www.thoughtworks.com/radar/tools/esb

Answer 4

Introducing a layer that is not needed either it is ESB or API gateway I see it as a burden to the application and sometimes it might be a single point of failure.

As @joshp said, you aren't gonna need it. That simple.

And as @sameh said, to expose parts of the services to customers, you can use an API gateway. But keep it for only this specific set of services needed by the customer and not for all services.

Answer 5

TLDR;

Do not use/add ESB when you don't have any "Concrete and Serious Requirement " for it and you won't need it till the next 6 months. You can easily add this layer whenever you want and it would be done seamlessly to your BE/FE if you have considered only some basic principles in defining application APIs.

More Details

ESB is a solution for enterprises to facilitate integration between a set of service providers and consumers that have different protocols and difficulties. Recently, the enterprises are going to use some modern architectures and components like "API Gateway" that is a little bit different and provides less headache and more efficiency.

For your case, I believe API Gateways like "Kong" would be a better option because you can provide a proper authentication layer for consumers on the API GW layer and not making your BE complex due to handling credentials of different consumers while it might have its own client permission management and it would make it very complex. So, I think if you are going to have new consumers for your BE, you can easily add Kong to your architecture within 1 -2 weeks and set up the authentication layer for their use. Moreover, Kong would provide a lot of tools like Rate Limiter, Load Balancer, Caching and so forth for your service that would be very useful and the most important part for the Kong is that it's very light and it would not consume a lot of resources so you can save a lot of money compared to other ESB tools like WSO2.

You can also read more about Kong on this Link.

Principles for URL Definition

It's easy, Just add some prefixes to your APIs, it would let you to have a proper control on your APIs via ESB/API Gateway and you can easily add ESB/API GW layer to your architecture without impacting current Front-end apps, Also, you're able to have backward compatibility.

Sample API:

https://api.yourdomain.com/book/add

Better Structure for API GW:

https://api.yourdomain.com/library/v1/book/add