I've been reading a lot about microservice architectures for server applications, and have been wondering how the internal network usage is not a bottleneck or a significant disadvantage compared to a monolith architecture.

For the sake of precision, here are my interpretations of the two terms:

  1. Monolith architecture: One application in a single language that handles all of the functionality, data, etc. A load balancer distributes requests from the end user across multiple machines, each running one instance of our application.

  2. Microservice architecture: Many applications (microservices) handling a small portion of the functionality and data. Each microservice exposes a common API that is accessed through the network (as opposed to inter-process communication or shared memory on the same machine). API calls are stiched mostly on the server to produce a page, although perhaps some of this work is done by the client querying individual microservices.

To my naive imagination, it seems like a microservices architecture uses slow network traffic as opposed to faster resources on the same machine (the memory and the disk). How does one ensure that API querying through the internal network will not slow down the overall response time?

up vote 51 down vote accepted

Internal networks often use 1 Gbps connections, or faster. Optical fiber connections or bonding allow much higher bandwidths between the servers. Now imagine the average size of a JSON response from an API. How much of such responses can be transmitted over a 1 Gbps connection in one second?

Let's actually do the math. 1 Gbps is 131 072 KB per second. If an average JSON response is 5 KB (which is quite a lot!), you can send 26 214 responses per second through the wire with just with one pair of machines. Not so bad, isn't it?

This is why network connection is usually not the bottleneck.

Another aspect of microservices is that you can scale easily. Imagine two servers, one hosting the API, another one consuming it. If ever the connection becomes the bottleneck, just add two other servers and you can double the performance.

This is when our earlier 26 214 responses per second becomes too small for the scale of the app. You add other nine pairs, and you are now able to serve 262 140 responses.

But let's get back to our pair of servers and do some comparisons.

  • If an average non-cached query to a database takes 10 ms., you're limited to 100 queries per second. 100 queries. 26 214 responses. Achieving the speed of 26 214 responses per second requires a great amount of caching and optimization (if the response actually needs to do something useful, like querying a database; "Hello World"-style responses don't qualify).

  • On my computer, right now, DOMContentLoaded for Google's home page happened 394 ms. after the request was sent. That's less than 3 requests per second. For Programmers.SE home page, it happened 603 ms. after the request was sent. That's not even 2 requests per second. By the way, I have a 100 Mbps internet connection and a fast computer: many users will wait longer.

    If the bottleneck is the network speed between the servers, those two sites could literally do thousands of calls to different APIs while serving the page.

Those two cases show that network probably won't be your bottleneck in theory (in practice, you should do the actual benchmarks and profiling to determine the exact location of the bottleneck of your particular system hosted on a particular hardware). The time spent doing the actual work (would it be SQL queries, compression, whatever) and sending the result to the end user is much more important.

Think about databases

Usually, databases are hosted separately from the web application using them. This can raise a concern: what about the connection speed between the server hosting the application and the server hosting the database?

It appears that there are cases where indeed, the connection speed becomes problematic, that is when you store huge amounts of data which don't need to be processed by the database itself and should be available right now (that is large binary files). But such situations are rare: in most cases, the transfer speed is not that big compared to the speed of processing the query itself.

When the transfer speed actually matters is when a company is hosting large data sets on a NAS, and the NAS is accessed by multiple clients at the same time. This is where a SAN can be a solution. This being said, this is not the only solution. Cat 6 cables can support speeds up to 10 Gbps; bonding can also be used to increase the speed without changing the cables or network adapters. Other solutions exist, involving data replication across multiple NAS.

Forget about speed; think about scalability

An important point of a web app is to be able to scale. While the actual performances matter (because nobody wants to pay for more powerful servers), scalability is much more important, because it let you to throw additional hardware when needed.

  • If you have a not particularly fast app, you'll lose money because you will need more powerful servers.

  • If you have a fast app which can't scale, you'll lose customers because you won't be able to respond to an increasing demand.

In the same way, virtual machines were a decade ago perceived as a huge performance issue. Indeed, hosting an application on a server vs. hosting it on a virtual machine had an important performance impact. While the gap is much smaller today, it still exists.

Despite this performance loss, virtual environments became very popular because of the flexibility they give.

As with the network speed, you may find that VM is the actual bottleneck and given your actual scale, you will save billions of dollars by hosting your app directly, without the VMs. But this is not what happens for 99.9% of the apps: their bottleneck is somewhere else, and the drawback of a loss of a few microseconds because of the VM is easily compensated by the benefits of hardware abstraction and scalability.

  • Sure, we can say that JSON responses are small, but what about their quantity? I feel like a website under heavy load would have more network traffic in a microservice architecture than a monolithic architecture (where the only network traffic is to/from the database server(s)). Caching can help, but for real time and/or dynamically generated content, I don't know how far caching would go. – James Mishra Mar 9 '15 at 23:55
  • @JamesMishra: I edited my answer to address your concerns. – Arseni Mourzenko Mar 10 '15 at 0:59
  • Your answer is perfect. Not only did you answer every objection I can think of, you answered objections I didn't think of. – James Mishra Mar 10 '15 at 20:10
  • 5
    My 2 Cents from the real world: A system composed of very chatty microservices can suffer performance issues purely due to a choked network. Caching and an Event Stream based design is your friend in such cases. Besides network, CPU and memory, a microservice based system also needs to incorporate resiliency in its design: what happens if a microservice is down? How do you build retries, distributed transactions, self healing, monitoring - I suggest searching for "you must be this tall to use microservices" – dotnetguy Jan 27 '17 at 13:14
  • 1
    Please correct me if I'm wrong, but as far as I know, if you have a 1Gbps network, it means you can theoretically send 1Gb worth of data per second through that network. Regardless the number of connections. The higher the number of connections, the lower the bandwidth for each connection. So, your actual limit without upgrading your network to support a higher bandwidth would be 26.214 responses per second. Adding more servers won't increase your network bandwidth. If a single cluster can spit out that amount of traffic, adding more servers generating even more data will congest your network. – Sebbe Jul 18 '17 at 15:40

I think you are reading too much into the 'micro' part. It doesn't mean replace every class with a network service, but componentize a monolithic application into sensibly sized components, each one dealing with an aspect of your program. The services won't talk to each other, so at worst you've split a large network request into several smaller ones. The data returned won't be significantly different to what you're receive anyway (though you might return more data and consolidate it in the client)

  • 2
    "The services won't talk to each other." I imagine that microservices might have shared dependencies (authentication, perhaps?) that one might separate into another microservice. LDAP, in a sense, is an authentication microservice and I imagine that all other microservices talk to it. Or... does authentication only happen once? How does each microservice check permissions against authentication to prevent Direct Object Access attacks? – James Mishra Mar 9 '15 at 18:06
  • 2
    @JamesMishra well.. it depends. When I last used microservice architecture each service was totally independent of the others for security purposes (but also corporate silo reasons). Auth was handled by each differently, though controlled by a architecture policy. Still, there's no reason they couldn't talk to auth for example, or just have auth based on a library. But.. I was trying to say that they shouldn't be passing lots of calls between themselves, not that they shouldn't consume services as clients themselves. – gbjbaanb Mar 10 '15 at 8:33
  • @JamesMishra, auth is often it's own service in these environments, and so each service should just make use of that rather than doing a full implementation themselves. – Paul Apr 3 '17 at 10:02

You can use various forms of caching to eliminate some of the bottlenecks of retrieving data. You can also consider using a document based database (NoSQL) which can improve performance.

Here is an article which discusses performance in microservices: https://blog.dropletpay.com/lessons-learnt-building-a-microservice-architecture/

Lesson 6: Cache everything

Microservices tend to be very chatty. Instead of trying to optimise every call, it's easier to first optimise the communication between services. In our case, having a single client that handles all communication helps immensely. As long as services are well behaved and send back ETags or Last-Modified headers, the response can be cached in a shared Redis database. Making multiple HTTP requests is a lot quicker when a service is returning 304s. An added benefit of using a shared database here is that new services will already have a prewarmed cache for HTTP requests.

  • 1
    Your link is not working – darasd Dec 12 '17 at 11:14

By structuring your code and resource access such that the resulting system can be flexible enough to run as a monolithic application or a distributed one via configuration. If you abstract away the communication mechanism behind some common interface and you build your system with concurrency in mind, you can easily optimize everything after you have profiled your system and found the real bottle necks.

  • An example to explain what i assume that @mortalapeman mean: you have a java/c# interface IProductAvailibitiy where all IProductAvailibitiy consumers are linked against. There is also a class ProductAvailibitiyImpl that implements this interface and a ProductAvailibitiyMicroservice that uses ProductAvailibitiyImpl. The consumers can be configured to use either a local ProductAvailibitiyImpl or a remote proxy to ProductAvailibitiyMicroservice – k3b Apr 3 '17 at 17:04

Your naive imagination is right. And often that doesn't matter. Modern machines are fast. The main advantages of micro service architecture are seen in development and maintenance effort and time.

And of course there is no rule saying you can't use shared memory or even physically deploy multiple services in one executable. Just as long as you design it not to be dependent on that.

  • CPUs are fast. Memory is fast. SSDs are fast. But are network cards and routers and switches "fast"? Another answer insists so, but I'm not sure. – James Mishra Mar 9 '15 at 23:58
  • It definitely is easy to run into network speed problems. Run one service in San Francisco, another in Amsterdam, and consume them in Sydney. Delay is the key, not bandwidth. So don't do that. And make services as large as make sense – Stephan Eggermont Mar 24 '15 at 13:52

As a lot of people mentioned, it's not about network bottlenecks. It's more about network brittleness. So the first step is to avoid synchronous communication. It's easier than it sounds. All you need is services with right boundaries. Right boundaries result in services being autonomous, loosely coupled and highly-cohesive. Good service doesn't need information from another service, it already has it. The only way good services communicate is via events. Good services are eventually-consistent as well, so there are no distributed transactions.

The way to achieve this goodness is to identify your business-capabilities first. Business-capability is a specific business-responsibility. Some contribution to overall business-value. So here is my step sequence that I take when think about system boundaries:

  1. Identify higher-level business-responsibilities. There will be a few of them. Treat this services as a steps your organisation should walk through to achieve its business-goal.
  2. Delve deeper within each service. Identify lower-level services services comprising a parent one.
  3. Alongside the first two points think about service communication. They should do it primarily via events, just to notify each other about their business-process result. Events should not be considered as data conveyors.

Keep in mind that business-service includes people, applications, business-processes. Usually only part of it is represented as technical authority.

This could sound a bit abstract, so probably an example of service boundaries identification would be of some interest.

Just another factor to add to the current answers. With a coarse-grained services. You want to avoid the latency from all the calls so instead of making 10 calls you make a call that gets 10 pieces of data needed in a DTO.

And remember that microservices are not as micro as people think.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.