Max recommended downstream calls for micro service inter-communication

Question

Broadly speaking how many downstream calls would be acceptable for a synchronous Microservice interaction, from say a gateway API down to a a few supporting services? We are seeing big swings in response time with 4-6 calls - some pass-through proxies. Is this expected?

Obviously the fallacies of distributed computing should be heeded here but I am curious what would be considered sane given a network of services on the same LAN.

Answer 1

What is sane will vary.

• If your service is hidden behind a responsive UI, it might be fine to wait.
• If on the other hand its preventing users from doing another task, it is a problem.

Expectations

Figure out how responsive your site/service must be. Pin that down in a measurable way.

80% are within Xms, 95% are within Yms, ... 100% are within Zms

It may make sense to break that expectation down by current system load, or time of day.

• How should your system behave with x thousand users, what if there where y thousand?
• How should your system behave between 8am and 8pm?

You may want to get more specific:

• Maximum concurrent users, and anything more is considered exceptional service.
• Service Degradation: to maintain responsiveness sacrifice the response quality. ie. don't return statistics with the user name and badge.
• Red line processes: to maintain functionality at the expense of other functionality. ie. prioritise processing orders over browsing comments and reviews.

Collect Data

Run several clients (selenium, or custom web-requests scripts) that ping a page, or group of pages with some pattern of distribution. Try to make it look like peek load, or general days. Measure the response times at the client.

If 80% are within Xms, 95% are within Yms, ... 100% are within Zms then you are good. Otherwise figure out what is slowing down that group of requests.

Also make note of the service degradation, and red lining. Tight response times aren't helpful if service quality is degrading too much, or the red lines are being too dominant over other features.

Experiment

Now that you know specifically what types of request are too slow, too poor, or needing more resources, and under what loads, find ways to improve it.

Perhaps:

• Hardware like caches, proxies, load balancers.
• Front-end changes to avoid or delay loading that data, perhaps splitting the pages.
• Back-end changes by combining, optimising, or splitting services.

Collect more data, did you make things better or worse?

Validate

Always ensure that your experiment holds true in production.

To do this you need to have good monitoring in production. Compare what you saw before to what you see now. Has the quality apparently improved?

Chaos Engineering

Only once you've matured the system, and believe it will be fine. Try to engineer a load pattern, and observe if the metrics meet your expectations.

Make use of the client scripts designed against the test system. Obviously be picky about which tests are to be used, no need to corrupt your system here.

Also warn everyone about the experiment way in advance. Don't surprise your own teams. If they are uncomfortable with the scale, or the target, listen to them. Reduce the scope and make the test tighter, build up confidence slowly.