8

I am designing a AWS web service which is going to get 1000 TPS from devices(Android) and it has dependency on multiple downstream services. The usecase is to hit this service periodically from device, get a piece of data and cache it in the device memory. Since device does not require the data immediately, I designed this way

Device: puts a request in queue (SQS)
Service: polls messages from SQS, process the requests and publish the result to devices via FCM

The Problem is service takes at max 2 seconds to process the request and downstream services would not scale as much as service. In short, I can only process fraction of incoming requests per second (Lets say 200 requests, 20% of actual TPS). This leads to backpressure build up in the request queue. Reading through Internet, I found that general strategies to handle backpressure are

  1. make queue bounded, throttle the producer when it exceeds its size and make producer retry after some delay
  2. Increase number of consumers. (In this case, This is not an option due to downstream bottleneck)

Questions:

  1. How throttling helps to solve backpressure problem? If the TPS is consistent, Wouldn't it create the same problem even when producer retry after delay? At the end some producers will exhaust retries and requests go unprocessed.

  2. Initially I wasn't aware of backpressure and was thinking storing messages in queue will aid asynchronous processing but now I am starting to feel queue is creating more problems than It helps. Is queue even relevant for this usecase ?

  3. What are the real benefits of having a queue in front of service?

Appreciate any help!!

3
  • 5
    "Throttling the Producer" in this case would be returning error codes to the Android clients, thereby not having to actually process those requests. If TPS is consistent, then you need to make sure those Android clients don't retry with delay. Jul 19, 2021 at 22:27
  • 1
    run 6 copies of the service?
    – user253751
    Jul 20, 2021 at 10:52
  • 2
    You would normally drop or expire messages if you have a bound queue size and can’t throttle the producers. In some cases this can be made explicite with a sampling strategy
    – eckes
    Jul 20, 2021 at 11:15

3 Answers 3

22

A good analogy here is to think of a dam on a river. The river corresponds to the incoming data, and the dam to your consumer. There are three possibilities at any given point:

  1. The incoming river's flow is greater than the dam's outflow
  2. The incoming river's flow is less than the dam's outflow
  3. The incoming river's flow is the same as the dam's outflow

In situation 1, a lake grows behind the dam. This corresponds to your queue. In scenario 2, the lake shrinks. In scenario 3, the lake's volume doesn't change. A big part of why we have dams is to make the downstream flow more consistent. That is, when there's a heavy rain, the lake will get larger but the flow out can be limited. When there is a drought, the lake's reserves are drawn down to keep the outflow higher than it would be otherwise.

So the volume of the lake is equivalent to the total inflow minus the outflow. There is a limit, however, to the amount of water the lake can hold. When the lake is full, you either need to release more water or somehow divert the incoming water.

This corresponds to the iron law of queuing: the depth of the queue is the number of messages received minus the number of messages processed (or removed.) There's no magic. If you don't, on average, pull as many messages as you are putting on the queue, it will grow and eventually hit some sort of size limitation. Queues don't allow you to process more messages; they act as a buffer to help even out the flow and prevent failures when the incoming volume spikes. They also help with distributing the messages to multiple consumers efficiently. Alternately, they can be used as a 'holding area' for batch processing as noted by 'supercat' in the comments. But the law still holds: your overall processing rate must accommodate the incoming rate or your queue will grow.

The upshot: to resolve this issue, you need to either send less to the queue or process them faster. There is no other solution. Backpressure is actually a good thing in a lot of scenarios. It allows the producers to know when the queue is filling up so they can react.

It sounds to me that your issue is the 'downstream bottleneck'. You will never be able to process the volumes that you have coming in until you resolve that. A queue will simply delay how long it takes until you can no longer accept the incoming data.

7
  • 2
    "Queues don't allow you to process more messages; they act as a buffer to help even out the flow and prevent failures when the incoming volume spikes" - Big thanks for sharing this tip. Now I am getting how queues help to streamline the load.
    – Jegan Babu
    Jul 19, 2021 at 17:31
  • @JeganBabu I'm glad it helped. You might want to post a new question about your downstream bottleneck. Those kinds of questions tend to get a lot of answers.
    – JimmyJames
    Jul 19, 2021 at 17:38
  • Queues can help one to process more messages if the downstream side processes messages intermittently, and the peak rate at which the upstream side can supply messages is lower than the peak rate at which the downstream side can accept them. When connecting some older dot matrix printers to computers with slow printer ports, adding a queueing print buffer could increase the speed of high-resolution graphics printing by about 25-50% because the printers couldn't accept data while they were printing. Without an added print queue, a computer would have to wait while a line was output...
    – supercat
    Jul 20, 2021 at 5:54
  • ...before it could start sending data for then next line. If peak rate of the buffer's downstream connection was faster than the peak rate of the computer's connection, having the computer load up the buffer while the printer output a line would make it possible for the buffer to feed the next line to the printer faster than the computer could have done, thus reducing the amount of time the print mechanism is waiting for data.
    – supercat
    Jul 20, 2021 at 5:56
  • 1
    @TheRubberDuck I probably wasn't clear: the volume of the lake is equal to the total volume of water that has flowed in minus the total volume that has flowed out. (I'm ignoring evaporation etc. to keep things simple.) Likewise, the depth of a queue at any point is the total messages that have been added minus the total messages removed at any point in time.
    – JimmyJames
    Jul 20, 2021 at 14:31
4

How throttling helps to solve backpressure problem? If the TPS is consistent, Wouldn't it create the same problem even when producer retry after delay? At the end some producers will exhaust retries and requests go unprocessed.

Throttling very explicitly reduces TPS because it causes the client to discard some requests it would otherwise have sent. So the TPS is not consistent.

Whether this is done in the client or the server isn't important (unless the client<->server connection is itself the bottleneck). It still reduces effective TPS at the choke point.

Initially I wasn't aware of backpressure and was thinking storing messages in queue will aid asynchronous processing but now I am starting to feel queue is creating more problems than It helps. Is queue even relevant for this usecase ?

It's normal to keep a request queue per client and simply limit its size. Letting it grow without bound will eventually kill your server and make your client sluggish or unresponsive in the meantime.

What are the real benefits of having a queue in front of service?

You always have a queue in front of a service.

If it's not explicit, you have an implicit queue of in-flight requests stored somewhere in your network stack, and the server can't easily inspect or manage its length.

Keeping an explicit queue is just a way of exposing this reality directly to the server so it can manage it deliberately.

For example, if you can collapse or combine requests from the same client, then you can choose to favour latency when the queue is empty, and throughput when the queue is full. This is a really useful way to adapt to variable load.

0

If you want to implement backpressure then you need some messages going back up to the client when the queue is filling up. This can be done by blocking the queue, RabbitMQ does this automatically.

You would need to implement something on the Client which notices when it cant send messages as fast as it wants to and drops or batches the messages.

I would guess with a mobile client you would implement this anyway to deal with poor connections.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.