I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.

To give an impression of the compute requirements, when deploying this to a k8s cluster, I allocate about 3.5 GB of RAM and 1 vCPU to each pod, and the VM is a high-compute performance type. The task takes about 2 minutes to run and uses up 100% of the vCPU and the RAM usage varies between 2 and 3.5 GB depending on the request parameters.

Ideally, the service just processes the request and returns the result in the response! However, that means the service is unavailable for 2 minutes before it can process the next request.

So, I thought to use Celery to offload this work to a background worker on a task queue. This works well; the API can now handle more requests and wait for the worker to finish them. However, the request/response cycle didn't exactly get any faster, as the API is literally just waiting for the worker, then needs to get the result from Redis and return that to clients.

I guess it can be seen as an improvement since requests are now being processed through a queue, but I feel like it comes at the cost of tons of overhead and a pretty sensitive dependency on both Redis and RabbitMQ since the API has to wait for results to come in. Handling network failure modes is pretty tricky to get right, and proper Celery configuration in general is complicated.

I can think of some alternatives:

  • Stick to the API doing the processing directly without Celery...
    • ... and just scale API instances
    • ... and use some off the shelf solution to queue requests (do these exist?)
  • Move the problem to the client by providing separate HTTP API endpoints to submit processing requests and to collect results

But I still feel like I'm missing something. Are there better solutions out there?

  • 2
    dont process long running requests in the api. even if you pass it off to a background thread its not a good idea. Have a seperate program that picks up the jobs, proccesses them and marks them done
    – Ewan
    Commented May 31, 2022 at 21:44

1 Answer 1


Move the problem solution to the client by providing separate HTTP API endpoints to submit processing requests and to collect results

Fixed that for you.

Doing this gives you a number of advantages:

  • Clients can get updates on progress: "Oh, that job is 90% done, update progress bar for user" (or whatever paradigm you have)
  • Clients can cancel tasks they decide they don't want any more: "Whoops, wrong params, please don't waste your time on that one"
  • Means you're not trying to hold open an HTTP connection for minutes at a time. Unless you're completely in control of your networking environment, you may discover some infrastructure closes connections for you if they're idle for too long.
  • 1
    "Clients can cancel tasks" Well that is true for fully synchronous requests as well isn't it? HTTP servers can (and often should) abort operations if the connection is closed. Commented May 31, 2022 at 14:29
  • Another potential advantage: it makes it much more natural to re-use the results of a previous calculation, if the same input parameters are given. Both inside your application, but also (if the same input results in the same URL to later get the response) in other layers of the network. Commented May 31, 2022 at 14:31
  • 1
    How would I avoid polling? Ideally the client is actively notified when the task is done, and even of progress updates. I'd prefer being able to control the traffic from the server side, if you will. What do you think?
    – Korijn
    Commented Jun 1, 2022 at 8:41
  • @Korijn WebSockets is the answer if you want something "HTTP-like" but with the ability to push to the client. If you're stuck with actual HTTP, polling isn't terrible. Commented Jun 1, 2022 at 8:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.