Long-running compute-intensive tasks in APIs: background workers?

Question

I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.

To give an impression of the compute requirements, when deploying this to a k8s cluster, I allocate about 3.5 GB of RAM and 1 vCPU to each pod, and the VM is a high-compute performance type. The task takes about 2 minutes to run and uses up 100% of the vCPU and the RAM usage varies between 2 and 3.5 GB depending on the request parameters.

Ideally, the service just processes the request and returns the result in the response! However, that means the service is unavailable for 2 minutes before it can process the next request.

So, I thought to use Celery to offload this work to a background worker on a task queue. This works well; the API can now handle more requests and wait for the worker to finish them. However, the request/response cycle didn't exactly get any faster, as the API is literally just waiting for the worker, then needs to get the result from Redis and return that to clients.

I guess it can be seen as an improvement since requests are now being processed through a queue, but I feel like it comes at the cost of tons of overhead and a pretty sensitive dependency on both Redis and RabbitMQ since the API has to wait for results to come in. Handling network failure modes is pretty tricky to get right, and proper Celery configuration in general is complicated.

I can think of some alternatives:

• Stick to the API doing the processing directly without Celery...
  • ... and just scale API instances
  • ... and use some off the shelf solution to queue requests (do these exist?)
• Move the problem to the client by providing separate HTTP API endpoints to submit processing requests and to collect results

But I still feel like I'm missing something. Are there better solutions out there?

Answer 1

Move the problem solution to the client by providing separate HTTP API endpoints to submit processing requests and to collect results

Fixed that for you.

Doing this gives you a number of advantages:

• Clients can get updates on progress: "Oh, that job is 90% done, update progress bar for user" (or whatever paradigm you have)
• Clients can cancel tasks they decide they don't want any more: "Whoops, wrong params, please don't waste your time on that one"
• Means you're not trying to hold open an HTTP connection for minutes at a time. Unless you're completely in control of your networking environment, you may discover some infrastructure closes connections for you if they're idle for too long.