I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.
To give an impression of the compute requirements, when deploying this to a k8s cluster, I allocate about 3.5 GB of RAM and 1 vCPU to each pod, and the VM is a high-compute performance type. The task takes about 2 minutes to run and uses up 100% of the vCPU and the RAM usage varies between 2 and 3.5 GB depending on the request parameters.
Ideally, the service just processes the request and returns the result in the response! However, that means the service is unavailable for 2 minutes before it can process the next request.
So, I thought to use Celery to offload this work to a background worker on a task queue. This works well; the API can now handle more requests and wait for the worker to finish them. However, the request/response cycle didn't exactly get any faster, as the API is literally just waiting for the worker, then needs to get the result from Redis and return that to clients.
I guess it can be seen as an improvement since requests are now being processed through a queue, but I feel like it comes at the cost of tons of overhead and a pretty sensitive dependency on both Redis and RabbitMQ since the API has to wait for results to come in. Handling network failure modes is pretty tricky to get right, and proper Celery configuration in general is complicated.
I can think of some alternatives:
- Stick to the API doing the processing directly without Celery...
- ... and just scale API instances
- ... and use some off the shelf solution to queue requests (do these exist?)
- Move the problem to the client by providing separate HTTP API endpoints to submit processing requests and to collect results
But I still feel like I'm missing something. Are there better solutions out there?