Does it make sense to spawn request driven jobs in a thread or process?

Question

I've developed a large application that performs jobs, such that it receives input x and outputs y. For this application, speed is the top priority. The application is capable of running concurrently. I want to modify this application so that it can be used as an REST API, where users sends x and receives y as its response, hosting the application on a server.

It is possible to serve the requests by spawning a new thread to run the job, or a new process.

I have some concerns with multithreading:

Depending on the users input, the job can hang indefinitely. Therefore it is essential that some kind of time out mechanism is implemented which will terminate the job if a time limit is exceeded. If the job was to run as a process, this can easily be done by simply ending the process. However if the job was ran on a thread, I would have to modify the actual application to cooperate with the time out code and also handle garbage collection, which would be a lot work and also may have implications for performance (e.g. continuously checking if the time limit has been exceeded).

I'm honestly a bit perplexed as to which route I should go down and was wondering if anyone had any advice or insight into how a server application that performs requests driven jobs should optimally work.

Answer 1

"Does it make sense to spawn request driven jobs in a thread or process?"

Generally speaking no. You don't control the number of requests and there will be an optimal number of threads to use per machine. Exceeding that will decrease performance.

If you host an API in a webserver of some kind the webserver itself will manage the threads for incoming requests. But you will want to implement a timeout, as if the time to process a request exceeds the rate of incoming requests your application will grind to a halt.

Anything over a couple of seconds and I would be looking at offline processing.

Api request comes in, write to a queue, return OK we have received your request, check back later

Backend processers on multiple machines check the queue for new requests and process them

On completion they write the result to a database

API Request "have you finished yet?" check database and return result.