Background: We're providing an API that provides information about all users within a given group. Our API is a high level facade over another low-level REST API.
To gather those information we first have to get a list of all users in the group via REST call to the low-level API and then for each user do two additional requests to get the required information.
Our API is a Spring Boot application running in kubernetes.
Problem: The response time of our API increases linearly with the number of users in the group since we have to do many consecutive requests to the low-level API.
We could parallelize the requests for individual users to the low-level API to speed up the process.
However that's where I'm not sure if it's a good idea or how to handle the threads used for that. It feels wrong two spin up additional threads for each incoming request.
Our application uses the default web thread pool size
server.tomcat.threads.max of 200.
- Use a new thread pool with e.g. 10 threads to do the additional processing for each request
- Large overhead for starting threads
- Could lead up to 2000 threads if all webserver threads are in use
- Easy to implement
- Number of threads depends on the CPUs of the node
- All requests share the same low number of threads
- Create a shared thread pool with e.g. 200 threads that can be used by all requests
- We have to be careful with resource management and returning threads to the pool
- Good control over maximal load and good speedup
Is there any general advice how to handle multiple threads for individual requests?