Background: We're providing an API that provides information about all users within a given group. Our API is a high level facade over another low-level REST API.

To gather those information we first have to get a list of all users in the group via REST call to the low-level API and then for each user do two additional requests to get the required information.

Our API is a Spring Boot application running in kubernetes.

Problem: The response time of our API increases linearly with the number of users in the group since we have to do many consecutive requests to the low-level API.

We could parallelize the requests for individual users to the low-level API to speed up the process.
However that's where I'm not sure if it's a good idea or how to handle the threads used for that. It feels wrong two spin up additional threads for each incoming request.

Our application uses the default web thread pool size server.tomcat.threads.max of 200.


  • Use a new thread pool with e.g. 10 threads to do the additional processing for each request
    • Large overhead for starting threads
    • Could lead up to 2000 threads if all webserver threads are in use
  • Use Stream.parallel()
    • Easy to implement
    • Number of threads depends on the CPUs of the node
    • All requests share the same low number of threads
  • Create a shared thread pool with e.g. 200 threads that can be used by all requests
    • We have to be careful with resource management and returning threads to the pool
    • Good control over maximal load and good speedup

Is there any general advice how to handle multiple threads for individual requests?

  • 5
    It might help a bit, but you run the risk of having all the request contend for the same database resources/locks. It would probably be more efficient to have the lower level API take a list of userIds. That should help efficiency by letting the database optimize the queries as well as it can. It is in general a good idea to move loops as low as possible in the stack, since it can remove a bunch of overhead.
    – JonasH
    Mar 24 at 10:43
  • 1
    How about async I/O?
    – user253751
    Mar 24 at 20:30
  • @JonasH This would be the optimal solution. The problem is that the low-level API is somewhat "external" and we would need a change request to that party which might not be approved. Therefore we're looking for a way to optimize on our side.
    – das Keks
    Apr 4 at 11:20
  • @user253751 async requests also could be a very good idea, but I'm not familiar with it in a Spring/Java context. I will look into it. If you have any references they are very welcome!
    – das Keks
    Apr 4 at 11:22
  • IMO, a lot of paradigms can work here as long as you maintain the ability to throttle requests and ideally perform exponential backoff as-needed.
    – svidgen
    Apr 17 at 14:26

1 Answer 1

we first have to get a list of all users in the group via REST call
to the low-level API and then 
for each user do two additional requests

Per the comments, you should start by reaching out to the authors of the low level API to see if you can work out a different access pattern:

  • Bulk/batch fetch - where you provide the id's or
  • A search function - if all the desired records can easily be categorized.

If I am the author of the upstream server and I detect someone is doing what you are proposing to do - create a thread pool and hammer me - the first thing I am going to do is put in code to throttle you to X requests per second, so all your multi-threading code goes to waste - because you won't get more throughput than you would have got by calling my API sequentially.


If you are going down the multithreading path anyway, I recommend you read the Hystrix documentation. It gives a very good overview of the challenges of integrating multiple REST services.


The core of the recommendation is that there is a "per-dependency thread pool" which allows you to size the total number of requests that will go to a particular external service.

Note: The Hystrix project itself is now in maintenance only mode - so I am not recommending you integrate it - just leverage their docs to understand the problem.

Non-Blocking IO

Note: It is not actually required (from an OS perspective) to have 1 (or more) threads dedicated to servicing a socket connection. In a nutshell the OS can provide functions such as select()/pselect() which allow a single thread to monitor multiple connections simultaneously. This is functionality that is supported by the JVM:


However despite some googling I have not found any well maintained HTTP client libraries that use this technique. **

** - Every single Java (and Kotlin) library I checked - that claimed to be "async" wasn't actually using Non-Blocking IO - it was just a wrapper to hide a worker thread buried somewhere underneath.

Edit: NetFlux

Per @chris's comment, I have found two HTTP servers, that support both non-blocking IO and Springs WebFlux framework:

  • Undertow (JBoss)
  • Netty

My understanding is that if both the server and client are using WebFlux (and a compatible server), it is likely that the entire communication will be non-blocking.

What is not clear to me, is whether NetFlux can be used / provides Non-Blocking IO when initiating an outbound connection to a REST service that is non WebFlux aware.

Edit2: Ktor with CIO

It appears that if you use the Kotlin Ktor client with the CIO engine (on JVM) that it does use Non-Blocking IO - Not sure how useful that is for pure Java projects.

  • 1
    Every single Java (and Kotlin) library I checked - that claimed to be "async" wasn't actually using Non-Blocking IO => not sure that's quite true. For example, the way I understand Spring WebClient, its behavior depends on whether it's running on Tomcat or Netty, i.e. Servlet or WebFlux stack. On the Servlet stack it will indeed just wrap a worker thread, but on Netty it should be truly non-blocking. Don't quote me on this though, I don't claim expert status on this.
    – chris
    Apr 15 at 10:00
  • I've actually issued a feature request for a bulk API but they are at least hesitant to just implement it. Let's see what further discussion will bring. I was also eager to try out FeignAsync since Feign is the default http client in our project. But after you pointed out that many are just using blocking I/O under the hood I looked into their implementation and disappointingly also found an ExecutorService there.
    – das Keks
    Apr 16 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.