Multithreaded processing of single REST requests

Question

Background: We're providing an API that provides information about all users within a given group. Our API is a high level facade over another low-level REST API.

To gather those information we first have to get a list of all users in the group via REST call to the low-level API and then for each user do two additional requests to get the required information.

Our API is a Spring Boot application running in kubernetes.

Problem: The response time of our API increases linearly with the number of users in the group since we have to do many consecutive requests to the low-level API.

We could parallelize the requests for individual users to the low-level API to speed up the process.
However that's where I'm not sure if it's a good idea or how to handle the threads used for that. It feels wrong two spin up additional threads for each incoming request.

Our application uses the default web thread pool size server.tomcat.threads.max of 200.

Ideas:

• Use a new thread pool with e.g. 10 threads to do the additional processing for each request
  • Large overhead for starting threads
  • Could lead up to 2000 threads if all webserver threads are in use
• Use Stream.parallel()
  • Easy to implement
  • Number of threads depends on the CPUs of the node
  • All requests share the same low number of threads
• Create a shared thread pool with e.g. 200 threads that can be used by all requests
  • We have to be careful with resource management and returning threads to the pool
  • Good control over maximal load and good speedup

Is there any general advice how to handle multiple threads for individual requests?

Answer 1

we first have to get a list of all users in the group via REST call
to the low-level API and then 
for each user do two additional requests

Per the comments, you should start by reaching out to the authors of the low level API to see if you can work out a different access pattern:

• Bulk/batch fetch - where you provide the id's or
• A search function - if all the desired records can easily be categorized.

If I am the author of the upstream server and I detect someone is doing what you are proposing to do - create a thread pool and hammer me - the first thing I am going to do is put in code to throttle you to X requests per second, so all your multi-threading code goes to waste - because you won't get more throughput than you would have got by calling my API sequentially.

---

Multithreading

If you are going down the multithreading path anyway, I recommend you read the Hystrix documentation. It gives a very good overview of the challenges of integrating multiple REST services.

https://github.com/Netflix/Hystrix/wiki/How-it-Works#isolation

The core of the recommendation is that there is a "per-dependency thread pool" which allows you to size the total number of requests that will go to a particular external service.

Note: The Hystrix project itself is now in maintenance only mode - so I am not recommending you integrate it - just leverage their docs to understand the problem.

---

Non-Blocking IO

Note: It is not actually required (from an OS perspective) to have 1 (or more) threads dedicated to servicing a socket connection. In a nutshell the OS can provide functions such as select()/pselect() which allow a single thread to monitor multiple connections simultaneously. This is functionality that is supported by the JVM:

https://docs.oracle.com/javase/7/docs/api/java/nio/channels/Selector.html

However despite some googling I have not found any well maintained HTTP client libraries that use this technique. **

** - Every single Java (and Kotlin) library I checked - that claimed to be "async" wasn't actually using Non-Blocking IO - it was just a wrapper to hide a worker thread buried somewhere underneath.

Edit: NetFlux

Per @chris's comment, I have found two HTTP servers, that support both non-blocking IO and Springs WebFlux framework:

• Undertow (JBoss)
• Netty

My understanding is that if both the server and client are using WebFlux (and a compatible server), it is likely that the entire communication will be non-blocking.

What is not clear to me, is whether NetFlux can be used / provides Non-Blocking IO when initiating an outbound connection to a REST service that is non WebFlux aware.

Edit2: Ktor with CIO

It appears that if you use the Kotlin Ktor client with the CIO engine (on JVM) that it does use Non-Blocking IO - Not sure how useful that is for pure Java projects.