1

Background: We're providing an API that provides information about all users within a given group. Our API is a high level facade over another low-level REST API.

To gather those information we first have to get a list of all users in the group via REST call to the low-level API and then for each user do two additional requests to get the required information.

Our API is a Spring Boot application running in kubernetes.

Problem: The response time of our API increases linearly with the number of users in the group since we have to do many consecutive requests to the low-level API.

We could parallelize the requests for individual users to the low-level API to speed up the process.
However that's where I'm not sure if it's a good idea or how to handle the threads used for that. It feels wrong two spin up additional threads for each incoming request.

Our application uses the default web thread pool size server.tomcat.threads.max of 200.

Ideas:

  • Use a new thread pool with e.g. 10 threads to do the additional processing for each request
    • Large overhead for starting threads
    • Could lead up to 2000 threads if all webserver threads are in use
  • Use Stream.parallel()
    • Easy to implement
    • Number of threads depends on the CPUs of the node
    • All requests share the same low number of threads
  • Create a shared thread pool with e.g. 200 threads that can be used by all requests
    • We have to be careful with resource management and returning threads to the pool
    • Good control over maximal load and good speedup

Is there any general advice how to handle multiple threads for individual requests?

5
  • 5
    It might help a bit, but you run the risk of having all the request contend for the same database resources/locks. It would probably be more efficient to have the lower level API take a list of userIds. That should help efficiency by letting the database optimize the queries as well as it can. It is in general a good idea to move loops as low as possible in the stack, since it can remove a bunch of overhead.
    – JonasH
    Commented Mar 24, 2023 at 10:43
  • 1
    How about async I/O? Commented Mar 24, 2023 at 20:30
  • @JonasH This would be the optimal solution. The problem is that the low-level API is somewhat "external" and we would need a change request to that party which might not be approved. Therefore we're looking for a way to optimize on our side.
    – das Keks
    Commented Apr 4, 2023 at 11:20
  • @user253751 async requests also could be a very good idea, but I'm not familiar with it in a Spring/Java context. I will look into it. If you have any references they are very welcome!
    – das Keks
    Commented Apr 4, 2023 at 11:22
  • IMO, a lot of paradigms can work here as long as you maintain the ability to throttle requests and ideally perform exponential backoff as-needed.
    – svidgen
    Commented Apr 17, 2023 at 14:26

4 Answers 4

4
we first have to get a list of all users in the group via REST call
to the low-level API and then 
for each user do two additional requests

Per the comments, you should start by reaching out to the authors of the low level API to see if you can work out a different access pattern:

  • Bulk/batch fetch - where you provide the id's or
  • A search function - if all the desired records can easily be categorized.

If I am the author of the upstream server and I detect someone is doing what you are proposing to do - create a thread pool and hammer me - the first thing I am going to do is put in code to throttle you to X requests per second, so all your multi-threading code goes to waste - because you won't get more throughput than you would have got by calling my API sequentially.


Multithreading

If you are going down the multithreading path anyway, I recommend you read the Hystrix documentation. It gives a very good overview of the challenges of integrating multiple REST services.

https://github.com/Netflix/Hystrix/wiki/How-it-Works#isolation

The core of the recommendation is that there is a "per-dependency thread pool" which allows you to size the total number of requests that will go to a particular external service.

Note: The Hystrix project itself is now in maintenance only mode - so I am not recommending you integrate it - just leverage their docs to understand the problem.


Non-Blocking IO

Note: It is not actually required (from an OS perspective) to have 1 (or more) threads dedicated to servicing a socket connection. In a nutshell the OS can provide functions such as select()/pselect() which allow a single thread to monitor multiple connections simultaneously. This is functionality that is supported by the JVM:

https://docs.oracle.com/javase/7/docs/api/java/nio/channels/Selector.html

However despite some googling I have not found any well maintained HTTP client libraries that use this technique. **

** - Every single Java (and Kotlin) library I checked - that claimed to be "async" wasn't actually using Non-Blocking IO - it was just a wrapper to hide a worker thread buried somewhere underneath.

Edit: NetFlux

Per @chris's comment, I have found two HTTP servers, that support both non-blocking IO and Springs WebFlux framework:

  • Undertow (JBoss)
  • Netty

My understanding is that if both the server and client are using WebFlux (and a compatible server), it is likely that the entire communication will be non-blocking.

What is not clear to me, is whether NetFlux can be used / provides Non-Blocking IO when initiating an outbound connection to a REST service that is non WebFlux aware.

Edit2: Ktor with CIO

It appears that if you use the Kotlin Ktor client with the CIO engine (on JVM) that it does use Non-Blocking IO - Not sure how useful that is for pure Java projects.

2
  • 1
    Every single Java (and Kotlin) library I checked - that claimed to be "async" wasn't actually using Non-Blocking IO => not sure that's quite true. For example, the way I understand Spring WebClient, its behavior depends on whether it's running on Tomcat or Netty, i.e. Servlet or WebFlux stack. On the Servlet stack it will indeed just wrap a worker thread, but on Netty it should be truly non-blocking. Don't quote me on this though, I don't claim expert status on this.
    – chris
    Commented Apr 15, 2023 at 10:00
  • I've actually issued a feature request for a bulk API but they are at least hesitant to just implement it. Let's see what further discussion will bring. I was also eager to try out FeignAsync since Feign is the default http client in our project. But after you pointed out that many are just using blocking I/O under the hood I looked into their implementation and disappointingly also found an ExecutorService there.
    – das Keks
    Commented Apr 16, 2023 at 22:21
2

It feels wrong two spin up additional threads for each incoming request.

Yes, it is wrong. Not only inefficient, but also a security concern: you open your server to hard DOS attacks. And it doesn't even have to be malicious user, you will auto-DOS yourself during high traffic.

Use a new thread pool

Per request? Same as above.

Use Stream.parallel()

Unfortunately I don't know how that works (I'm not a Java programmer). But if it spawns a new threads/thread pool per request, then it is similar to my previous comments. If it uses a shared thread pool, then:

Create a shared thread pool

Yes, this is acceptable, especially when coupled with a bounded queue/channel. It would be better to use some async framework here, but that is only an optimization.

That solution handles two cases: if you have low traffic, then it will nicely increase performance of each request. But if you have high traffic, then it pretty much falls back to sequential execution. This is necessary for the server to keep running and be responsive. And then you can fine tune the number of threads so that it performs best. But never allow external actors to consume as many resources of your server as they want.

0
0

You can use parallel stream.Simple yet efficient approach for the use case. However,you need to have rough estimates of how many max users you are expecting.If they are huge, opt for completablefuture where you can customize threadpool size.

0

By now virtual threads are available. Assuming the other service can handle the traffic and you clarified that a bulk request endpoint is not possible / won't be added on the lower level service then the simplest and arguably most efficient approach by now is to use virtual threads. Simply start a virtual thread for each call collecting information for a user. Thus all those calls run in parallel, you don't need a separate thread pool and this is much more light weight than using thread pools.

Note: If the other service cannot handle the parallelism or only to a certain point you might need to add a mechanism that limits the calls made in parallel. A bulkhead could do - if it supports queueing.

But I agree with DavidT, that the best option would be to have a bulk retrieval endpoint at the other service.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.