The scenario is:

  1. client makes request to server A
  2. Server A makes potentially multiple requests to server B. Edit to clarify, server A makes the requests concurrently using Futures.
  3. Server A blocks until all results return
  4. Server A collates responses into single response and returns to client

So currently the approach is the block using java Futures and return the results when they come in. My concern is this is quite thread intensive and I'm worried we will run out of resources on the server. I was thinking of going async in server A by using DeferredResult but since there is a thread for each request to server B, that doesn't seem to carry much benefit.

Alternatively, we could do

  1. Server A makes multiple requests to Server B
  2. Server B immediate returns a token for each request
  3. Server A tracks the tokens and returns DeferredResult
  4. Server B gets each result, puts it in memcache, sends a JMS message to server A. Edit: I just realized I could send the result in the JMS message, so no need for memcache at all.
  5. Server A can complete the DeferredResult after it collects all the responses from JMS.

This has the advantage of being much much simpler in code (no complicated futures work) at the cost of being more complicated architecturally. It also introduces more layers (JMS) for failure. It also has the advantage of being much less resource intensive.

This is a high use system. Calls to system B can take anywhere from 1 to several seconds.

  • is this a java specify problem? .net wouldnt use threads for that – Ewan Aug 8 at 14:40
  • Is it possible to make the client responsible for choosing whether to block or not? Yes you'd have to implement both, but the block / no block gives your client the flexibility to arrange its processing, and with a reasonable distribution of requests, that might address your resource concerns. That is, of course, a bigger API change, but would be backward compatible and (possibly) more client friendly. – Kristian H Aug 8 at 14:42
  • @ewan yes this is Java – hvgotcodes Aug 8 at 14:50
up vote 0 down vote accepted

I think your original plan sounds better than the 'alternative' you described. KISS. Your current approach is very simple.

Structurally what you are doing has no need to be thread intensive (no more than one thread to service the call). You can invoke many calls to Service B just by invoking async tasks. Maybe that's already how the Java Future<> implementation works (not sure - would have to look). other stack overflow article appears to indicate it works that way. Otherwise, you can use another java mechanism to invoke async web requests.

If your web service engine in server A is written to specially handle DeferredResult objects (I doubt this but you can check) - then your plan to use DeferredResult would work well. But most likely, your webservice just will force a GET call on the result of the websevice call, and will block the webservice call handling thread while the work completes whereever it comes from.

You did hint at the possibility of another strategy. Depending on the nature of your calls from ServerA to ServerB, perhaps those could be sped up or cached. Using something like redis/memchached (or just an in memory cache inside your ServerA) MIGHT be an effective way to make your overall throughput much faster, but that depends ENTIRELY on the cachability of calls from ServerA to ServerB (which you said nothing about). If you want to experiement with this caching approach, use a HashMap, and some abstraction to count hits/misses, to see how effective the approach is, before trying to scale up to something more complex like memcached.

  • Thank you for the reply. I'm confused by your answer, can you clarify with serverA and serverB? How would DeferredResults help if a request to A causes N (Say 5) other threads to be spun up to service requests to B? It would eliminate stalling the thread servicing the request, but that thread creates N more so.....Regarding KISS, the threading code is very complicated, so moving to messaging and memcache to exchange data between A and B would be much simpler code wise (in exchange for more dependencies) – hvgotcodes Aug 8 at 14:54
  • @hvgotcodes I'm not sure using futures actually spins up extra threads in java. There is no need to (at least not in this case where there is a long network call and tiny CPU activity before and after). – Lewis Pringle Aug 8 at 18:08
  • @hvgotcodes IF your webserer package supported DeferredResults well, it could stop add a competion proc to the defered result, and then give up the CPU (return - leaving the thread that was handling the web request). Again - not sure what webserver you are using, so no idea if yours does this or not. – Lewis Pringle Aug 8 at 18:10
  • @hvgotcodes Recarding the KISS comments, the threading code need not be very complicated. And doing it right can make a big latency/performance difference. Using memcached or redis CAN be a good idea, but its largely orthoganal, and it depends ENTIRELY on how usefully cachable the computations are (if you never get the same request twice or the results cannot be cached because they depend on hidden system state variables you cannot cache). But using caching CAN be a good idea (and I suggested a QUICK way to tell if/how much it helps - use HashMap to test). – Lewis Pringle Aug 8 at 18:14
  • this is SpringBoot, so DeferredResult support is excellent. But we still have the problem that any incoming request spawns, say, 5, threads all hitting server b independently. Using the DeferredResult on the containers request-servicing thread returns 1 to the pool, but there are still 5 outstanding. Is DeferredResult worth it? – hvgotcodes Aug 8 at 18:45

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.