3

Assume I'm writing a library in Java for performing network calls. I want it to be efficient for cases like highly loaded reverse proxies etc (10s to 100s Krps coming in and out; client requests served in the same DC in under 1 ms; relatively simple processing of results), also, each incoming request might start several outgoing ones and combine results in a simple way.

The library uses non-blocking client sockets internally, but I want to expose some higher-level abstraction to the users of this library. I've started with CompletableFuture, but I'm feeling like it might generate too much garbage: each "composition" (thenCompose etc) creates a new object; also, it encourages using lambdas which are also separate objects (right?). The final future might consist of several stages (3-5 really: gathers, retries etc), so that feels like a bit too many objects per request.

What other options do I have that would provide at least some "composability" (read: can be used by library clients, and maybe by the library itself) but will generate less garbage?

Is it possible to have a "reusable" event loop of sorts? Or can the stream (reactive) paradigm help? Good things are said about coroutines which are equated to state machines for languages that don't support them natively, but I'm having trouble imagining how implementing that might even look like for a scenario like "call several services with retry, when done with some/all, return back a result, give means for the library clients to tap into that or enhance that"... We might have a number of processors connected by queues (blocking queue in the simplest form), but I have doubts about the efficiency of such system: these queues don't come for free, there will be elements added, elements chopped, all these will migrate through the whole "pipeline"... Or maybe Disruptor is my only hope? (Not that I understand how to apply it here.)

For these approaches pseudocode or a scheme describing how it might work or a link to an explanation is highly welcomed because I sort of know "about" them but don't understand how to use them or if they'll work in my case. Or maybe there's a still better approach that I completely missed?..

Edit: I've forgotten to mention that I'm actually extending an existing networking library to support asynchronous usage. The library reuses objects very heavily (buffers, sockets, response objects etc.), so, first, I thought maybe it was really important in this context; second, I'd like to pursue the same design goals if possible.

Some back of the envelope calculations supporting this desire: suppose there are 4 futures per request (the original one, retrying logic, gathering several results logic, and then client logic). Suppose there are 50K server requests coming per second and each translates to 2 client requests. So that's already 4 * 2 * 50_000 * (object_size) bytes per second assuming that callbacks are reused. Assume each object takes 20 bytes (header, at least one listener, result reference). Assume callbacks are reused so there are none created and collected. That's 8Mb per second, but easily more. That's 1 Gb of garbage per couple of minutes. Maybe not too much, really, but still would be interesting to understand if that could be improved.

Improve this question
2
  • 1
    Object Pooling. Related: stackoverflow.com/q/21762826
    – Robert Harvey
    Jul 19, 2017 at 0:21
  • 1
    Be careful. Unless you have really, truly, measured this using rigorous statistic methods, and repeatable, representative benchmarks, outguessing the garbage collector will often be wrong. Object Pooling negates both parts of the Generational Hypothesis (Objects Die Young and Older Objects Don't Reference Younger Objects) upon which most modern high-performance GCs are built. You have to be very sure that you can buy back this disadvantage. In general, both allocation and collection of objects in the nursery is as fast as stack allocation in C, and faster than malloc / free.
    – Jörg W Mittag
    Jul 19, 2017 at 20:00

1 Answer 1

Reset to default
4

I've started with CompletableFuture, but I'm feeling like it might generate too much garbage:

Exactly where is this 'feeling' coming from? Have you tested this?

each "composition" (thenCompose etc) creates a new object;

Object creation is just pointer addition, and sometimes initialization. The expense of both is highly dependent on the design of the object. This is a big reason why constructors that do real work are frowned on.

also, it encourages using lambdas which are also separate objects (right?).

Right, but again not all objects are created equal. For that matter, not all JVM's are created equal. You might want to look at the Java® Language Specification, chapter 15.27.4. Run-time Evaluation of Lambda Expressions

Particularly this little nugget:

  • A new object need not be allocated on every evaluation.

So some JVM's might optimize away your pile of redundant lambda objects. Some might not.

The final future might consist of several stages (3-5 really: gathers, retries etc), so that feels like a bit too many objects per request.

Really? So if I did it all with just one 8 gigabyte object that we build and destroy for every request you'd be fine with that? Resources are limited but object counting isn't the best way to manage them.

What I'm trying to show you is that it's really impossible to predict how this design will perform until you test it.

Now sure, you could go for 100% statically allocated objects and KNOW you aren't going to blow the heap but that's just trading one problem for another.

Improve this answer
4
  • "Exactly where is this 'feeling' coming from? Have you tested this?" It comes from the fact that the underlying library I'm using reuses everything from sockets to byte buffers for requests and then even has an option to reuse resulting unmarshalled request objects, I thought they were doing it for a reason. Perhaps I should really test it first, but then -- I thought maybe it's a well-know problem with established solutions.
    – starikoff
    Jul 19, 2017 at 5:27
  • Actually I'm not only using this library but mainly extending it to support asynchrony, so another thing I'm striving for is matching its low garbage generation rate because it looks like an explicit design decision.
    – starikoff
    Jul 19, 2017 at 6:33
  • Thanks for the JLS link though, the no-real-work constructor reasoning and your whole answer overall!
    – starikoff
    Jul 19, 2017 at 6:35
  • It's a well known problem with established hedging. Whether those optional optimizations are needed very much depends on how you code and what your deployed system is subjected to. The best advice I have for you now is write readable code. Surprise problems are going to happen no mater how much you research. Be sure when they do diving back into the code to fix them isn't going to be wading through a nightmare of hard to read premature optimizations.
    – candied_orange
    Jul 19, 2017 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.