I'm looking to persist a stream of events and I want it to be:

  • fast in both latency and throughput, looking at at least 100k events per second
  • ordered (not by timestamp, as multiple events within the same millisecond need to be ordered consistently)
  • for events sent by a single thread, they should be persisted in the order they are sent
  • replicated / able to fail over in some way

My options seem to be:

  1. Append to a file, and send to another server that also appends to a file, waiting for a response before considering it persisted
  2. Use a database that considers something persisted once a request has been returned from two nodes

Which is faster? Which are the best databases to use for option 2?

  • 100K event per second is going to take more engineering than I assume you'll have time for. I would go with any one of top 3 streaming event solutions depending on your hosting preferences: self hosted Kafka, AWS hosted Kinesis, or Azure hosted Event Hub. – Brad Irby Jul 28 '18 at 10:25

First, instead of of re-inventing the wheel, you should at least consider an existing event streaming/q system, like

Secondly, if you need to do something like this yourself, and those solutions aren't fast enough (or have some other defect) - you are close to right that the fastest you can do is appending to a file (or more appropriately I would say your solution will likely be disk I/O limited).

The bit about sending over the network CAN be faster or as fast, but not over the long run as that must get written to a file too. The only way to beat that is to split the contents over multiple machines (so no one machine contains ALL the data).

IF you need the redundancy of multiple copies on multiple computers, and you really care about performance, you MIGHT consider some UDP-based (with recovery logic obviously) protocol for sharing data, since TCP can easily slow down your sharing quite a bit.

Note - UDP can be MUCH faster than straight TCP (so therefore faster than HTTP) - because you can build retries and ACKs directly into your protocol. And becuse you can use MULTICAST to greatly decrease your network bandwidth usage.

One SIMPLE to implement thing you could start with is write a 'tee' program, then takes the input stream of events and splits the data among some set of N downstream servers (does no disk IO itself), and then count on those downstream servers to write to disk.

Then the splitter can inject sequence counters providing a total ordering on events.

Then if you want to get fancy, have your protocol ACK successful receipt of events, and when you get enough ACKS (for a given event) - stop resending it. That way, you can probably totally avoid EVERY resending a packet, even if a few get dropped.

And you can use MUTLICAST to send your UDP packets, greatly reducing your network bandwidth.

Then the downstream servers could (depending on your level of tolerance for losing data on crash) - could bundle up hundreds or thousands of messages, compress them, and write them to a single disk file. (Compressing can give you a big performance boost since you are then sending much less data to the disk, and the structure of the event records is often quite compressible).

All together, you can use MUCH less network bandwidth sending your data to helper writer servers than you would have taken sending the straight via TCP(HTTP). And you can get as many copies of each event written to some number of downstream servers. Use compression to limit disk (and possibly also network) IO. And you have a pretty efficient solution.

  • UDP + recovery logic = TCP. Are you, maybe, advocating recovery logic that is somehow specialized to this particular problem? If so, how? – candied_orange Jul 26 '18 at 0:40
  • Not quite so (about UDP+revoverylogic=TCP). TCP also guarantees order of delivery. This is a VERY costly thing to guarantee, and is not strictly needed for this application (sure it helps). That is ESPECIALLY true if you only care that each record gets written to a certain number of machines (not all). The recievers ACK, and if you get enough acks, and someone drops the packet, no need to resend. AND - you can send the data via MULTICAST (potentially a HUGE savings). So - no - not an equality relation. – Lewis Pringle Jul 26 '18 at 0:44
  • Awesome, please explain that in the answer. – candied_orange Jul 26 '18 at 0:46
  • OK - I tried to update my answer to take into account some of that. I am new to this site, and I wasn't sure how materially I was permitted to change my answer (wtihout just posting another answer). Hope that was OK... – Lewis Pringle Jul 26 '18 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.