Fastest way to persist an append-only ordered event stream

Question

I'm looking to persist a stream of events and I want it to be:

• fast in both latency and throughput, looking at at least 100k events per second
• ordered (not by timestamp, as multiple events within the same millisecond need to be ordered consistently)
• for events sent by a single thread, they should be persisted in the order they are sent
• replicated / able to fail over in some way

My options seem to be:

1. Append to a file, and send to another server that also appends to a file, waiting for a response before considering it persisted
2. Use a database that considers something persisted once a request has been returned from two nodes

Which is faster? Which are the best databases to use for option 2?

Answer 1

First, instead of of re-inventing the wheel, you should at least consider an existing event streaming/q system, like

• https://kafka.apache.org/documentation/streams/
• or https://www.rabbitmq.com/
• or MongoDB (with its replication feature) directly.

Secondly, if you need to do something like this yourself, and those solutions aren't fast enough (or have some other defect) - you are close to right that the fastest you can do is appending to a file (or more appropriately I would say your solution will likely be disk I/O limited).

The bit about sending over the network CAN be faster or as fast, but not over the long run as that must get written to a file too. The only way to beat that is to split the contents over multiple machines (so no one machine contains ALL the data).

IF you need the redundancy of multiple copies on multiple computers, and you really care about performance, you MIGHT consider some UDP-based (with recovery logic obviously) protocol for sharing data, since TCP can easily slow down your sharing quite a bit.

Note - UDP can be MUCH faster than straight TCP (so therefore faster than HTTP) - because you can build retries and ACKs directly into your protocol. And becuse you can use MULTICAST to greatly decrease your network bandwidth usage.

One SIMPLE to implement thing you could start with is write a 'tee' program, then takes the input stream of events and splits the data among some set of N downstream servers (does no disk IO itself), and then count on those downstream servers to write to disk.

Then the splitter can inject sequence counters providing a total ordering on events.

Then if you want to get fancy, have your protocol ACK successful receipt of events, and when you get enough ACKS (for a given event) - stop resending it. That way, you can probably totally avoid EVERY resending a packet, even if a few get dropped.

And you can use MUTLICAST to send your UDP packets, greatly reducing your network bandwidth.

Then the downstream servers could (depending on your level of tolerance for losing data on crash) - could bundle up hundreds or thousands of messages, compress them, and write them to a single disk file. (Compressing can give you a big performance boost since you are then sending much less data to the disk, and the structure of the event records is often quite compressible).

All together, you can use MUCH less network bandwidth sending your data to helper writer servers than you would have taken sending the straight via TCP(HTTP). And you can get as many copies of each event written to some number of downstream servers. Use compression to limit disk (and possibly also network) IO. And you have a pretty efficient solution.