Bounty Ended with 50 reputation awarded by Nik Kyriakides
2 added 1608 characters in body
source | link

Input stream

Input stream

It is not clear to me if your 1000 events/second represent peaks or if it's a continuous load:

  • if it's a peak, you could use a message queue as buffer to spread the load on the DB server. over a longer time;
  • if it's constant load, the message queue alone is not sufficient, because the DB server will never be able to catch up. Then you'd need to think about a distributed database (for example using a specialization of the server by group of keys).

Proposed solution

Proposed solution

  • All events are systematically published on a kafka topic
  • A consumer would subscribe to the events and store them to the database.
  • A query processor will handle the requests from the clients and query the DB.

This is highly scalable at all levels:

  • If neededDB server is the bottleneck, just add several consumerconsumers. Each could subscribe to the topic, each writing inand write to a different DB server. However, if the distribution occurs randomly across the DB servers, the query processor will not be able to predict the DB server to take and have to query several DB servers. This could lead to a new bottleneck on the query side. 
  • The DB distribution scheme could even alreadytherefore be anticipated by organising the event stream into several topics (for example, using groups of keys or properties, as previously mentionedto partition the DB according to a predictable logic).
  • If one message server is not sufficient to handle a growing flood of input loadevents, you could even add kafka partitions ofto distribute kafka topics across several physical servers.

So this architecture would be highly scalable at all levels.

Offering events not yet written in the DB to clients

Offering events not yet written in the DB to clients

Option 1: Using a cache to complement db queries

I have not analysed in depth, but the first idea that comes to my mind would be to make the requestquery processor(s) a consumer(s) of the kafka topics, but in a different kafka consumer group. The request processor would then receive all the messages independently ofthat the DB writer will receive, andbut independently. It could then keep them in a local cache. The queries would then run on DB + cache (+ elimination of duplicates).

But aThe design would then look like:

enter image description here

The scalability of this query layer could be achieved by adding more query processors (each in its own consumer group).

Option 2: design a dual API

A better approach IMHO would be to offer a dual API (use the mechanism of the separate consumer group):

The advantage, is that you let the client has to decide what it'sis interesting. This could avoid that you systematically merge DB data with freshly cashed data, when the client is only interested in new incoming events. If the delicate merge between fresh and archived events is really needed, then the client haswould have to organise it.

Variants

I proposed kafka because it's designed for very high volumes with persistent messages so that you can restart the servers if needed.

You could build a similar architecture with RabbitMQ. However if you need persistent queues, it might decrease performance. Also, as far as I know, the only way to achieve the parallel consumption of the same messages by several readers (e.g. writer+cache) with RabbitMQ is to clone the queues. So a higher scalability might come at a higher price.

Input stream

It is not clear to me if your 1000 events/second represent peaks or if it's a continuous load:

  • if it's a peak, you could use a message queue as buffer to spread the load on the DB server.
  • if it's constant load, the message queue alone is not sufficient, because the DB server will never be able to catch up. Then you'd need to think about a distributed database (for example using a specialization of the server by group of keys).

Proposed solution

  • All events are systematically published on a kafka topic
  • A consumer would subscribe to the events and store them to the database.
  • If needed, several consumer could subscribe to the topic, each writing in a different DB servers. The distribution scheme could even already be anticipated by organising the event stream into several topics (for example, using groups of keys, as previously mentioned).
  • If one server is not sufficient to handle a growing input load, you could even add kafka partitions of topics across several servers.

So this architecture would be highly scalable at all levels.

Offering events not yet written in the DB to clients

I have not analysed in depth, but the first idea that comes to my mind would be to make the request processor(s) a consumer(s) of the kafka topics, but in a different kafka consumer group. The request processor would then receive all the messages independently of the DB writer, and could keep them in a local cache. The queries would then run on DB + cache (+ elimination of duplicates).

But a better approach IMHO would be to offer a dual API (use the mechanism of the separate consumer group):

The client has to decide what it's interested in. If the delicate merge between fresh and archived events is really needed, then the client has to organise it.

Input stream

It is not clear if your 1000 events/second represent peaks or if it's a continuous load:

  • if it's a peak, you could use a message queue as buffer to spread the load on the DB server over a longer time;
  • if it's constant load, the message queue alone is not sufficient, because the DB server will never be able to catch up. Then you'd need to think about a distributed database.

Proposed solution

  • All events are systematically published on a kafka topic
  • A consumer would subscribe to the events and store them to the database.
  • A query processor will handle the requests from the clients and query the DB.

This is highly scalable at all levels:

  • If DB server is the bottleneck, just add several consumers. Each could subscribe to the topic, and write to a different DB server. However, if the distribution occurs randomly across the DB servers, the query processor will not be able to predict the DB server to take and have to query several DB servers. This could lead to a new bottleneck on the query side. 
  • The DB distribution scheme could therefore be anticipated by organising the event stream into several topics (for example, using groups of keys or properties, to partition the DB according to a predictable logic).
  • If one message server is not sufficient to handle a growing flood of input events, you could add kafka partitions to distribute kafka topics across several physical servers.

Offering events not yet written in the DB to clients

Option 1: Using a cache to complement db queries

I have not analysed in depth, but the first idea that comes to my mind would be to make the query processor(s) a consumer(s) of the kafka topics, but in a different kafka consumer group. The request processor would then receive all the messages that the DB writer will receive, but independently. It could then keep them in a local cache. The queries would then run on DB + cache (+ elimination of duplicates).

The design would then look like:

enter image description here

The scalability of this query layer could be achieved by adding more query processors (each in its own consumer group).

Option 2: design a dual API

A better approach IMHO would be to offer a dual API (use the mechanism of the separate consumer group):

The advantage, is that you let the client decide what is interesting. This could avoid that you systematically merge DB data with freshly cashed data, when the client is only interested in new incoming events. If the delicate merge between fresh and archived events is really needed, then the client would have to organise it.

Variants

I proposed kafka because it's designed for very high volumes with persistent messages so that you can restart the servers if needed.

You could build a similar architecture with RabbitMQ. However if you need persistent queues, it might decrease performance. Also, as far as I know, the only way to achieve the parallel consumption of the same messages by several readers (e.g. writer+cache) with RabbitMQ is to clone the queues. So a higher scalability might come at a higher price.

1
source | link

Input stream

It is not clear to me if your 1000 events/second represent peaks or if it's a continuous load:

  • if it's a peak, you could use a message queue as buffer to spread the load on the DB server.
  • if it's constant load, the message queue alone is not sufficient, because the DB server will never be able to catch up. Then you'd need to think about a distributed database (for example using a specialization of the server by group of keys).

Proposed solution

Intuitively, in both cases, I'd go for a Kafka based event-stream:

  • All events are systematically published on a kafka topic
  • A consumer would subscribe to the events and store them to the database.
  • If needed, several consumer could subscribe to the topic, each writing in a different DB servers. The distribution scheme could even already be anticipated by organising the event stream into several topics (for example, using groups of keys, as previously mentioned).
  • If one server is not sufficient to handle a growing input load, you could even add kafka partitions of topics across several servers.

So this architecture would be highly scalable at all levels.

Offering events not yet written in the DB to clients

You want your clients to be able to get access also to information still in the pipe and not yet written to the DB. This is a little more delicate.

I have not analysed in depth, but the first idea that comes to my mind would be to make the request processor(s) a consumer(s) of the kafka topics, but in a different kafka consumer group. The request processor would then receive all the messages independently of the DB writer, and could keep them in a local cache. The queries would then run on DB + cache (+ elimination of duplicates).

But a better approach IMHO would be to offer a dual API (use the mechanism of the separate consumer group):

  • a query API for accessing events in the DB and/or making analytics
  • a streaming API that just forwards messages directly from the topic

The client has to decide what it's interested in. If the delicate merge between fresh and archived events is really needed, then the client has to organise it.