First, it is important to understand and be able to leverage the difference between Commands and Events.
As this question succinctly points out, Commands are things we would like to happen, and Events are things that have already happened. A command does not necessarily result in a significant event in the system, but it usually does. For example, a send ...
A streaming app is an app that consumes a stream of data.
A stream of data is transmitted data formatted in a way that can be useful even when incomplete. Since partial stream data does not require complete transmission this allows consumers to join and leave at any time. It also allows for transmission to be continuous, though it may start and stop on ...
Kafka deals in ordered logs of atomic messages. You can view it sort of like the pub/sub mode of message brokers, but with strict ordering and the ability to replay or seek around the stream of messages at any point in the past that's still being retained on disk (which could be forever).
Kafka's flavor of streaming stands opposed to remote procedure call ...
Kafka/Kinesis is modelled as a stream. A stream has different properties than messages.
Streams have context to them. They have order. You can apply window functions on streams. Although each item in a stream is meaningful, it may be more meaningful with the context around it
Because streams have order, you can use that to make certain statements about the ...
Ideally, you don’t. This sort of approach is fraught with partial failure scenarios. What happens if B and C process, but D dies irrecocably? What happens when you get duplicate data from any of the workers?
In general, you get a new service E which listens for results from BCD, writing the partial results to its own db. Potentially it could be a part of A -...
I haven't done extensive unit testing in java script. But the principles remain fairly same in all the languages.
Properties of a good unit test would be
Runs quickly and provides nearly immediate feedback
Doesn't have side effects, like charging a credit card, or modifying a database or event publishing an event
What needs to be tested in your case
Commands and queries will be persisted on S3 for purpose of auditing and restoring.
For auditing, sure. For restoring ? That's weird, and likely to cause you headaches.
If you are going to be event sourcing, you want to be rehydrating state from events (things that happened in the past) not commands. This saves you from most of the problems associated ...
With an 'E'
B, C, and D each publish their own completion event. E subscribes to those new events, collecting information from each and storing its state until it recognizes that it has received all the events, potentially in any order. Once it has received them all, it completes its own processing with all the information it needs, possibly publishing its ...
Kafka and redis are not used for the same purpose. Redis is a key-value storage engine and Kafka is "streaming platform". I have never used Kafka but it seems to act like a message broker optimized for streaming
We have come across a similar demand at the company I work for. We had to mark seats as occupied on a seating plan along with people having their ...
I would suggest something along the following lines.
Scraper: scrapes the data and published to Kafka
Formatter/Persistence: Reads from Kafka, sends data to the storage layer
Storage: 1 "real" database where you performs writes. Replicate this db to as many read only copies as you need.
API: Accesses only the read-only replicas to serve the data.
The producer response protocol provides the offset of the first message sent in the messageset. Provided the API/Implementation you're using exposes this information, you should be able to retrieve it from the response to producing your message.
You shouldn't do that from a class that acts as a producer, like in most queue systems, as its responsibility is to just fire and forget the messages. The broker will do the appropriate meta data handling with id's, offsets, whatever it needs to handle the messages in a correct manner.
You can get the offsets as a consumer of messages from a Kafka broker. ...
Spark programs are just java, scala or python code, so they can write data to all the same places any program can write them. In fact, spark does not actually do anything unless you write the end result somewhere with an output operation.
If the end result of a spark job is small, it can be written to a relational database or a web service or something of ...
I would say yes, they both implement asynchronous messaging. The way I look at Kafka is as a improvement to the old JMS/MQ model where queues and topics are unified into a single entity (called a topic.) This is an improvement because the decision of whether something was a queue or a topic had to be made very early on and there were major trade-offs in the ...
I believe Kafka has the notion of "Partitions". If you put messages in the same partition they will be processed in order by a single processor
"By having a notion of parallelism—the partition—within the topics,
Kafka is able to provide both ordering guarantees and load balancing
over a pool of consumer processes. This is achieved by assigning the
The very point of asynchronous systems such as Kafka is that you gain performance in exchange for some kind of guarantees - in your case the guarantee for in-order processing. Forcing one event to wait for another pretty much removes the benefits Kafka offers to you - if you want to go that way, better use a queueing system instead of an asynchronous one ...
Here is a resume of what you said from my understanding (please validate):
You have N microservices, where N ~= 10 (could grow). Let's call them consumers
Each microservice will require data
The data required is stored 1 database
Each microservice requires this data differently
In order to support these requirements, you propose that a specific ...
You're right in general about throughput that you would get with a low-level message broker solution vs an iPaas - however you should consider whether you really need that throughput. Some iPaas solutions are quite fast and may be 'fast enough' for your use case.
We deployed an integration platform successfully to connect real-time systems (IoT/control ...
I think that Kafka is exactly the wrong choice in this situation.
Kafka works best when a topic is read by consumers that are able to take a chunk of records, process them, and record the offsets.
In your case, you have consumers that will process single messages. And while Kafka can do this, you're paying a high IO penalty relative to throughput.
architecture that somehow partitions posts per geo area (which I want
Unfortunately that is exactly what you should be doing.
First of all, drop kafka. Its not a message queue.
Secondly, when a user posts you want to tag that post with one or more area aggregate keys; say post code or grid square or city rather (or as well as) the geo-location....
Yes, that is how I interpret that. Data stored in data warehouses is imported from primary sources. The effort required to bring in data is non-trivial so therefore much of the source data is not available in the data warehouse. Typically this means that you might have every address of every customer but you don't have any birth dates, for example. It's ...
Should the API directly consume the Kinesis stream and relay the
messages to the client? Or is it better to have a middle layer that
sits between the Kinesis stream and the API client to manage some
The answer to this hinges on the answers to the other questions.
What is the recommended approach to manage replay? Suppose a client
I am not sure why are there docker and kubernetes in your question. It's on you how you want to setup docker to accommodate your architecture and shouldn't play a role in architectural decisions.
Yes you can always span multiple graphql servers on your application. It's up to you how you want to handle requests coming on these multiple graphql servers. I ...
We have decided to use the architecture similar to the architecture which proposed Todd Palino (Staff Site Reliability Engineer from LinkedIn). I.e. the architecture they are using in LinkedIn.
We have decided to use a unique topic for each server. Topic name has to have a data center identifier, and a server identifier (in that data center).
So, for ...
The third is not valid; all consumers on a topic get all messages. Partitions are used to spread load across multiple consumer instances (same group) and to maintain message order for specific keys.
For point-to-point messaging you need a separate topic for each app.
With JMS you can use message selectors to select messages you are interested in (as long ...
In a Kafka streaming application, the producer:
Publishes messages into the void.
Cannot expect an immediate reply from an intended destination (that would be RPC).
May not even have an intended destination (think logs).
May be given assurances that the messages are durably stored by Kafka and will not be lost.
May be given assurances that the messages are ...
If you already have Kafka, you could define another topic, that contains only URLs that need to be crawled.
Crawlers listen on this topic, poll the next URL to crawl. Crawlers should share a group id, so that each message is delivered only to a single crawler. When the crawlers detect new URLs, they don't crawl recursively, but put the new URLs into the URL ...
Think of your situation as data and processes that depend on it. What results is this:
Service A creates entity a
Service B creates entity b
Service D performs a process d which depends on a and b
The decision on when d should happen is made by service C
D has to do its duty when C tells it to, but C does not provide all dependencies for that process.
gRPC and queues|brokers are not mutually exclusive. It's not like we have to choose one or another. We might choose one, both or none. We could also use an SMTP server as a queue, or a plain text file.
They can complement each other since they satisfy different purposes.
Think in a pipeline where each stage is a process (service).
gRPC can do asynchronous requests, but it is still an RPC pattern and so that is mostly for when you don't care about the answer, not for implementing message (sub/pub) systems.
What I do is, if I need RPC I use gRPC for the communication and ProtocolBuffers to serialize the data, and then when I need messaging I use AMQP, also with ProtocolBuffers.