and sorry if the question seems a bit naive.
I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received.
Here's what I'm trying to understand :
A lot of events <=> Kafka broker <=> Spark receiver <=> Map/Reduce/Transform/Aggregate/MLearning <=> Storage ?? <=> access by end-users ??
I understand the left part of the workflow, you have some stream of events, distributed by a broker, then consumed by Spark receivers.
I've read about alot of features from Spark, which is able to transform RDDs into other RDDs (basically), using in-memory storage (which can also be persisted or cached). But then ??
I don't have a specific use case in mind, but imagine I want to : - keep an event log of the stream of events "for the record" - aggregate the data (a simple count for example) - apply some machine learning example (let's say regression) - keep some value of the last event that happened for fast operational access
In my mind, this involves different data storage systems, say Hadoop for the logs, Redis for the last event, etc.
Then I'd like users to be able to query every of that persisted data : - a simple REST API to get the latest event value - a complicated query-like system to fetch the event log - some reporting API to get the prediction of the ML algorithm
How would this been achieved through Spark ? Is Spark designed for such use ? Does Spark offer such database-persistent storage ? Or should this be different Kafka consumers of the same event ?
Thanks for the help, I'm a bit confused.