We're working on a large microservices platform for a company in the live events industry. It's made up of containerized RESTful APIs, mostly built in Node, working with Apache Kafka and Cassandra on the back-end. On top of this system, we're rolling our own analytics layer for the data collected in the APIs. This system is tasked with providing both RESTful APIs of its own, as well as a web-based UI with a dashboard of metrics. This allows customers to pull directly from the APIs for use in their own analytics software, or the option of using our UI to view.
Originally, we thought we'd build a "simple" service that would respond to Kafka topics, pull and crunch the data, and drop the analyzed data into tables for the metrics tables to pick up for display. Our POC revolves around this design, today. What we don't have is the ingestion/analysis layer...and here's where Spark potentially comes in.
I spent the last week vacuuming up as much Spark knowledge as possible, but have zero experience with it, beyond that.
What I'm hoping is; we can essentially replace this homegrown "ingestion" layer we had planned on writing, with Apache Spark. My requirements are:
- Near real-time view of analytics/metrics
- Fast data retrieval across all APIs in origin system
- Metric views by date-range (pre, during, and post-event)
I've provided a rough diagram of my thoughts. In a nutshell, my hope is that I can build a Spark application that will respond to a Kafka consumer event, execute a query against our Cassandra data, and push that data to another set of Cassandra tables, in as real-time as possible speed.
We do expect a great deal of data, especially over time. However, all data in the system is filtered by a specific Event ID, which is timeframe-based, and should result in a much smaller subset of Cassandra data. Sometime in the near-future, however, we'll also want to compare year-on-year.
Based on this (very) rough high-level view of what we're up against, does Spark seem like a viable tool for the job?