Brief overview of general data flow
The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic analysis to make it searchable and to be able to draw links between the files.
The general flow goes like this:
- User uploads files to S3 bucket through web server
- Web server completes upload
- Web server send message over message bus containing S3 bucket name and file name
- Document processing service notified of file over message bus
- Document processing service parses document and extracts metadata using Apache Tika
- If extracted content contains text, text should be processed with Apache Spark in order to perform some analysis on the text (i.e., using Spark-NLP)
I have a general proof of concept working here, but had some concerns before continuing to build out the system.
- Files to be processed may be very small, or very large
- Running Apache Tika inside a Spark job may not be the best idea. IIRC, there's no native integration there.
- Since I'm new to Spark, I think I might be using it for a job it isn't really designed to do
- These files are small, and need to be processed as they come in, but I don't want to send the whole file content over Kafka, etc. to use Spark Streaming
- Getting a file name and bucket location doesn't feel like the right way to start the Spark job (at least not with my limited set of knowledge)
- Current processing of these files is unpredictable
- Sometimes it gets completed in 0.5 seconds, sometimes 5 seconds. Likely because I have three worker nodes on the system.
- These jobs are initiated inside of a microservice. When the service starts, I create a SparkSession and re-use it every time a file processing request comes in on the message bus.
Overall, I'm concerned that I'm using Spark in an incorrect way. Does it even make sense to use Spark citing my overall architecture?