2

Brief overview of general data flow

The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic analysis to make it searchable and to be able to draw links between the files.

The general flow goes like this:

  1. User uploads files to S3 bucket through web server
  2. Web server completes upload
  3. Web server send message over message bus containing S3 bucket name and file name
  4. Document processing service notified of file over message bus
  5. Document processing service parses document and extracts metadata using Apache Tika
  6. If extracted content contains text, text should be processed with Apache Spark in order to perform some analysis on the text (i.e., using Spark-NLP)

I have a general proof of concept working here, but had some concerns before continuing to build out the system.

Concerns

  • Files to be processed may be very small, or very large
  • Running Apache Tika inside a Spark job may not be the best idea. IIRC, there's no native integration there.
  • Since I'm new to Spark, I think I might be using it for a job it isn't really designed to do
    • These files are small, and need to be processed as they come in, but I don't want to send the whole file content over Kafka, etc. to use Spark Streaming
    • Getting a file name and bucket location doesn't feel like the right way to start the Spark job (at least not with my limited set of knowledge)
  • Current processing of these files is unpredictable
    • Sometimes it gets completed in 0.5 seconds, sometimes 5 seconds. Likely because I have three worker nodes on the system.
  • These jobs are initiated inside of a microservice. When the service starts, I create a SparkSession and re-use it every time a file processing request comes in on the message bus.

Overall, I'm concerned that I'm using Spark in an incorrect way. Does it even make sense to use Spark citing my overall architecture?

1 Answer 1

2

Answers to each question below:

1. Running Apache Tika inside a Spark job may not be the best idea. IIRC, there's no native integration there.

I haven't used Tika before, but it looks like its a framework implemented in Java. It also looks like it has a binding for python as well.

If you are using Spark with Scala or Java, using Tika is as simple as adding it to your JVM classpath and using the relevant API calls you need (MediaType detect(java.io.InputStream input, Metadata metadata) seems like a useful one in your situation).

If using Spark with Python, you just need to make sure Tika is imported in your environment.

2. Since I'm new to Spark, I think I might be using it for a job it isn't really designed to do

Actually, the motivation for creating Spark was the need for iterative data processing and to avoid writing to disk (as in the case of Map Reduce) in each iteration.

Spark can effectively leverage in-memory processing and caching to run iterative algorithms (which are most ML algorithms) to process the data efficiently and quickly. A similar iterative job in Map Reduce can take several hours (due to the slowness of writing to disk for each iteration).

While I'm not familiar with APIs of Spark-NLP, if it has what you need to perform the processing, it can be a good way to process large quantities text efficiently.

3. These files are small, and need to be processed as they come in, but I don't want to send the whole file content over Kafka, etc. to use Spark Streaming

You don't really need to use kafka to notify Spark Streaming of a new file in an S3 bucket. Spark streaming can use HDFS/an S3 bucket as a data source.

Whenever a new file is created in the bucket, it will be streamed. However, changes to existing files will not be streamed.

It is generally recommended to process your data as close as you can to your source data. So running spark on AWS and reading the data from S3 can be a good way of reducing the costs/slowness of data transmission over the network.

However, it looks like you need some way to identify if a document is in a specific text or language and then process it using spark streaming. So you may want to have some way to filter out documents before sending them to an S3 bucket.

I would need to understand your use case better, but you can probably use a separate application to process messages from Kafka/read the file location and then detect the language in a file before moving it to an S3 bucket for processing by spark streaming. (I'm making lot of assumptions here though. So I'd suggest digging a bit deeper in your use case to see what fits your need).

4. Current processing of these files is unpredictable

Can't really comment on performance without knowing more details of the job or the cluster you are running on. Skew in processing speed can exist across different nodes in a cluster. And your job will be as fast as the slowest node in the cluster. You may need to improve your spark code - proper partitioning and reducing data skew might help.

5. These jobs are initiated inside of a microservice. When the service starts, I create a SparkSession and re-use it every time a file processing request comes in on the message bus.

I haven't really used spark in a microservice before. And I think you might be using spark as a service, rather than a processing framework, which is what it is designed as. In any case, I recommend running your spark job as close to the data as possible to reduce costs of network transmission delays and to allow your job to scale more effectively with larger files.

9
  • Thank you for your detailed answer. I suppose I was not specific enough about my use case. I’m totally ok with Tika being run before the Spark workflow to extract the data and detect the language and mime type. I’m actually already running Spark as physically close to S3 as possible. Everything is currently running on my local machine in Docker (including S3, as I’m using MinIO). In prod this will be all within the same Kubernetes cluster. What I mean by the doc processing: I’m not actually performing ops on it yet. Just reading it from the bucket on the same server can take forever or no time Feb 6, 2021 at 15:37
  • I’m not using any SaaS products here, so I’m not using Spark as a service either. Everything is local. It’s sort of bizarre to me that it was advised not to reuse the same ‘SparkSession’ through an applications lifetime, but rather to expensively re-create it every time I would go to process a file. Also, I’m concerned about Sparks ability to “watch” S3 because I have dynamically allocated buckets (one bucket for each grouping of documents uploaded by the user). A lot of these concerns stem from my lack of knowledge about Spark and it’s capabilities. I can’t seem to find many similar use cases Feb 6, 2021 at 15:46
  • If you are running everything for your proof of concept on local, it will be slower than on prod, since you don't really have the level of parallelism of a bigger cluster (well, you sort of do with multiple cores, but still it won't be comparable performance of a production cluster).
    – m_vemuri
    Feb 7, 2021 at 23:17
  • Running multiple spark contexts on the same JVM is not recommended. The multiple spark context configuration was only allowed for internal testing for spark development. If you use it in user programs, it will lead to unpredictable behaviour. So you may want to rethink your code and design so you are not using the multiple contexts.
    – m_vemuri
    Feb 7, 2021 at 23:20
  • 1
    Thank you again for all of your input Feb 9, 2021 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.