Questions tagged [spark]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
120 views

Data Ingest Architecture Advice

I have a requirement where we need to collect N different events and store them for analysis. I am having trouble coming up with a general architecture for this. FINAL REQUIREMENTS The end goal of the ...
user avatar
4 votes
1 answer
327 views

How do you perform accumulation on large data sets and pass the results as a response to REST API?

I have around 125 million event records on s3. The s3 bucket structure is: year/month/day/hour/*. Inside each hour directory, we have files for every minute. A typical filename looks like this: ...
user avatar
  • 51
2 votes
1 answer
114 views

How (whether to?) include Apache Spark in my Architecture

Brief overview of general data flow The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic ...
user avatar
-2 votes
1 answer
247 views

Export huge excel file

I develop a web application in Angular (frontend) and Scala (backend) for a big data team. Because they use large files for export/import, I build a module which is a copy of Microsoft Excel. So, what ...
user avatar
  • 117
-2 votes
1 answer
49 views

What are the benefits of running Apache Spark on Kubernetes?

When running Apache Spark one submits jobs to a Cluster Manager. The cluster manager is delegated with the task of accepting / declining requests for resources. The cluster manager could either be ...
user avatar
  • 111
2 votes
0 answers
30 views

How to manage scheduled ETL jobs that are time sensitive?

We have some ETL jobs that are scheduled to run every day, and some that are scheduled to run every week via Control-M. These types of jobs tag data with the date the job was run and perform filter ...
user avatar
  • 2,343
-3 votes
1 answer
90 views

Can accessing the same API from different languages be more performant?

I've just started my first proper internship in industry (not learning to code but learning to write software that does stuff). My employer makes use of Apache Spark, as they do a lot of Big Data ...
user avatar
2 votes
0 answers
23 views

How to design a report processing model using Spark in the most efficient way

I have a reporting system which gets time-series data from numerous meters (here I am referring it as raw_data) I need to generate several reports based on different combinations of the incoming ...
user avatar
-1 votes
1 answer
58 views

Where do you put tests that are not unit tests in a Maven project?

I'm building a Spark-based, text analysis package using both Java and Scala. I have a series of transform functions, which take in one dataframe and spit out another, and that can be chained together ...
user avatar
-1 votes
1 answer
120 views

How to incrementally update value of features in a machine learning pipeline?

I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
user avatar
0 votes
1 answer
2k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
user avatar
0 votes
1 answer
62 views

Could Apache spark be an option?

Today we are using SQL server with multiple indexed views. Whenever we update the source tables for the view there is too long delay. I have no experience with Spark, so the question is: Can we input ...
user avatar
  • 269
1 vote
0 answers
2k views

what's the maximum number of simultaneous java socket connections in the cluster?

we work within a cluster of 1 gb/s of bandwidth, we use java sockets to perform some data transfer between the cluster's nodes like broadcast and shuffle (nodes of the cluster exchange data) in the ...
user avatar
3 votes
0 answers
461 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
user avatar
  • 1,856
2 votes
1 answer
347 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
user avatar
  • 123
7 votes
4 answers
2k views

Can someone explain the technicalities of MapReduce in layman's terms?

When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
user avatar
4 votes
1 answer
112 views

spark output back to web page

We are running jobs whose parameters come from a web page and are executed on large files on a spark cluster. After processing, we want to display the data back, written to text files using rdd....
user avatar
  • 557