Questions tagged [spark]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
3 answers
239 views

Method naming conventions "setX" vs "withX"

Why learning about Fluent Interfaces, I came across this post which states that using set hints one is mutating the object whereas with is returing a new object. I have seen this pattern first hand ...
Ezequiel Castaño's user avatar
1 vote
3 answers
687 views

Python: Is returning self in method chaining a violation of Demeter's law?

In Python it is very common to see code that uses method chaining, the main difference with code elsewhere is that this is also combined with returning an object of the same type but modified. This ...
Ezequiel Castaño's user avatar
2 votes
1 answer
147 views

Data Ingest Architecture Advice

I have a requirement where we need to collect N different events and store them for analysis. I am having trouble coming up with a general architecture for this. FINAL REQUIREMENTS The end goal of the ...
Sriram R's user avatar
5 votes
1 answer
1k views

How do you perform accumulation on large data sets and pass the results as a response to REST API?

I have around 125 million event records on s3. The s3 bucket structure is: year/month/day/hour/*. Inside each hour directory, we have files for every minute. A typical filename looks like this: ...
Namah's user avatar
  • 61
2 votes
1 answer
201 views

How (whether to?) include Apache Spark in my Architecture

Brief overview of general data flow The general goal of my system is to allow users to upload many different types of files containing data (PDF, CSV, ZIP, etc.), then index it and perform some basic ...
foxtrotuniform6969's user avatar
-2 votes
1 answer
568 views

Export huge excel file

I develop a web application in Angular (frontend) and Scala (backend) for a big data team. Because they use large files for export/import, I build a module which is a copy of Microsoft Excel. So, what ...
AlleXyS's user avatar
  • 117
-2 votes
1 answer
73 views

What are the benefits of running Apache Spark on Kubernetes?

When running Apache Spark one submits jobs to a Cluster Manager. The cluster manager is delegated with the task of accepting / declining requests for resources. The cluster manager could either be ...
vi_ral's user avatar
  • 121
2 votes
0 answers
36 views

How to manage scheduled ETL jobs that are time sensitive?

We have some ETL jobs that are scheduled to run every day, and some that are scheduled to run every week via Control-M. These types of jobs tag data with the date the job was run and perform filter ...
Igneous01's user avatar
  • 2,343
-3 votes
1 answer
98 views

Can accessing the same API from different languages be more performant?

I've just started my first proper internship in industry (not learning to code but learning to write software that does stuff). My employer makes use of Apache Spark, as they do a lot of Big Data ...
HenryPrickettMorgan's user avatar
2 votes
0 answers
27 views

How to design a report processing model using Spark in the most efficient way

I have a reporting system which gets time-series data from numerous meters (here I am referring it as raw_data) I need to generate several reports based on different combinations of the incoming ...
Remis Haroon - رامز's user avatar
-1 votes
1 answer
73 views

Where do you put tests that are not unit tests in a Maven project?

I'm building a Spark-based, text analysis package using both Java and Scala. I have a series of transform functions, which take in one dataframe and spit out another, and that can be chained together ...
kingledion's user avatar
-1 votes
1 answer
144 views

How to incrementally update value of features in a machine learning pipeline?

I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
spoderman's user avatar
0 votes
1 answer
3k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
Syed Jafri's user avatar
0 votes
1 answer
66 views

Could Apache spark be an option?

Today we are using SQL server with multiple indexed views. Whenever we update the source tables for the view there is too long delay. I have no experience with Spark, so the question is: Can we input ...
Mr Zach's user avatar
  • 269
1 vote
0 answers
2k views

what's the maximum number of simultaneous java socket connections in the cluster?

we work within a cluster of 1 gb/s of bandwidth, we use java sockets to perform some data transfer between the cluster's nodes like broadcast and shuffle (nodes of the cluster exchange data) in the ...
Soulimane Kamni's user avatar
3 votes
0 answers
464 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
user3198603's user avatar
  • 1,896
2 votes
1 answer
356 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
Javier92's user avatar
  • 123
7 votes
4 answers
2k views

Can someone explain the technicalities of MapReduce in layman's terms?

When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
Eddie Bravo's user avatar
4 votes
1 answer
129 views

spark output back to web page

We are running jobs whose parameters come from a web page and are executed on large files on a spark cluster. After processing, we want to display the data back, written to text files using rdd....
tgkprog's user avatar
  • 607