Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [spark]

The tag has no usage guidance.

-1
votes
1answer
30 views

Where do you put tests that are not unit tests in a Maven project?

I'm building a Spark-based, text analysis package using both Java and Scala. I have a series of transform functions, which take in one dataframe and spit out another, and that can be chained together ...
-2
votes
0answers
29 views

How to an fit sklearn model across many dataframes in parallel

I have a trained (e.g. fitted) machine learning model from sklearn in python. The model was fit using a fairly small dataset (~200,000 points) so I could easily fit the training data on a single ...
1
vote
1answer
40 views

How to incrementally update value of features in a machine learning pipeline?

I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
0
votes
1answer
39 views

How to implement scalable RBAC library/service?

I need to design and implement a RBAC (role based access control) library to protect RPC calls. A role is a list of verbs (RPC methods). One should be able bind a user (or group of users) to a role ...
1
vote
0answers
31 views

Apache Spark guidance - Can it do what I need?

We're working on a large microservices platform for a company in the live events industry. It's made up of containerized RESTful APIs, mostly built in Node, working with Apache Kafka and Cassandra on ...
0
votes
1answer
201 views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
35 views

Could Apache spark be an option?

Today we are using SQL server with multiple indexed views. Whenever we update the source tables for the view there is too long delay. I have no experience with Spark, so the question is: Can we input ...
1
vote
0answers
543 views

what's the maximum number of simultaneous java socket connections in the cluster?

we work within a cluster of 1 gb/s of bandwidth, we use java sockets to perform some data transfer between the cluster's nodes like broadcast and shuffle (nodes of the cluster exchange data) in the ...
0
votes
0answers
86 views

What would be good architecture options for multi-index data aggregation?

We have a use case where we need to store a large set (many millions) of data "objects" with multiple attributes (that may differ over time/for different "collections", e.g. for diffrent "type" of ...
2
votes
0answers
176 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
1
vote
1answer
251 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
5
votes
4answers
708 views

Can someone explain the technicalities of MapReduce in layman's terms?

When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
5
votes
1answer
98 views

spark output back to web page

We are running jobs whose parameters come from a web page and are executed on large files on a spark cluster. After processing, we want to display the data back, written to text files using rdd....