The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

4

MapReduce actually has a grouping phase. The map phase essentially consists in transforming inputs into pairs of (key,value) elements. Because the reduce phase consists in "aggregating" all the values associated to the same key, you cannot avoid the need to group all values by key before the reduce phase. This may need a lot of time since values must be ...


2

Spark programs are just java, scala or python code, so they can write data to all the same places any program can write them. In fact, spark does not actually do anything unless you write the end result somewhere with an output operation. If the end result of a spark job is small, it can be written to a relational database or a web service or something of ...


2

But what is MapReduce itself? I think it's best to start from the basics: map is an operation that applies a single-argument function to a sequence of inputs, producing an equal-sized sequence of outputs. For example, assume an sequence of inputs [1, 2, 3] and a function f(x) => x * 2. The output would be [2, 4, 6]. As long as map is given a pure ...


2

the compute requirements of the calls you're making to the API are orders of magnitude larger than that required by the rest of the application. For all the stuff you're talking about, this is almost certainly the case. It doesn't matter if the calls in your language binding take 1 μs, 1 ms or even 1 s if the actual processing inside the guts of Spark takes ...


2

But, I also have test "mains" that are intended to be run using spark-submit at the command line. There are two primary functions for these test mains: the first is to ensure that I don't run into memory problems with very large partitions, and the second is to time various stages through the Spark UI to optimize runtime. This sounds to me as though what ...


1

Kafka used previously only an at-least-once message handling. From the Kafka's documentation: When publishing a message we have a notion of the message being "committed" to the log. Once a published message is committed it will not be lost as long as one broker that replicates the partition to which this message was written remains "alive". In ...


1

MapReduce allows you to create a program that will run on a distributed data center. The program is divided in two phases. The first is sent to the servers where the data is and make a pre-calculation. The results of this phase is sent to the second part of the program to be consolidated. This works best with data processing where the first part results ...


1

Without getting into premature optimization, consider the following design principles: Convention. It seems like you already made the choice to have predictable path names in HDFS (based on a user session ID). You can extend this to have predictable paths for each job. If the jobs are initiated by a web application, then that web app can generate whatever ...


Only top voted, non community-wiki answers of a minimum length are eligible