Hot answers tagged

2

the compute requirements of the calls you're making to the API are orders of magnitude larger than that required by the rest of the application. For all the stuff you're talking about, this is almost certainly the case. It doesn't matter if the calls in your language binding take 1 μs, 1 ms or even 1 s if the actual processing inside the guts of Spark takes ...


2

But, I also have test "mains" that are intended to be run using spark-submit at the command line. There are two primary functions for these test mains: the first is to ensure that I don't run into memory problems with very large partitions, and the second is to time various stages through the Spark UI to optimize runtime. This sounds to me as though what ...


Only top voted, non community-wiki answers of a minimum length are eligible