In the past couple of months I've worked with both Apache Spark and Druid for some projects. As I've gone through the process of learning how to use these tools, I've spent some time reading through their documentation and peaking into their inner workings (the under-the-hood code rather than the API level code).

In both projects, they mention both on-heap memory and -- to use a project generic term -- "worker-memory" (the amount of memory a particular worker has when performing a job/task). I'm interested in learning more about the latter.

In these projects, did the developers create classes to model contiguous memory itself (perhaps using Arrays)? If not, I can't quite figure out how they'd go about allowing the "worker-memory" setting to be defined in the first place.

If anyone has any insight into the particulars revolving around this, I'd be interested in learning about it.

  • Some similar systems use containers for managing this. – James Youngman Oct 29 '16 at 7:15
  • @JamesYoungman, can you be more specific? Containers akin to Docker or something else? – dmux Oct 30 '16 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.