4

In Hadoop, objects passed to reducers are reused. This is extremely surprising and hard to track down if you're not expecting it. Furthermore, the original tracker for this "feature" doesn't offer any evidence that this change actually improved performance (unless I missed it).

It would speed up the system substantially if we reused the keys and values [...] but I think it is worth doing.

This seems completely counter to this very popular answer. Is there some credence to the Hadoop developer's claim? Is there something "special" about Hadoop that would invalidate the notion of object creation being cheap?

  • I asked this exact same question of an expert from one of the main Hadoop consulting firms, and he just kept asserting it was necessary. I generally don't like party-line answers like that, but I was willing to play along at the time. – Rob Feb 12 '14 at 4:44
  • @RobY you should convert you comment to an answer... – Andrew White Feb 12 '14 at 12:02
  • lol thanks, but like his response, I don't think mine is actually an answer. – Rob Feb 16 '14 at 7:20
1

If you read the article you linked, it says

Running a simple unit test on your desktop machine should highlight that creating 1x10^6 new String objects with random byte content is slower than using a single Text object and calling the set method to configure the underlying byte contents

Well, that is self-evident. Creating a million new strings is always going to be slower than using a StringBuilder to manipulate a single string; everyone knows that. But this may be a straw man; last time I checked, you still needed an individual string for each key in a collection.

If his argument is that allocating a million new strings to make a copy of the collection is expensive, well, yes it is. Strings are reference types, after all; you could just store references to the original strings.

I guess we'll have to wait for him to complete his benchmarks.

  • The problem is that my simple unit test case shows 0 difference. The JIT seems reasonably smart to not copy if it doesn't need to. Besides 1M strings is much too small of a sample given my test created 6M+ in under 10s. I double we'll ever see a completed benchmark since this has been in Hadoop for nearly two years now. – Andrew White Feb 11 '14 at 17:49

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.