In Hadoop, objects passed to reducers are reused. This is extremely surprising and hard to track down if you're not expecting it. Furthermore, the original tracker for this "feature" doesn't offer any evidence that this change actually improved performance (unless I missed it).

It would speed up the system substantially if we reused the keys and values [...] but I think it is worth doing.

This seems completely counter to this very popular answer. Is there some credence to the Hadoop developer's claim? Is there something "special" about Hadoop that would invalidate the notion of object creation being cheap?

  • I asked this exact same question of an expert from one of the main Hadoop consulting firms, and he just kept asserting it was necessary. I generally don't like party-line answers like that, but I was willing to play along at the time.
    – sea-rob
    Commented Feb 12, 2014 at 4:44
  • @RobY you should convert you comment to an answer... Commented Feb 12, 2014 at 12:02
  • lol thanks, but like his response, I don't think mine is actually an answer.
    – sea-rob
    Commented Feb 16, 2014 at 7:20

1 Answer 1


If you read the article you linked, it says

Running a simple unit test on your desktop machine should highlight that creating 1x10^6 new String objects with random byte content is slower than using a single Text object and calling the set method to configure the underlying byte contents

Well, that is self-evident. Creating a million new strings is always going to be slower than using a StringBuilder to manipulate a single string; everyone knows that. But this may be a straw man; last time I checked, you still needed an individual string for each key in a collection.

If his argument is that allocating a million new strings to make a copy of the collection is expensive, well, yes it is. Strings are reference types, after all; you could just store references to the original strings.

I guess we'll have to wait for him to complete his benchmarks.

  • The problem is that my simple unit test case shows 0 difference. The JIT seems reasonably smart to not copy if it doesn't need to. Besides 1M strings is much too small of a sample given my test created 6M+ in under 10s. I double we'll ever see a completed benchmark since this has been in Hadoop for nearly two years now. Commented Feb 11, 2014 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.