I have a summary tool written in python that reads input files, and writes them into a summary file. I have the following stipulations:

  1. No duplicates.
  2. If it exists, add a count to it.

Is it better / faster to write one GIANT output file THEN de-duplicate the entries or to dedupe as you go (ie, each write command checks before writing)?

The small files are about 100-300k each, there can be hundreds of thousands of these, and the final output file is usually 1-4 MB.

A sample line in the file would be like this:


I would be checking the string for dupes. If it were a dupe, I would output:


as in, I would keep add the numbers every time I'd get a duplicate, and keep a count of how many times it was duplicated.

I would say there are mostly duplicates.

  • What's the key for the duplication? A string? How many duplicates are there (is almost everything a duplicate, is almost nothing a duplicate)? – Telastyn Apr 18 '14 at 20:16
  • Good point... I added info. – Jeff Apr 18 '14 at 20:26

Old question, but I'll take a stab at it. Writing to memory is almost always going to be faster than writing to disk. It would be better to use a data type like a Set in Java that doesn't have duplicate entries and keep track of them as you go. Then, once you have the data structure in memory, write it to disk.

|improve this answer|||||
  • Definitely if it the end output is 1-4MB, since that means lots of duplicates -> no need to fear that there's not enough memory -> no need to go to the file system for a merge sort. – Kayaman Jul 9 '19 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.