I have a summary tool written in python that reads input files, and writes them into a summary file. I have the following stipulations:
- No duplicates.
- If it exists, add a count to it.
Is it better / faster to write one GIANT output file THEN de-duplicate the entries or to dedupe as you go (ie, each write command checks before writing)?
The small files are about 100-300k each, there can be hundreds of thousands of these, and the final output file is usually 1-4 MB.
A sample line in the file would be like this:
I would be checking the string for dupes. If it were a dupe, I would output:
as in, I would keep add the numbers every time I'd get a duplicate, and keep a count of how many times it was duplicated.
I would say there are mostly duplicates.