Performance strategies for repeatedly iterating through large JSON files
The performance strategy is to not do that. In fact, do not do any of the things in that sentence:
- Don't use files.
- Don't use JSON.
- Don't iterate.
- And most certainly don't repeatedly iterate.
Problem #1 might solve itself, because after the first iteration through the file, it will be in the filesystem cache, and thus on subsequent iterations, you are at least no longer hitting the disk.
You could solve problem #4 by sorting both the file and the input, then at least you only need to iterate once.
And problem #3 solves itself when you solve problem #2 by choosing a more appropriate data structure. A simple data structure for your use case would be one that is actually literally named after what you are doing: the dictionary.
A standard in-memory dictionary or map data structure backed by a hash table should be more than enough. If that turns out to still be a bottleneck, you can look to something like Tries.
However, don't forget to follow the steps of performance improvement:
- Do you even need performance? If the answer is "No", there is no point in improving performance if you don't actually need performance.
- What do you mean by "performance"? Throughput? Latency? Memory usage? Are you talking about peak, steady-state, cold-cache, warm-cache? If you don't know the answer to those, there is no point in improving performance, because you don't know what to improve.
- At what point is performance acceptable? I.e. having answered the questions from #1, what number means "performant enough"? If you can't answer this question, there is no point in improving performance, because you never know when you have reached your goal.
- Benchmark, benchmark, benchmark! Without a way to measure the number from point #2, there is no point in improving performance, because you can never tell whether the performance is improving. Note that benchmarking is hard. Really, really, really, really hard. There is a reason standard benchmarks are written by specialized benchmark engineers whose full-time job is writing benchmarks. And even they still get it wrong sometimes.
- Profile, profile, profile! Once you have figured out using #3 that your performance is lacking, you need to find out where, how, why, and when it is lacking. There is no point in improving performance if you don't know where it needs improving and how to improve it.
- Can you scale vertically? Throwing faster / better hardware at the problem is almost always cheaper than throwing engineers at the problem.
- Can you scale horizontally? Throwing more hardware at the problem is more complex with faster diminishing returns than throwing faster hardware at the problem (especially if the system is not designed for horizontal scaling), but still cheaper than engineers.
- Algorithms and data structures yield bigger gains than micro-optimizations.