1

I'm designing a project which will require iterating through a foreign language dictionary in JSON format. The idea is clicking on a word in a text will trigger a search on the JSON file for a match.

As the JSON file in question is large, i.e. many megabytes in size with over 100,000 lines, what are some performance strategies I could apply to make this repeated iteration more manageable?

The requirement is that a text be uploaded into the application, and an analysis performed against it up front - i.e. if a text of 1000 words (excluding duplicates) is uploaded, then 1000 read-throughs of the file will be needed to provide definitions for each unique word in the text.

I was considering splitting up the JSON file into many smaller files, perhaps by letter, so any word beginning with M will know to look at the M.json file. This would prevent iteration across the entire data set.

Is there a more optimal strategy?

3
  • 3
    Load the JSON into database. Use database's indexing to optimize lookup. – Euphoric Feb 17 at 5:59
  • @DocBrown - All the lookups would be done up front in a single go when the text is uploaded. So if the text to be analyzed has, say 1000 words (excluding duplicates), then I would need to run through the file 1000 times to find each definition in the JSON. – vordhosbn Feb 17 at 7:29
  • I've updated the question to make a bit clearer. Thanks for the replies. – vordhosbn Feb 17 at 7:29
4

Performance strategies for repeatedly iterating through large JSON files

The performance strategy is to not do that. In fact, do not do any of the things in that sentence:

  1. Don't use files.
  2. Don't use JSON.
  3. Don't iterate.
  4. And most certainly don't repeatedly iterate.

Problem #1 might solve itself, because after the first iteration through the file, it will be in the filesystem cache, and thus on subsequent iterations, you are at least no longer hitting the disk.

You could solve problem #4 by sorting both the file and the input, then at least you only need to iterate once.

And problem #3 solves itself when you solve problem #2 by choosing a more appropriate data structure. A simple data structure for your use case would be one that is actually literally named after what you are doing: the dictionary.

A standard in-memory dictionary or map data structure backed by a hash table should be more than enough. If that turns out to still be a bottleneck, you can look to something like Tries.

However, don't forget to follow the steps of performance improvement:

  1. Do you even need performance? If the answer is "No", there is no point in improving performance if you don't actually need performance.
  2. What do you mean by "performance"? Throughput? Latency? Memory usage? Are you talking about peak, steady-state, cold-cache, warm-cache? If you don't know the answer to those, there is no point in improving performance, because you don't know what to improve.
  3. At what point is performance acceptable? I.e. having answered the questions from #1, what number means "performant enough"? If you can't answer this question, there is no point in improving performance, because you never know when you have reached your goal.
  4. Benchmark, benchmark, benchmark! Without a way to measure the number from point #2, there is no point in improving performance, because you can never tell whether the performance is improving. Note that benchmarking is hard. Really, really, really, really hard. There is a reason standard benchmarks are written by specialized benchmark engineers whose full-time job is writing benchmarks. And even they still get it wrong sometimes.
  5. Profile, profile, profile! Once you have figured out using #3 that your performance is lacking, you need to find out where, how, why, and when it is lacking. There is no point in improving performance if you don't know where it needs improving and how to improve it.
  6. Can you scale vertically? Throwing faster / better hardware at the problem is almost always cheaper than throwing engineers at the problem.
  7. Can you scale horizontally? Throwing more hardware at the problem is more complex with faster diminishing returns than throwing faster hardware at the problem (especially if the system is not designed for horizontal scaling), but still cheaper than engineers.
  8. Algorithms and data structures yield bigger gains than micro-optimizations.
4
  • I have an app processing a JSON file with 200,000 records, about 10 MB, and that was handled reasonably well on an iPhone 4s by loading everything into an in-memory dictionary. – gnasher729 Feb 17 at 10:01
  • Yep. "megabytes" is really not that large, unless we're talking microcontrollers with storage space measured in bytes and clock speeds measure in kilohertz. Hence the checklist. Don't worry about performance unless you need to worry about performance. – Jörg W Mittag Feb 17 at 10:04
  • 1
    Of course “megabytes” is huge if you parse it 1,000 times! – gnasher729 Feb 17 at 22:34
  • Amazing answer, vielen Dank, @JörgWMittag! I clearly need to learn more around performance concepts, since I was using the term too simply to mean "reads the file and returns the result quickly". Perhaps, as @gnasher729 suggested, I could use an in-memory database, something like Redis. The primary concern is choosing an approach or technology conducive to a very high number of read operations. Clearly, parsing a file on disk is does not fit the bill! – vordhosbn Feb 18 at 0:20
4

As a principle, you should use JSON only for data transport and storage. If you have a large JSON file that represents a database, then you parse it and convert it to a database once. It could be a persistent database, or a dictionary in memory that is loaded once at program start time. Either way you should be able to access the information that you need instantly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.