There are a number of factors here but we can definitely lay out some principles around these kinds of situations. Let's start with the basic framework. Consider the following visualization:
time it takes to load |----------|
time it takes to process |----------|
The length of the line represents time. The units involved matter in practice but not at ...
For many formats you have no choice but parsing the complete file. For example, with JSON adding a single zero byte to the end of a perfectly fine JSON file makes it invalid. And parsing the complete structure is likely easier than having a function that processes line by line.
That said, you avoid problems with very large files by passing largish blocks (...
You can always measure, but you might be surprised at the results, especially for sequential access. People don't think about optimizations done at lower levels of abstraction. For example, your operating system is caching files to memory:
$ free -h
total used free shared buff/cache available
Mem: 31Gi 4....
For 90% of problems most people would encounter, reading the file in its entirety and then completely parsing them is faster, simpler, and easier. This should be your default choice when working with smaller data.
You should only use incremental parsing/stream processing when your program may be used in a context where it need to process a very large input, ...
This is often a tradeoff between
memory usage, and
ease of implementation
As you already noted by yourself, reading a file entirely first has the drawbacks of requiring more memory and making it more complicated to report progress.
However, reading a structured file entirely first may be necessary (or at least simpler) when further processing cannot be ...