I'm in the process of designing a new desktop application which is very different from other stuff I did before, and so I'll be happy if I could be pointed towards the right direction regarding the basic building blocks of it.
The application should read a binary file, process it "line by line" and after some chunk of data has been read and processed should write it back to disk. The raw data, i.e. the original binary files, are usually too large to load into memory, so I have to processes them bit by bit. The second phase (the processing) isn't too computationally intensive, and from previous experience I'm sure that the writing-back-to-disk part will take the most time.
What I currently have in mind are three threads (and not processes) - one is in charge of reading chunks of data to disk, the other does the processing and the latter does the writing back to disk. The main application (Python or Rust, not sure yet) will allocate the memory buffer for the first thread and will be in charge of scheduling the three threads in general.
Does this make sense? I'm aware that my requirements are very similar to that of standard async web apps, so I might be missing some important tools here that could help me avoid writing all of this from scratch.