I was asked this question in an interview of Crypto company for the role of backend developer.
Please suggest me another stack-site if this is not the place to ask this question.
There is a subsystem A which sends the data to another subsystem B as a file in.txt. Subsystem B then processes this file and produces an output and writes this into out.txt file which will then be read by subsystem C which does some tasks on its own.
Lets suppose that all that in.txt contains is a numbers separated by new line, and subsystem B has sum all the numbers and write the output into out.txt. You have the basic logic to do this and a new requirement has come or some change occurred in the system so that the size of the input file is not fixed and can be 10kb or can be 10gb.
But this processing has to be done in a fixed amount of time x (say a minute or few minutes), you have to process the entire file and write the output into out.txt within the timeframe. After the x period of time (at x+1 time) your data is not unusable and will be discarded automatically. Your normal adding algorithm might more than given time. So how would you solve the problem and output the summation of all numbers within the given time frame.
My approach is to divide the file into multiple chunks and read through multiple threads (or instances?) and process each chunk on its own and each thread will add the value to out.txt. My approach seems too basic, I'm still learning backend development and want to know how to approach this problem, also could you please provide references or any material so I could dive more into the topic.