High-level overview or architecture for processing a huge file within a fixed time frame

Question

I was asked this question in an interview of Crypto company for the role of backend developer.

Please suggest me another stack-site if this is not the place to ask this question.

There is a subsystem A which sends the data to another subsystem B as a file in.txt. Subsystem B then processes this file and produces an output and writes this into out.txt file which will then be read by subsystem C which does some tasks on its own.

Lets suppose that all that in.txt contains is a numbers separated by new line, and subsystem B has sum all the numbers and write the output into out.txt. You have the basic logic to do this and a new requirement has come or some change occurred in the system so that the size of the input file is not fixed and can be 10kb or can be 10gb.

But this processing has to be done in a fixed amount of time x (say a minute or few minutes), you have to process the entire file and write the output into out.txt within the timeframe. After the x period of time (at x+1 time) your data is not unusable and will be discarded automatically. Your normal adding algorithm might more than given time. So how would you solve the problem and output the summation of all numbers within the given time frame.

My approach is to divide the file into multiple chunks and read through multiple threads (or instances?) and process each chunk on its own and each thread will add the value to out.txt. My approach seems too basic, I'm still learning backend development and want to know how to approach this problem, also could you please provide references or any material so I could dive more into the topic.

Answer 1

Your approach looks reasonable. Note if the file size is small, there is no need to multi-thread processing it at all.

Something else worth considering:

IO bound?

It's better to profile how much time it is spent on disk access vs sum the numbers, along with how chunk size affects the IO time. So you can decide the best possible subfiles you want to split into.

Data processing

One possible optimization is to process the chunk file as binary data. Every time a byte buffer is allocated to read the preceding x MB from the file.

A Java example with integers:

private long sumInBinary() throws IOException 
{
    var f = new RandomAccessFile(file, "r");
    int fileLength = (int) f.length();
    byte buf[] = new byte[16 * 1024];
    int acc = 0;
    long total = 0;
    int read = 0;
    while (read < fileLength) 
    {
        int len = Math.min(buf.length, fileLength - read);
        f.readFully(buf, 0, len);
        read += len;
        for (int i = 0; i < len; i++) {
            if ((buf[i] >= 48) && (buf[i] <= 57))
            {
                acc = acc * 10 + buf[i] - 48;
            }   
            else 
            {
                total += acc;
                acc = 0;
            }
        }
    }
    f.close()
    return total;
}

Multithread

Other than multithread chunks reading, it's also possible to apply multithreaded data process for each chunk. And this involves some parameters (e.g, buffer size) tweaking too.