So at a high level my use case is as follows -

I periodically (every 24 hours) get a very large file (size can vary from MBs to 10s of GBs) which I need to process within 24 hours. The processing involves reading a record, apply some Business Logic and updating a database with the record.

Current solution is a single threaded version which

  1. initially reads the entire file in memory, that is, it reads each line and constructs a POJO. So essentially it creates a big List
  2. It then iterates on the List and applies business logic on each Pojo and saves them in database

This works for small files with less than 10 million records. But as the systems are scaling we are getting more load, i.e. larger files (with >100 million records occasionally). In this scenario we see timeouts, that is we are unable to process the entire file within 24 hours

So I am planning to add some concurrency here.

A simple solution would be-

  1. Read entire file in memory (create POJOs for each record, as we are doing currently) or read each record one by one and create POJO
  2. Spawn threads to concurrently process these POJOs.

This solution seems simple, the only downside I see is that the file parsing might take time since it is single threaded (RAM is not a concern, I use a quite big EC2 instance).

Another solution would be to -

  1. Somehow break the file into multiple sub-files
  2. Process each file in parallel

This seems slightly complicated since I would have to break the file up into multiple smaller files.

Any inputs on suggestions here on the approaches would be welcomed.

  • 2
    Is it necessary to have the entire List in memory all at once, or can you process an Element at a time? If all you're doing is iterating over the List to process Elements, try changing your code to read just enough data to produce one Element, process it, then repeat. Nov 28, 2016 at 13:55
  • Yes, I have mentioned that in my solution 1. But my concern here overall is that this file parsing (one line at a time) will still be single threaded and I might not get an optimal performance.
    – AgentX
    Nov 28, 2016 at 13:58
  • 7
    You should profile your application and see where the majority of the time is spent. When you identify the hotspot and bottleneck, then evaluate whether multithreading is a good solution to that problem. For example, if your DB is on the network and you don't batch your entries, there is pretty much no way you can get sub-millisecond insertion times, and multithreading is not going to do you any good (you'd need proper batching and DB connection reuse to solve that). Or your DB may be slow, in which case again your multithreading improvements are worthless.
    – Ordous
    Nov 28, 2016 at 14:19
  • 2
    @DanPichelman I with Dan here. Before you go down the fraught path of trying to multi-thread, you should see how not reading everything in to memory affects performance. Almost every time I've worked with someone on performance in their Java program, the cause of the issue was creating a giant List. It takes a lot of time to allocate all that memory. For sure that's a big drag on your performance and you'll get far more improvement from that than you will with multithreading. Do that first.
    – JimmyJames
    Nov 28, 2016 at 14:52
  • @JimmyJames I agree that I need to profile the system and I will do that. It's just that I wanted to get some inputs to see if I am heading in the right direction.
    – AgentX
    Nov 28, 2016 at 15:05

2 Answers 2


The most likely efficient way to do this is:

  • Have a single thread that reads the input file. Harddisks are at their fastest when reading sequentially.
  • Do not read it into memory all at once! That is a huge waste of memory which could be used much better to speed the processing!
  • Instead, have this single thread read a bundle of entries (maybe 100, maybe 1000, this is a tuning parameter) at once and submit them to a thread to process. If each line represents a record, the reading thread can defer all the parsing (other than looking for newlines) to the processing threads. But even if not, it is very unlikely that the parsing of records is your bottleneck.
  • Do the thread handling through a fixed size thread pool, choose the size to be the number of CPU cores on the machine, or maybe a bit more.
  • If your database is an SQL database, make sure the individual threads access the database through a connection pool and do all their DB updates for one bundle of entries in a single transaction and using batch inserts.

You might want to use Spring Batch for this, as it will guide you towards doing the right thing. But it is somewhat overengineered and hard to use.

Keep in mind that all of this might still be futile if the DB becomes your bottleneck, which it very easily can be - SQL databases are notoriously bad at dealing with concurrent updates, and it might require quite a but of finetuning to avoid lock contention and deadlocks.

  • Thanks for the answer. I will start with optimizing the file reads first and then take it from there. I will also likely keep some threads to delegate the tasks and measure the performance. I am using DynamoDB so that should not be a bottleneck. This is a temporary solution we had developed and until the final system is ready I would like to ensure that we do not get pages everytime the file has some 100 million records.
    – AgentX
    Nov 28, 2016 at 14:59

Let's start with some basic arithmetic.

(* 24 60 60)

This means that there are 86400 seconds in one day.

(/ 100e6 86400)

What this means is that to process 100 million records in one day, you must be able to process 1157.4 records per second.

Going one step farther:

(/ 1.0 1157.4074074074074)

That means you must be able to process one record, end-to-end, in 864 microseconds.

No matter what you do, this is ground truth. If it takes more than 864 microseconds to process a record completely, you aren't going to be able to process 100 million records in 24 hours.

Adding "threading" will make it worse, not better, because you add overhead and don't remove any of the underlying workload.

I suspect, after almost 40 years in this crazy racket, that reading the file into memory and writing the results to your DBMS is eating you alive.

  • Thanks John for the answer. We use DynamoDB as the database and what we have observed that after the file is read in-memory 1 thread can at max consume 150 write capacity of the database. So we have been breaking up the file manually and running scripts over it to consume the sub-files parallely and achieve greater DynamoDB write throughput. So I thought of incorporating that concurrency in the code itself.
    – AgentX
    Nov 28, 2016 at 15:02
  • 2
    I agree with the last sentence completely but I'm not sure I follow why multithreading can't help. If, for example, the program can only process one per millisecond, adding a second thread gets you 172,800,000 done in a day assuming there are no locks. Even if the overhead or locking brings the per-thread performance down 40%, you still make the 100 million mark.
    – JimmyJames
    Nov 28, 2016 at 15:03
  • 1
    @JimmyJames: Any given computer has X number of instructions per second available to do whatever. You can dedicate all X instructions per second to user code, which is the single-thread approach, or you can fork off Y instructions per second for thread-switching and thread management, leaving X - Y instructions for user code. If you can't make your schedule at X instructions per second, you darned sure can't make it at X - Y instructions per second. Nov 28, 2016 at 15:26
  • 2
    The computer I'm typing this on has 8 CPUs. A single threaded app cannot use more than 1/8 of the total instructions-per-second available on the box.
    – JimmyJames
    Nov 28, 2016 at 15:29
  • 2
    @JohnR.Strohm Who isn't running multi-core servers in 2016?
    – JimmyJames
    Nov 28, 2016 at 15:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.