1

I have an application that returns projects. A Project has many Tasks each Task has a field called hours. A project returns the total hours for all the Tasks that belong to it.

My application processes hundreds of thousands of tasks updating and synchronizing. In the past, I have done no calculating when a task gets saved or updated to the database. Currently, when someone requests the Project, at the time of the request I do a calculation to see what the current total is at the time of the user's request.

Unfortunately, this is not efficient. The overhead cost of calculating the total on the fly is enough to make the response times too slow and I want faster response times.

When my app synchronizes itself with the source of where it is scraping data from it gets each task as HTTP requests in groups of between 10-400 individual POST/PUT requests each for the same Project.

I am concerned that if I trigger a background task to run for each task, recalculating the total for each one that I might cause too much stress on my app/database host CPU. I will essentially be going from calculating the total a few times on a project whenever needed, to potentially 10-400 times in under a minute. Are there any industry-standard ways to recalculate totals?

I want to send the recalculations to a background task, but I don't want it to run 399 times when it only needs to run once. I have no way of knowing when my app receives the last request.

2

There is an optimization that can work here but it creates a danger. One of the best design principles to follow is to have a single source of truth. Make clear to fellow developers what your source of truth is.

Calculating state when requested means your single source of truth is in the records of each task. This is a fine model as long as it's performant.

Your alternative is to recalculate the total every time the records are updated. But you're not sure this is better because it means lots of calculating all at once.

You're up against one of the hardest problems in computer science: cache invalidation.

Your biggest concern seems to be the user experience so I'll focus on that.

You say new tasks come from scraping data. That takes time right there. If the user didn't trigger that then they will always be expecting old data. That means it's ok if your results are stale. That gives you time to calculate before they ask for the data.

Scraping is slow in comparison so I'd be surprised if adding totalling to it would be the real slow down. If it is you can always keep a running total and just add new hours to it. An ID or timestamp of when it was last updated can be a handy way to prevent records from getting double counted or missed. Test this against doing a full recalculation as you develop to be sure it's working.

Now if a request comes in and it's ok to give a stale result you can just give whatever was last calculated at whatever time. You never need to calculate after a request.

But I wouldn't do any of this if I wasn't sure the on demand calculation was really making the user wait. Keeping shadow state in a DB is always a pain.

2

If you think of TaskHoursAdded as an event, you can look into event sourcing and the CQRS pattern for an abstract description of how others have solved similar problems. One concrete potential implementation is to use Kafka stream aggregation.

The basic idea is you push TaskHoursAdded events into a Kafka topic, then set up a Kafka stream that listens to that topic and keeps a running total per project. Instead of 10-400 times per minute performing a calculation that restarts from the beginning, 10-400 times per minute you are adding 1 task's hours to the saved total. Then you can perform an interactive query at any time to get the current total.

Note this adds complexity that pretty much everyone who talks about it says to avoid unless necessary, but you might be at that point since you already have a simpler solution that works but not fast enough.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.