2

I have a predictive model which runs 10,000+ scenarios to predict the outcome. It's a micro-simulator model for an insurance company. The data part is almost static with changes from time-to-time, meaning data is kind of read-only and data update happens outside of this simulator.

Each simulation run accepts several input parameters through a user interface. Upon submission of a run (with all user defined parameters), this model spawns 10,000 parallel jobs in cloud environment using 10,000 cores. Each of these jobs reads entire data at the beginning from a SQL database and then starts crunching data. Each job is CPU intensive and consume the CPU capacity to the extent possible. So far, memory usage is not an issue. A recent code optimization has brought the run-time down from 4+ hours to less than 1.5 hours. Meaning calculation engine takes around 90 mins to complete data crunching for a single job in a high end dedicate physical server (this is just to give you a sense of run-time). So far end users have no complaint and everything looks good.

The calculation using is written using un-managed (to reduce layers of libraries to manage things from behind the scene) C++. The data volume is not that big (around 1,500,000 rows in a table) and the calculation engine reads the data in a single connection at beginning which takes less than a minute.

Recently we are feeling for a revisit to the architecture to determine whether this distributed architecture is the best or there may be a better way to handle this. Management has a wish to bring down the run-time within 15 mins which seems difficult unless we made some drastic changes in design. We are exploring for options like GPUs, FPGAs or things like an Inifiniband backbone, use of Hadoop etc.

Any similar experience / idea to deal with this kind of architecture? Any idea if it can be done differently instead of the way we are doing it now.

Improve this question
5
  • 1
    Any similar experience/idea to deal with this kind of architecture? any similarity could be merely anecdotical. Since you already have a working architecture you are the only one in a good position to tell what could be done to improve performance in this specific scenario. We would need references or better context of such architecture. I wonder if an overview of the system, alongside with the hotspots (mentioned by Robert) might help the community to provide answers not merely speculative. ATM. the answers are going to be pure guesswork.
    – Laiv
    Mar 28, 2019 at 15:28
  • Though Robert Harvey's proposal would uncover this problem, I would investigate database load first. 10,000 execution threads calling one SQL database for the "entire data" set all at once sounds like a recipe for a database bottleneck.
    – Kyle McVay
    Mar 28, 2019 at 15:31
  • can you tell us exactly what cloud servers you are using? ie EC2 instance type
    – Ewan
    Mar 28, 2019 at 15:45
  • 2
    If 1 modeling scenario takes 90 minutes from end to end, then it simply does not matter how many jobs you perform in parallel or how well you parallelize, you cannot get the process to take under 90 minutes. So it sounds like your issue is how to make a single job run faster first, such as by examining if you can take advantage of multiple cores to speed a single modeling scenario.
    – BrianH
    Mar 28, 2019 at 16:36
  • One thing you might want to clarify is whether you want to increase throughput of your simulations, or perceived latency. For example, you can perhaps reduce perceived latency by precomputing some things every ten minutes rather than running them solely on demand. But some things simply take time, no way around it.
    – amon
    Mar 28, 2019 at 21:20

1 Answer 1

Reset to default
2

Management has a wish to bring down the run-time within 15 mins.

Start measuring your code to find out where the "hot spots" are. Hot spots are the places where your code spends the most time executing. Usually, they comprise less than 10 percent of your code. Once you find them, focus your efforts solely on improving the performance of the code in these hot spots.

Other things to check for:

  1. Database latency/locks. If it's SQL Server, explore NOLOCK
  2. The time spent sending data over the network
  3. The overhead involved in spinning up 10,000 jobs simultaneously. Maybe use fewer, less granular jobs?
Improve this answer
2
  • Thanks for your suggestion, but code optimization has been done in recent past which brought the run-time down from 4+ hours to less than 1.5 hours. Now I am looking for some architectural option, whether we can do it differently instead of the way we are doing it now. Our data volume is not that big (around 1,500,000 rows in a table) to call it a big-data, so not sure whether Hadoop will be a help.
    – PB4133944
    Mar 28, 2019 at 15:58
  • See my update. I don't really see this as an architectural problem. Architecture is meant to provide organization and maintainability, not raw performance per se.
    – Robert Harvey
    Mar 28, 2019 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.