Let me preface this by admitting that I am a novice - this is the first real project I am doing outside of a class.

I am running a single-threaded C++ simulation several hundred times (i.e. I have several hundred 'jobs'). I would like to be able to have a queue to which I add jobs (perhaps changing a parameter or two each time) and from which I can remove jobs as I please.

I have written a simple multithreaded program in the past, and was considering making each job a thread - but I thought of two problems with this approach:

  • if I ctrl-C out of my program, then all threads stop, not just the one I am interested in. So if I want to stop one job but not the others, I don't have a way to do it.
  • I cannot just add jobs, because I need to compile my entire program before I can run it.

I'd love to hear any suggestions as to how I can manage individual jobs - as far as I can see, multithreading may not be the way to go.

  • 1
    The part about ctrl+C makes it sound like you want to read up on "job management" commands in the shell, but if the goal is to efficiently run several jobs in parallel you really do not want to do this at the level of shell forking. Background shell jobs are meant to be for one-off things that take a long time. If the goal is paralellism, threads ought to be more efficient than processes, much less subshells. That said, do you even need it to be parallel? You haven't cited any performance issues yet (and parallelism is rarely worth the effort/bugs until you have unmet performance needs).
    – Ixrec
    Nov 26 '15 at 7:47
  • My simulation runs most efficiently without multithreading (the overhead is just too much to merit creating multiple threads for a single job). However, I need each job to run on a separate thread. I will be running these simulations on a large cluster at a university research lab for periods of days to weeks.
    – GnomeSort
    Nov 26 '15 at 7:50
  • If each simulation takes that long, and there's no user interaction or anything, then maybe shell-based job management would be better than C++ thread management in this case, so you can simply add jobs by running another instance of the program with different parameters. Presumably googling something like "job managment" or "shell scripting" will get you the rest of what you need.
    – Ixrec
    Nov 26 '15 at 7:54
  • 4
    Multi-threading is the wrong solution for your problem. If you have multiple threads, each simulation can access the state of all other simulations and affect their outcome. The right solution is to run multiple copies of your simulation program and manage those using a script or external program. Nov 26 '15 at 7:59