0

I have a reinforcement learning project. For this I created a vectorized environment in C++, which is a handler for multiple instances of a simple game. It is highly parallelizable. Each worker can work on their own batch of instances, there is no information exchange between them. (For the most part.)

My current approach is to use std::asynch in the following manner:

[...]
auto foo = [&](int thread_id) { 
    size_t chunk_size = env_count / (worker_count) + 1;
    size_t my_start = thread_id * chunk_size;
    size_t my_end = std::min(my_start + chunk_size, env_count);

    // Take care of instances between the given interval
    move_piece(my_start, my_end, ...);
    raise_turns(my_start, my_end, ...);
    raise_score(my_start, my_end, ...);
    do_magic(my_start, my_end, ...);
         
    return;
};

std::vector<std::future<void>> futures;
futures.reserve(worker_count);

for (auto p = 0; p < worker_count; ++p) {
    futures.emplace_back(std::async(std::launch::async, foo, p));
}

// Need to wait for all to finish
for (auto &elem : futures) {
    elem.wait();
}

This code runs each time the envs take an action.

I was wondering if the constant re-starting of threads (e.g. recreating the std::async for each time the envs take action) causes significant overheads and taken a look at thread pooling. I implemented this solution (with extra flags to know if all threads are finished) that utilizes a single job queue and multiple workers. It was slower then using std::asynch.

Is there a possibly better way to utilize concurrency? I was thinking about a one-queue-per-worker thread pool implementation, but you might be able to give me an even better approach.

Thanks in advance!

P.S.:

As an extra info, I tried to utilize data oriented design and therefore my functions share structure with the following toy example:

struct A {
  int x;
  int y;
};

struct B {
  int z;
};

void foo(size_t my_start, size_t my_end, std::vector<A> in, std::vector<B> out){
  for (size_t ii = my_start; ii < my_end; ++ii){
    out[ii].z = in[ii].x + in[ii].y;
  }
}

1
  • 1
    Perhaps take a look at the Producer Consumer Design Pattern.
    – mrflash818
    Feb 4, 2021 at 17:41

2 Answers 2

3

I was wondering if the constant re-starting of threads causes significant overheads and taken a look at thread pooling.

Constant re-starting of threads is indeed a bad idea, but std::async doesn't tell you how it manages its threads. It certainly isn't required to launch a new thread for every task. Perhaps your implementation is already managing a thread pool better than the one you wrote yourself.

If it's performing well enough, then you got everything you need for free already. Congratulations! You have a good implementation. Use it.

  • Edit to clarify since this seems to be confusing: when I said "std::async doesn't tell you how it manages its threads. It certainly isn't required to launch a new thread for every task." I did indeed mean that each call to std::async(std::launch::async, ...) is not required to launch a new thread. It is permitted to use a persistent pool of reusable threads, and it's possible that your implementation is already doing this.

    The requirement is that the function is invoked as if in a new thread of execution, where "as if" is the normal standard weasel wording for feel free to do something that performs better so long as it behaves the same.

Is there a possibly better way to utilize concurrency? I was thinking about a one-queue-per-worker thread pool implementation, but you might be able to give me an even better approach.

If you have a queue per thread, then the producer needs to decide which thread does the work. This is generally worse, but might be fine if all the tasks take the same time. If you think the single queue is a significant bottleneck, then you should profile to figure out why. It shouldn't be holding the lock very long.

If your tasks are very fine-grained, the queue can be a bottleneck, but that's because (un)locking the mutex is expensive compared to the task itself. Just make your tasks bigger or batch them together to amortize the synchronization cost.

4
  • I might not have expressed myself precisely. By re-starting the threads, I was not referring to how std::asynch handles things. I was referreng to the creation of those with futures.emplace_back(std::async(std::launch::async, foo, p)). As the program bounces back and forth between GPU and CPU computations, those futures.emplace_back(...) parts are called rather often.
    – Dudly01
    Feb 4, 2021 at 17:03
  • I edited the original question to include this clarification.
    – Dudly01
    Feb 4, 2021 at 18:23
  • 1
    And I said that calling std::async - even with std::launch::async - is not required to start a new thread. Move-constructing the future certainly doesn't. The requirement is that the function is invoked as if in a new thread of execution. You need to check what your implementation is actually doing, because it may already use a thread pool.
    – Useless
    Feb 4, 2021 at 21:52
  • You are right. Cppreference also says that it "might be a part of a thread pool". I will try to look into it. Thanks for the help.
    – Dudly01
    Feb 4, 2021 at 22:08
0

The only thing that worries me, for now, is the error handling and if any of the tasks hang... From my experience, you only should use wait() for short functions and that you really do not care so much about the result.

I would suggest using std::future::wait_for and promises with set exception: std::promise::set_exception

1
  • My functions are fairly fast. Stepping with 2^18 envs takes 200 ms on old hardware with one worker. (However, I am making 30k+ steps per training so every little counts.) Most of the functions are also noexcept and I do not want to handle any errors that occur in the "parallelizable" functions.
    – Dudly01
    Feb 4, 2021 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.