Multithreading ( thread per area ) for an MMO server, good idea?

Question

As a personal project, I'm setting up a MMO, and writing the server in C#/.NET Core.

As of now my architecture is as follows: The world exists of different areas ( you can go from area to area trough portals ). An area is called a 'room' on my server. Each room has it's own list of clients ( players, each with their TCP stream ).

Now I was wondering if it was ok that I run each room on it's own thread. Each room will manage it's own update cycle ( game loop / room ), as it's independant of other rooms.

My worry is that as the game grows, I will get lots of areas and that a thread per area isn't that great of an idea.

So my question is: Am I doing something wrong or non-benificial in the future ? Am I better off having 1 thread for the game logic for all the rooms ? Or is it ok to have a thread per room ?

Answer 1

Separating the game rooms into different threads is a good start. It is advisable to go beyond that. How much data is shared between rooms? If little data is shared you might design an internal API (e.g. service calls or via a shared database) for sharing data between rooms and have each room be a separate process. This would ease hosting rooms on different machines.

Perhaps each room can listen at its own endpoint (IP address + port). When a player enters a portal to travel from room A to room B: process A hands the player object to process B via the internal API and channel and simultaneously the player's game client opens a new connection to the endpoint where process B is listening.

Threads can be used to enhance concurrency within a room: communication with the game clients can be handled by a pool of threads, separate from the main game loop. Physics can have its own thread. AI can have its own thread.

Answer 2

I think this might be a viable architecture. The biggest source of bugs in multithreaded applications is shared access to data. So when the rooms have little to no communication with each other and need little to no access to the data not explicitly owned by them, then they seem like the obvious candidate for units of parallelization.

If you are worried about too much parallelization, keep in mind that it is usually easier to reduce threading than to introduce more of it. When it turns out you would like to handle rooms sequentially after all, then it is usually not much of a refactoring effort to change your update-method to process multiple rooms in a loop instead of just one. On the other hand, chopping a single-threaded architecture into a multi-threaded one after the fact can quickly end up in a buggy mess.

By the way, you might want to organize your code around the ThreadPool class and pass Tasks to it where each task consists of performing a single update-tick for a single room. The ThreadPool can be configured to use a specific number of threads. You shouldn't set it higher than the number of (virtual) CPU cores. When you give it more tasks than it has threads, those tasks will be enqueued and processed as soon as a thread is finished. That means you can leave your update scheduling to the framework.

Answer 3

Important facts about multithreading:

• Multithreading can be supported by hardware: up to the maximum number of concurrent threads supported by a given CPU, you will be able to achieve a much higher throughput.
• However, some experimental results show that while a higher throughput is achieved in total, the individual throughput of each thread could slightly decrease.
• If you have more active threads than the CPU hardware can support natively, the OS thread scheduler will have to use time slicing and make some threads in excess wait from time to time. Windows for example uses a preemptive priority-driven scheduling algorithm. Switching thread contexts consumes a little overhead.
• There's a technical limit to the maximum number of threads that you can run. On a 32bit Windows you're limited to 2000 threads. On a 64bit Windows you can hope up to 50 000 threads (under linux, the limits will be similar). But this limit will decrease further depending on the resources consumed by each thread.

Consequences for your architecture:

• Your room-threading will be much more pleasant for the user: better a slightly slower single room but without delays, rather than an ultrafast single thread letting users experience lags because of all the other rooms that are to be processed in a sequential manner !
• In an MMO, most of the time your threads will wait for network I/O. This natural waiting effect (voluntary switching) ensures a better usage of the CPU and more than compensate the overhead of context switches.
• However, your maximum number of rooms will be limited. And this limit stands even if older rooms are unused. If you're thinking of numbers far bellow the figures mentionned, then ok. But otherwise, you should consider another alternative.

Alternatives:

Your multithreading approach is a very good start. But instead of allocating a thread to each single room, you could consider to have a thread managing several rooms. How many could be either determined in the configuration or calculated dynamically at runtime.

Another question is how you manage the I/O. Do rooms manage their user connections ? If no, you'd better allocate more threads for the I/O and decouple it with a queue based architecture.

Finally, the problem with multithreaded rooms is that you can't scale beyond the server like real life MMO do. So you need to think how to scale all this, and how you could in future package group of threads into communicating processes, so that you could add servers with additional processes to manage increasing workload/connections.