Make multithreaded program asynchronous

Question

There is an application that processes TCP/IP requests according to some logic, and pretty always it connects to DB.

Architecture is very common: one client -> one processing thread -> one DB connection -> one DB thread to process a query(DB thread exists inside DBMS application and mentioned here to show how insufficient current performance is).

The system is based on recent Linux kernel, GCC used is of 6.2 version.

Problem is that DB query can last, and so application's thread is sleeping and clients with tasks which require possibly much faster queries have to wait for a thread which is actually do nothing.

I'm thinking of different possibilities making application asynchronous to some point. The problem is in size and age of application - there is no way I'll find time to fully rewrite each "service" to support some async model (like FSM objects able to store their state while waiting for DB to respond and so able to be moved from stack to some storage). Yet each "service" is a class itself with one function which calls DB function(always same) at some/few point/s inside, so all processing state is on stack(inside this overriding functions), no single class member that can be stored exists, and as mentioned above rewriting it to store something inside "service" object to allow continuation using straight C++ techniques (I know about :-) ) is a half-year job at its best.

Last idea was to provide each thread with custom stack using pthread_attr_setstack(), and manipulate this data manually after DB function call(i.e. DB connection(socket) and stack data can wait in some queue, while current thread can jump to higher stack address(assuming x86 stack grows down)) restoring it from memory and setting stack pointer to appropriate place(asm esp and etc). While manipulating stack data doesn't seem to be a problem(currently), I don't know how to bulk-restore other valuable CPU registers and possibly OS specific thread-related structures, to allow problem-free continuation from some stack address, am I right that restoring stack memory and the stack pointer is not sufficient to continue execution?

I'm looking for some info on how such problems can be/were solved and if an idea of mine can be realized. Any links are appreciated.

Another idea is to have a large pool of threads where most of them are in dead sleeping. If system can normally handle 128 simultaneous threads (128 physical CPU cores) then we can create 1280 threads, but there should be a schedule which will allow only 128 from them to execute at the same time, thus if 1279 threads have been blocked while waiting for DB response, next user request will be processed in thread #1280, and if that thread hasn't been blocked yet further request can be processed in another free thread from, this will continue while we will not get 128 running threads. If DB response has been successfully "epolled" then we can wake up waiting for this response thread. This is not fully asynchronous model as in worst scenario all 1280 threads will be sleeping while waiting for DB, but it will provide an opportunity to have 10 times pending DB requests.

Answer 1

There is more than one way to add asynchronicity. You have explored one option, to make the single threaded application more asynchronous. Perhaps there is another option, to make a wrapper application that calls multiple copies of the original application.

Here I am assuming that somehow your original application has an API, or you can create an API, so you can run that application with no user interface. The user then makes a request, the request is passed to one instance of the application using inter-process communication, and the wrapper application resumes running. When the result is computed, the wrapper application shows the result.

If while waiting for the response, another request comes in, you will maintain an "application process pool", consisting of a limited number of instances of the original application process. The next request goes to the next instance in the pool that is not occupied. And so on. The instances can be dynamically started.

Answer 2

Depending on your setup and requirements, if queries are mostly reads and not much writes (such as a reporting system), you might be able to replicate the data to a database of your own which supports SQL calls from multiple clients without the same limitations. The one DB thread in the current database application is just used to extract data and update your own DB which the clients will query.

In my case for a similar problem, it was an IBM DB2 database with restrictions, so the idea was to replicate to a MySQL database using a custom Java application to read from DB2 regularly and write to the MySQL database which we could query without any limitations.

Answer 3

IMHO, if the code is too bad to be refactored into async behavior, there is no hope that it could be refactored into threaded one. So available options are:

1. Add a front end to start a new process for every heavy request.
2. Cache database data.
3. Bite the bullet and refactor the code into async behavior.