1

I am designing a program with the flow as outlined below. Note, these are all network calls, there is no system I/O (hard drive).

Initially, multiple independent API calls need to happen - they don't rely on the others at all. The response from these will be processed and if they meet certain criteria, they should spawn a websocket connection and stop their API calling until the socket is closed. For example, if there are 4 API calls happening and one meets the requirements, it stops and connects to a websocket. Now there should be 3 API calls happening as well as a websocket receiving data. There should never be more websockets than the number of initial API calls.

Now, the websocket(s) are also processing the recieved data looking for specific requirements. If any one of the sockets meets the requirement, a SECOND websocket is then connected to. At this point, all outstanding API calls/websockets should close/stop their processing. Only this new websocket should be running looking for its final set of requirements. Once this final set is met, the whole process should restart (meaning the API calls start up again).

Let me walk you through a scenario: 4 API calls start. After 3 seconds one meets the requirements. There are now 3 API calls STILL happening, and 1 websocket running. After 2 seconds, the websocket fails its requirements. There are now 4 API calls happening again. Soon, 2 API calls both meet requirements. Now there are 2 API calls and 2 websockets listening. After another 3 seconds, one of the websockets meets its requirements. All websockets and API calls should stop, and a final websocket is opened. After this final one either succeeds or fails, the initial 4 API calls should be restarted and the process repeats all over again.

My question: Would this benefit from some form of threading (instead of only using async) and if so, what method would you use? Each API call on it's own thread that spawns its following websockets on it too? Maybe all the API calls on single thread but then spawn off websockets on their own thread? Maybe the whole program can be run in a single thread with good performance? The operations are time sensitive so the faster I can determine requirements being met the better! Thanks in advance!

Improve this question
4
  • does your cpu run at 100%
    – Ewan
    Commented Dec 23, 2017 at 15:58
  • 1
    I had a lot of trouble understanding concurrency before reading Java Concurrency In Practice. They simply didn't teach us about concurrency in school, and it's not something I picked up from reading documentation. I'm sure there's something similar for C#, or the book I mentioned would be relevant enough.
    – Max
    Commented Dec 23, 2017 at 22:57
  • 1
    My honest suggestion is to keep it single threaded until you understand more what you are doing. There are a number of disguised ways you can introduce errors to a multithreaded program. Code that would run fine in a single threaded environment can be broken in a multithreaded environment. After you find a resource out there, whether a more senior coworker or a book, you can consider making it multithreaded.
    – Max
    Commented Dec 23, 2017 at 23:02
  • 3
    You only need additional threads if you are doing something using a lot of CPU. Since you are blocking, the scenario you describe is actually close to what the async/await implementation was designed to optimize: perform other work while blocking, and reduce the number of threads. The answer by Mike Robinson is correct, but what he left out, is the worker bee action items can be Tasks in .net, and it can do the scheduling for you.
    – Frank Hileman
    Commented Jan 16, 2021 at 19:57

4 Answers 4

Reset to default
1

It is often possible to divide your system into a client communication layer, which handles the inbound requests and socket connections, and a back-end layer consisting of "worker bees" who remove action-items from a queue and place outbound results onto another. The actual number of workers is set by constraints such as the optimal number of database connections: each worker obtains and holds a connection and uses it for all the work that it receives.

"Thread/Process per unit of work" is what I refer to as "the flaming arrows approach." (Take an arrow, light, and shoot it into the air ...) It does not scale well. Better, I think, to have a limited pool of (possibly, specialist) workers which can run at "full capacity" all the time, relying on the two queues to take up the slack. Meanwhile the communication layer keeps a "tote board" of what it knows each client is doing.

Improve this answer
1

Independent of anything else, you should try to find out how to cancel a network call, once it looks like you dont need or cant use the result anymore. Usually the situation is:

Your computer takes CPU time to send off the call.
Some server does some work and sends results to you. Meanwhile your CPU does nothing.
Data from the server arrives and gets processed, costing CPU time again.

At the very least you want to mark the request so that the results are not processed anymore; you don't want to retry failed requests, you don't want to give error messages for failed requests, you don't want to show a user interface for cancelled requests.

If possible, you tell the server that you are not interested in any further results. That may be tricky.

If you are very lucky, you can cancel a request before it is sent to the server.

Improve this answer
0

If the underlying API does querying to a relational database, then partitioned data parallelism may be useful:

Nodes used based on degree of parallelism needed by query

For BI and analytics queries for which larger amounts of data are likely to be processed, the query execution architecture should be able to parallelize at multiple levels. The first level is partitioned parallelism, so that multiple processes for an operation such as join or aggregation are executed in parallel.

The API can be setup in a number of ways. For example:

For each executable SQL statement in a context, the first run-time services call always tries to obtain a latch. If it is successful, it continues processing. If not (because an SQL statement in another thread of the same context already has the latch), the call is blocked on a signaling semaphore until that semaphore is posted, at which point the call gets the latch and continues processing. The latch is held until the SQL statement has completed processing, at which time it is released by the last run-time services call that was generated for that particular SQL statement. The net result is that each SQL statement within a context is executed as an atomic unit, even though other threads may also be trying to execute SQL statements at the same time. This action ensures that internal data structures are not altered by different threads at the same time. APIs also use the latch used by run-time services; therefore, APIs have the same restrictions as run-time services routines within each context. Contexts may be exchanged between threads in a process, but not exchanged between processes. One use of multiple contexts is to provide support for concurrent transactions.

References

Improve this answer
0

After another 3 seconds, one of the websockets meets its requirements. All websockets and API calls should stop

This implies that there is an undergoing communication between the API calls' processing.

Initially, multiple independent API calls need to happen - they don't rely on the others at all.

This is confusing considering first quote. Is there an API calls' processing join?

It is possible to implement the described scenario either way, multi threaded or single threaded, with a proper implementation of the communication between the API calls' processing implementing the observer design pattern.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.