How do you reproduce error conditions and view what is happening as the application executes?
How do you visualize the interactions between the different concurrent parts of the application?
Based on my experience, the answer to these two aspects are as follows:
Distributed tracing
Distributed tracing is technology that captures timing data for each individual concurrent component of your system, and presents it to you in graphical format. Representations of concurrent executions are always interleaved, allowing you to see what is running in parallel and what is not.
Distributed tracing owes its origins in (of course) distributed systems, which are by definition asynchronous and highly concurrent. A distributed system with distributed tracing enables people to:
a) identify important bottlenecks,
b) obtain a visual representation of ideal 'runs' of your application, and
c) provide visibility into what concurrent behaviour is being executed,
d) obtain timing data which can be used to assess differences between changes in your system (extremely important if you have strong SLAs).
The consequences of distributed tracing, however, are:
It adds overhead to all of your concurrent processes, as it translates into more code to execute and submit potentially over a network. In some cases, this overhead is highly significant - even Google only uses their tracing system Dapper on a small subset of all requests so as not to ruin user experience.
Many different tools exist, not all of which are interoperable with each other. This is somewhat ameliorated by standards like OpenTracing, but not wholly solved.
It tells you nothing about shared resources and their current status. You may be able to guess, based on the application code and what the graph you see is showing you, but it's not a useful tool in this regard.
Current tools assume you have memory and storage to spare. Hosting a timeseries server may not be cheap, depending on your constraints.
Error tracking software
I link to Sentry above primarily because it is the most widely used tool out there, and for good reason - error tracking software like Sentry hijack runtime execution to simultaneously forward a stack trace of the errors encountered to a central server.
The net benefit of such dedicated software in concurrent code:
- Duplicate errors are not duplicated. In other words, if one or more concurrent systems encounter the same exception, Sentry will increment an incident report, but not submit two copies of the incident.
This means you can figure out which concurrent system is experiencing which kind of error without having to go through countless simultaneous error reports. If you've ever suffered email spam from a distributed system, you know what hell feels like.
You can even 'tag' different aspects of your concurrent system (though this assumes you don't have work interleaved over exactly one thread, which technically isn't concurrent anyway since the thread is simply jumping between tasks efficiently but must still process event handlers to completion) and see a breakdown of the errors by tag.
- You can modify this error handling software to provide extra details with your runtime exceptions. What open resources did the process have? Is there a shared resource that this process was holding? Which user experienced this issue?
This, in addition to meticulous stack traces (and source maps, if you have to provide a minified version of your files), makes it easy to determine what's going wrong a large portion of the time.
- (Sentry-specific) You can have a separate Sentry reporting dashboard for test runs of the system, allowing you to catch errors in testing.
The disadvantages of such software include:
Like everything, they add bulk. You may not want such a system on embedded hardware, for instance. I highly recommend doing a trial run of such software, comparing a simple execution with and without it sampled over a few hundred runs on an idle machine.
Not all languages are equally supported, as many of these systems rely on implicitly catching an exception and not all languages feature robust exceptions. That being said, there are clients for a great deal of systems.
They may be raised as a security risk, as many of these systems are essentially closed-source. In such cases, do your due diligence in researching them, or, if preferred, roll your own.
They might not always give you the information you need. This is a risk with all attempts to add visibility.
Most of these services were designed for highly concurrent web applications, so not every tool may be perfect for your use case.
In sum: having visibility is the most crucial part of any concurrent system. The two methods I describe above, in conjunction with dedicated dashboards about hardware and data to get a holidtic picture of the system at any given timepoint, are widely used across the industry precisely to address that aspect.
Some additional suggestions
I've spent more time than I care for fixing code by people who tried to solve concurrent problems in terrible ways. Each time, I have found cases where the following things could greatly improve developer experience (which is just as important as user experience):
Rely on types. Typing exists to validate your code, and can be used at runtime as an extra guard. Where typing doesn't exist, rely on assertions and a suitable error handler to catch errors. Concurrent code requires defensive code, and types serve as the best kind of validation available.
- Test links between code components, not just the component itself. Do not confuse this with a full-blown integration test - that tests every link between every component, and even then it only looks for a global validation of the final state. This is a terrible way to catch errors.
A good link test checks to see if, when one component talks to another component in isolation, the message received and the message sent are the same aa you expect. If you have two or more components relying on a shared service to communicate, spin them all up, have them exchange messages via the central service, and see if they're all getting what you expect in the end.
Breaking up tests involving a lot of components into a test of the components themselves and a test of how each of the components communicate as well gives you increased confidence in the validity of your code. Having such a rigorous body of tests allows you to enforce contracts between services as well as catch unexpected errors that occur when they're running at once.
- Use the right algorithms to validate your application state. I'm talking about simple things, such as when you have a master process waiting for all of its workers to finish a task and only want to move to the next step if all the workers are fully done - this is an example of detecting global termination, for which known methodologies such as Safra's algorithm exist.
Some of these tools come bundled with languages - Rust, for instance, guarantees your code will have no race conditions at compile-time, while Go features an inbuilt deadlock detector that also runs at compile-time. If you can catch issues before they hit production, it is always a win.
A general rule of thumb: design for failure in concurrent systems. Anticipate that common services will crash or break. This goes even for code that isn't distributed across machines - concurrent code on a single machine can rely on external dependencies (such as a shared log file, a Redis server, a damn MySQL server) that could disappear or be removed at any time.
The best way to do this is to validate the application state from time to time - have health checks for each service, and make sure consumers of that service are notified of bad health. Modern container tools like Docker do this quite well, and should be made use of to sandbox things.
How do you figure out what can be made concurrent and what can be made sequential?
One of the biggest lessons I've learned working on a highly concurrent system is this: you can never have enough metrics. Metrics should drive absolutely everything in your application - you are not an engineer if you aren't measuring everything.
Without metrics, you cannot do a few very important things:
Assess the difference made by changes to the system. If you don't know if tuning knob A made metric B go up and metric C go down, you don't know how to fix your system when people push unexpectedly malignant code on your system (and they will push code to your system).
Understand what you need to do next to improve things. Until you know applications are running low on memory, you can't discern whether you should get more memory or buy more disk for your servers.
Metrics are so crucial and essential that I have made it a conscious effort to plan what I want to measure before I even think about what a system will require. In fact, metrics are so crucial that I believe they are the right answer to this question: you only know what can be made sequential or concurrent when you measure what the bits in your program are doing. Proper design uses numbers, not guesswork.
That being said, there are certainly a few rules of thumb:
Sequential implies dependence. Two processes should be sequential if one is dependent on the other in some fashion. Processes with no dependencies should be concurrent. However, plan a way to handle failure up stream that doesn't prevent processes downstream from waiting indefinitely.
Never mix an I/O bound task with a CPU-bound task on the same core. Don't (for example) write a web crawler that launches ten concurrent requests in the same thread, scrapes them as soon as they come in, and expect to scale to five hundred - I/O requests go to a queue in parallel, but the CPU will still go through them serially. (This single-threaded event driven model is a popular one, but it is limited because of this aspect - rather than understand this, people simply wring their hands and say Node doesn't scale, to give you an example).
A single thread can do a lot of I/O work. But in order to fully use your hardware's concurrency, use threadpools that together occupy all cores. In the example above, launching five Python processes (each of which can use a core on a six-core machine) just for CPU work and a sixth Python thread just for I/O work will scale much faster than you think.
The only way to take advantage of CPU concurrency is through a dedicated threadpool. A single thread is often good enough for a lot of I/O bound work. This is why event-driven web servers like Nginx scale better (they do purely I/O bound work) than Apache (which conflates I/O bound work with something requiring CPU and launches a process per request), but why using Node to perform tens of thousands of GPU calculations received in parallel is a terrible idea.
