The key to tracking down performance problems is:
- Know when they exist.
- Have sufficient context to figure out what was going wrong when they were slow.
The key for both of those is logging. The ideal is to have logging with optional logging levels that will spew out more detail which can be selectively turned on.
For a very good example of what works, take a look at Oracle. At all times, as part of the basic functionality of the system, it keeps track of what queries were run, and how long they took. DBAs can go and look at the situation to figure out where performance is going. (Don't just look at slow things, in a system under load a very common fast query can be a bigger problem than an occasional slow one.) Furthermore you have the ability to take a query, run it, and have Oracle dump out a detailed trace of exactly what happened, and where time went. Based on those dumps it is possible for an experienced DBA to figure out exactly what happened and where the bottleneck is.
Yes, there is a constant overhead from having this monitoring present. They try to minimize it, but it is still there. However the first time that it helps you locate a performance bottleneck that you hadn't realized was there, it pays for itself in spades. Without monitoring you're praying that you don't make any silly performance problems. Prayer is simply not a reliable way to get to scalability.
If you have a complex system with lots of RPCs, life gets more complicated. The unfortunate reality is that tracking down a seemingly random slow front end request to an RPC several layers deep that may or may not fire can turn into a nightmare. The solution, which practically nobody does, is to have your RPC mechanism have the ability to label a small fraction of requests as "tracer bullets". Those requests, and all RPCs recursively through your system, will get logged in detail, and those logs collected together to give an accurate picture of those requests. Sure, there is overhead to doing so. Which is why something like 0.1% of requests are logged in that detail. But when the system has trouble, you can now go in and look for a slow traced request, open it up, and immediately see where the performance problem is.
Yes, this is a lot of work. Yes, it is invasive. But I cannot stress enough how important this is to have in a large complex system. As the old saying goes, failure to plan is planning to fail. If you don't have a plan to figure out performance problems, when they happen you won't have a way to figure them out.