The first step in debugging performance issues is understanding the bottlenecks in your architecture. In a monolithic application (i.e. traditional n-tier development), you have a few main potential bottlenecks:
- CPU Load
- Memory utilization
- Network bandwidth
- Disk speed
If your application is taking 4 seconds to respond to a request, look at these three things to find what is maxed out. Each item can have different causes:
- CPU Maxed out:
- Look for inefficient algorithms by looking for hotspots in a profiler
- Increase your CPU to a faster model (pretty limited with this option today)
- Memory maxed out:
- Make sure you are not swapping to disk, that is very slow
- Increase memory either to the JVM or to your server
- Use a memory profiler to find memory leaks (i.e. memory that is not reclaimed after garbage collection that should be.
- Network maxed out:
- Add another network card?
- If in a VM environment, collocate your database VM with your app VM to take advantage of the high speed inter-VM networking
- Disk maxed out:
- Make sure you have indexes on your database to prevent full table scans
- Minimize disk use
- Get faster disks (SSDs might be worth the investment, or use RAM disks for temporary files)
If none of these are maxed out and you still have performance problems, there is a good chance you might be suffering from resource locking. If one person is updating a table while another person is querying it, there is a chance that the record(s) being updated will cause the other person to wait. If you can deal with dirty reads, you can reduce the locking overhead in your database.
You will inevitably hit a ceiling of what you can do with bigger hardware in a monolithic environment. At that point you really need to think about scaling out. Stack Overflow does a great deal to allow the system to scale out and remain performant. You can go a full microservice route, or just host your monolithic application on multiple servers.
The main thing you must strive for to enable scaling out is to completely avoid server side sessions. In a "shared nothing" environment, there is no reason to have server sessions. The information that would have gone into session variables either goes in the database, or they are stored in the front-end code in the browser.
You'll want to start working with clusters. The database with multiple cluster nodes can spread the work across each node to smooth out the load. The set of application servers can simply host additional copies of the app, and user a load balancer in front. Without any need for session affinity, you can use simple round-robin balancing which is fast.
Next you'll need to look into caching servers like Redis or some equivalent. If your resources take time to put together, but don't change often, this is the final piece of the puzzle to make the response times very fast.
If you think of any quick responding site on the network, I guarantee that they have invested heavily in scaling out rather than scaling up. The degree that they've done so is very different from each environment. For example, Stack Exchange has been able to do a lot with a hybrid monolithic architecture at it's core, but they are hosting on 9 different web servers (reference).
The bottom line is that it costs a lot of effort, and your daily hosting costs increase, when you invest in the type of changes needed to scale out.