0

We have a Single-Page (SPA) Ajax-based Java Spring/Hibernate app running in Tomcat 8.5. The app's performance is acceptable, but not lightning-fast. A typical Insert Record takes 3-4 sec., e.g.

05 Nov 2019 14:55:41,686 INFO  Insert Start
...
05 Nov 2019 14:55:45,766 INFO  Insert End

We have all the standard stuff, like DB indexes etc. They are working.

No one is complaining, but it's not an ultra-smooth web app like StackOverflow.com. StackOverflow is so fast that all operations take under 2 seconds.

What are some hardware improvements that can take a Web app to the next level?

  • DB: Increase memory & CPU on the DB server? But I've been told this won't help.
  • Increase JVM Heap Size? Already done: 2GB
  • Tomcat box hardware - memory & CPU?

One other thing I understand now is Hibernate (and similar ORMs) are a bad idea for performant apps. They're popular but the app would have more performance gains without them.

8
  • 3
    where is the time actually spent? You need more detailed profiling, just increasing resources won't really help. Also try logging queries, and let the database EXPLAIN the execution plan – just because you have indices does not mean they will be used.
    – amon
    Nov 5 '19 at 20:23
  • 2
    FYI, Indexes slow down inserts at the benefit of speeding up reads, not the other way around.
    – Graham
    Nov 5 '19 at 20:56
  • It's all standard textbook stuff. CRUD operations with Hibernate, Spring Controllers. Time isn't spent too much in any one area. Nothing unusual. My point is all of these work and have acceptable performance, but not top-notch performance. I'm interested in the performance of something like StackOverflow.com which is superb. No one ever clicks away from StackOverflow because it took to long to add a post or open a thread -- but some users click away from our app because it's 4 sec. rather than 1 sec.
    – gene b.
    Nov 5 '19 at 20:58
  • Four seconds for an insert could be very slow, or incredibly fast, depending on exactly what's going on. Are you inserting 1 row? 4000 rows? Are a lot of other queries taking place elsewhere after the insert? Is there a cache somewhere that's being invalidated at the same time? There are a lot of possibilities. Gathering specific timing data on everything that happens between those two logs will help expose any performance issues.
    – Jon E
    Nov 5 '19 at 22:16
8

Before blaming Hibernate for performance issues, you should profile your application. By profiling a given request (if the whole application feels slow, just take any request), you'll get a more precise picture of what exactly is slow. Depending of what you discover, the solution for improving performance would be radically different. Some examples:

  • The request is performing four requests, and then the same fifth request three thousand times, because some developer misunderstood how lazy evaluation works. By fixing the code, the number of requests could be reduced to only five. (This example is actually from a real application I audited a few years ago.)

  • The request asks for too much from the database; essentially, when needing one entry, it just loads the whole table containing about five hundred thousand items, and then searches inside. (Unsurprisingly, the example comes from the same project as mentioned in the previous point.)

  • The request spends time using the CPU. If this is the case, the profiling would pinpoint the location where CPU time is used. From there, you can start optimizing your application.

  • The request uses the memory a lot. Check the data structures.

  • The request makes other requests to other services. Check why, if this is the case, other services take too long to respond.

  • The request waits the database a lot even when performing some very basic queries, while the database is hosted on a very capable hardware. Check the connection between the application server and the database server. (For instance, hosting the app server in USA and database server in India is not the greatest thing you can do in terms of performance.)

  • The request does a lot of disk writes. Check what exactly is logged: logging a lot and forgetting the log level at “verbose” in production is not a good idea, especially if the log is configured to flush every message to disk. (Happened once in a project when a fellow programmer wanted to debug a weird crash in production, and once the problem was fixed, forgot to reset the logging configuration.)

Those are only a few examples of what could be happening under the hood. Again, you have to figure out precisely what causes the slowness, before you try optimizing, especially if your only optimization idea is to switch to a different technology or ask for more expensive hardware. Once you find the bottleneck, you'll probably figure out how to fix it. It may be a simple change in code or configuration. It might be that you need more expensive hardware.

A few anecdotes from a few more projects:

  • Once upon a time, a team of programmers developed a web app. It was a bit slow, and the customer was concerned about its performance. Since one of the guys was reading a lot of articles about the benefits of parallel computing, the team decided to optimize the app this way. Then, they spent a few months fixing bugs which appeared when the code, which was never intended for multithreading, was run in parallel. Unfortunately, the performance didn't improve. So the guys, who were complaining for the past six months that they don't have the hardware they deserve, used this opportunity to obtain octa-core processors, replacing the old quad core ones, but the application seemed to become even slower.

    Finally, a developer from another team figured it out. The problem was not the CPU, but the memory (especially because of a high number of unnecessary memory allocations). However, when the application was moved to octa-core CPUs, the application was also reconfigured, so it was spending more time creating threads and communicating between threads, making it indeed slower.

  • One of my projects was terribly slow, while the application wasn't even using that much the CPU or the memory: even a simple “Hello World” request would take about two seconds. After a while, I found that the problem was due to the fact that a bug in the application, mixed with a completely crazy configuration of IIS forced IIS to recycle the application pool after every request, making the application indeed quite clumsy. Fixing the bug and setting the configuration properly helped reducing the requests to a few dozens of milliseconds.

2

You're probably not going to like this answer, but... Getting from four seconds to two seconds might require an architectural overhaul. Essentially, this is probably as fast as your Java Spring/Hibernate app running in Tomcat 8.5 is likely to get.

Example:

ASP.NET Web Forms application running on IIS and SQL Server: 4 to 7 seconds per page.

Same app written in ASP.NET MVC running on IIS and SQL Server (like what Stack Overflow does): 1 to 3 seconds per page.

3
  • But what distinguishes ASP.NET WebForms from MVC? Are you saying some technologies are flawed? What about Spring/Hibernate is flawed - is it the ORM?
    – gene b.
    Nov 5 '19 at 20:43
  • 2
    But what distinguishes ASP.NET WebForms from MVC? -- Pretty much everything. It's essentially a completely different architecture. Nov 5 '19 at 20:49
  • 2
    While I think Robert is on to something here I hasten to add that you don't have to completely overhaul before you can test if he's right. Just mock up a little test of your most typical use case and see what difference it makes. One good test is worth 1000 expert opinions. Nov 5 '19 at 20:51
1

The first step in debugging performance issues is understanding the bottlenecks in your architecture. In a monolithic application (i.e. traditional n-tier development), you have a few main potential bottlenecks:

  • CPU Load
  • Memory utilization
  • Network bandwidth
  • Disk speed

If your application is taking 4 seconds to respond to a request, look at these three things to find what is maxed out. Each item can have different causes:

  • CPU Maxed out:
    • Look for inefficient algorithms by looking for hotspots in a profiler
    • Increase your CPU to a faster model (pretty limited with this option today)
  • Memory maxed out:
    • Make sure you are not swapping to disk, that is very slow
    • Increase memory either to the JVM or to your server
    • Use a memory profiler to find memory leaks (i.e. memory that is not reclaimed after garbage collection that should be.
  • Network maxed out:
    • Add another network card?
    • If in a VM environment, collocate your database VM with your app VM to take advantage of the high speed inter-VM networking
  • Disk maxed out:
    • Make sure you have indexes on your database to prevent full table scans
    • Minimize disk use
    • Get faster disks (SSDs might be worth the investment, or use RAM disks for temporary files)

If none of these are maxed out and you still have performance problems, there is a good chance you might be suffering from resource locking. If one person is updating a table while another person is querying it, there is a chance that the record(s) being updated will cause the other person to wait. If you can deal with dirty reads, you can reduce the locking overhead in your database.

Bigger Guns

You will inevitably hit a ceiling of what you can do with bigger hardware in a monolithic environment. At that point you really need to think about scaling out. Stack Overflow does a great deal to allow the system to scale out and remain performant. You can go a full microservice route, or just host your monolithic application on multiple servers.

The main thing you must strive for to enable scaling out is to completely avoid server side sessions. In a "shared nothing" environment, there is no reason to have server sessions. The information that would have gone into session variables either goes in the database, or they are stored in the front-end code in the browser.

You'll want to start working with clusters. The database with multiple cluster nodes can spread the work across each node to smooth out the load. The set of application servers can simply host additional copies of the app, and user a load balancer in front. Without any need for session affinity, you can use simple round-robin balancing which is fast.

Next you'll need to look into caching servers like Redis or some equivalent. If your resources take time to put together, but don't change often, this is the final piece of the puzzle to make the response times very fast.

If you think of any quick responding site on the network, I guarantee that they have invested heavily in scaling out rather than scaling up. The degree that they've done so is very different from each environment. For example, Stack Exchange has been able to do a lot with a hybrid monolithic architecture at it's core, but they are hosting on 9 different web servers (reference).

The bottom line is that it costs a lot of effort, and your daily hosting costs increase, when you invest in the type of changes needed to scale out.

5
  • Yeah StackOverflow/Exchange is definitely an example of a site done right. Scaling out is important, but if there are constraints and you only get 1 Web Server and 1 DB Server, is it a good idea to avoid the Hibernate ORM overhead and do straight JDBC? I'm thinking of small-scale immediate game changers.
    – gene b.
    Nov 5 '19 at 22:15
  • 1
    I'd take the time to get smart on Hibernate. Out of the box, Hibernate will get a lot more data than you need because it is "more correct". You can tell Hibernate you are only interested in certain fields, or to lazy load records that are associated with the one you are querying. You might just be loading a lot more data than you expected, and that alone will save you a lot of time per request. Nov 5 '19 at 22:19
  • 1
    You will find that hand crafted JDBC calls are going to be faster than Hibernate, but a lot more work to maintain--and you have to be really good about closing your resources (responses, statements, connections, etc.). I would probably leverage JdbcTemplate in Spring if you handcraft your queries. That way you keep connection pooling and a lot of the resource management is taken care of for you. Nov 5 '19 at 22:23
  • Yeah, I was thinking of Spring JdbcTemplate for all the infrastructure, but the queries are our own. I have no idea how Hibernate and ORM has gotten such a monopoly on the industry. These are bad tools that make applications worse, when it comes down to it.
    – gene b.
    Nov 5 '19 at 22:25
  • ORMs aren't bad, but there are trade-offs. You can make Hibernate or other ORM to be more efficient, but it requires a bit of study. They do come in handy when you are doing simple CRUD apps. Nov 6 '19 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.