I am creating some Web Services that would have 2000 concurrent users. The services are offered for free and are hence expected to get a large user base. In the future it may be required to scale up to 50,000 users.

There are already a few other questions that address the issue like - https://stackoverflow.com/questions/2567254/building-highly-scalable-web-services

However my requirements differ from the question above.

For example - My application does not have a user interface, so images, CSS, javascript are not an issue. It is in Java so suggestions like using HipHop to translate PHP to native code are useless.

Hence I decided to ask my question separately.

This is my project setup -

  1. Rest based Web services using Apache CXF
  2. Hibernate 3.0 (With relevant optimizations like lazy loading and custom HQL for tune up)
  3. Tomcat 6.0
  4. MySql 5.5

What are the best practices to abide by in order to make a Java based application scalable?

  • If you are exposing a REST service, using a reverse proxy like Varnish would help a great deal. How fresh does the data need to be? Are you sure you need a relational database? Could you partition the data? With the technology stack you are describing, I would focus on making sure as few as possible requests actually hit your endpoint. Have you looked into doing this in-memory with solutions like Hazel cast/Gigaspaces etc?
    – ebaxt
    Commented Jun 19, 2012 at 6:05
  • @ebaxt thanks for you suggestions. Gigaspaces does seem to be opensource. But Hazel cast looks interesting. Commented Jun 19, 2012 at 6:14
  • 1
    @ebaxt "Are you sure you need a relational database?" Adopting nosql would have drastic changes on application architecture. We're trying to keep complexity to minimum. Cost though is not a factor for us. So we'll stick with relational approach. Commented Jun 19, 2012 at 6:17
  • 1
    You can use Postgres, MySQL or what ever. What's about your infrastructure? Can you use disk-arrays? Are the servers hosted in the same location? Can you connect your cluster with heartbeat etc.? Can you put them in the same subnet?
    – edze
    Commented Jun 19, 2012 at 6:39
  • 1
    I'm a programmer, too. But if your relational database is the bottleneck you will tend to end up with this questions. There are databases in market some perform better than other in some situations. But they are using different default Transaction Isolation Levels and Optimistic Concurrency vs Pessimistic Concurrency etc.
    – edze
    Commented Jun 19, 2012 at 8:12

3 Answers 3


I dealt with the issue in the past, but still feel I have lots to learn on the field. I find this to be one of the most interesting fields there are in software development nowadays, here are some thoughts about this:
MySQL is fair enough database unless you're working with massively huge amount of data, and in this case you might consider NoSQL database, but you should carefully examine what is the best NoSQL database for your needs.

You should implement caching at your system - try to cache as much read-only data as much as possible, or define some caching strategies - for example, we had a scenario in which it was valid for a user to see "old data" as long as the recent update took place in the last hour.
I would consider JBoss Cache, or maybe Infinispan (which is more like a distributed data structure) or other popular caching framework for this.
In addition, as you mentioned tomcat, I assume you work in some request-respone module. Try to consider using a cache that exists in a scope of a given request, this can be even a simple HashMap that is associated with the thread local storage.
My idea here quite resembles to first level cache at Hibernate.

You should remember that files, transactions and other resources are expensive in terms of keeping them open. Make sure you close files and transactions as soon as possible, or you will end up with bugs that will reproduce on large scale setups

In addition you must understand what 2000 concurrent users - does this mean that 2000 users are accessing your server at once, or are they using your system? Distinguish between cases where 2000 users try to open a socket to your server, and a case where only 500 are, and 1500 are currently looking at results, of filling input at client side.

You should consider using clustering - you will have to deal with issues like load balancing, sticky session (which means the load balancer will redirect a request to the same server for the same session) and more.

If you need to have synchronization code - choose synchronization strategy carefully. I saw some systems in which a simple lock was used, but a ReaderWriterLock could have improved things, as most access was read-only.

Consider having client side caching and validation if possible, try to save calls to server, and to send only differences of data, in case most of your response for a request with same parameter does not change.
For example, at oVirt open source project we request to get statistics of a given virtual machine. some of the data of the VM rarely changes, so we send only MD5 of it, if the data changes the MD5 value is changed as well , we perform a request to get the full data, and not just the MD5.

I mentioned hibernate before - I would reocmmend you to carefully consider using it - if you need to perform lots of writes, and less reads, Hibernate might not be ideal for you, and you should consider maybe working with Spring-JDBC as a wrapper over JDBC.

Index your database wisely, and use a correct db scheme. Consider using a layer of stored procedures as they are precompiled and optimized

I would like to state that at past, I dealt with a system (single node) on mysql (mostly read only access) with jboss 4.2.1 and managed to reach 2000 concurrent users
(not accessing at once in terms of opening 2000 sockets against our server), but using/browsing our system, using JBoss Cache and preloading to the cache some of the most accessed data, or data we realized is going to be "hot and popular" but our solution was good for our architecture and our flows,
so as I say in these cases -
There are more tips and tricks , but it really depends on your architecture, and what flows you need to have in your system. Good luck!

  • I agree except for stored procs, don't use stored procs. And you can use a concurrent hashmap and atomic values, to make threadsafe Commented Jun 26, 2012 at 12:33

Good question. Probably hard to say which is best approach, but will try from my experience.

The best way to scale the Java based web application is to write it as stateless as possible (if you could). This allows you to horizontally scale the application, where you can add tomcat servers if there are more concurrent users.

However, as you noted, there could be issue with the database connections. But the question I have is, how are you getting the data? Is it user generated or you get the data from third party? This is very important because, if you are giving a service to your user with the data aggregated from third party application (say FB, Twitter etc), then what you can follow, is write to master database and replicate the data over to slave databases which are allocated to each tomcat instances. Then each tomcat server can get from its own slave database.

 Are there faster alternatives to Mysql?

You can go for MySQL cluster which have in-memory datastore. But beware of the fact that the application may need some changes. The sql joins are not well supported in MySQL cluster though in the latest version there are improvements for the same. If the cost is not a factor, then you can try Oracle.

The caching solution definitely will improve the performance. But then, it all depends on the architecture of the whole application. You should be well aware of when to push data to the cache, when to make it dirty(remove from cache).

Regarding distributing the load in multi server environment, I would suggest you to use load balancer than to use Apache for load balancing.

  • "I would suggest you to use load balancer than to use Apache for load balancing" Which approach/software would you suggest if not Apache? Commented Jun 19, 2012 at 7:56
  • I was basically recommending load balancer hardware, which your network administrator should be able to configure. This ofcourse has additional cost to the project. This load balancer will have its own IP (also called virtual IP) and basically you will be assigning this IP to your domain. When the request comes, this will route it to all connected server in round robin (also other algorithms available) fashion. You can use apache for this purpose if hardware is not an option, but I would prefer hardware as you don't need to tune apache for only this purpose.
    – Chandra
    Commented Jun 19, 2012 at 11:02
  • We are using a dedicated server with httpd to do the same thing. Hardware is not an issue. Commented Jun 19, 2012 at 11:08
  • You can use httpd and mod_cluster, if I remember correctly. I would consider carefully before going to the "overkill" solution of hardware LB, before checking httpd and mod_cluster
    – zaske
    Commented Jun 22, 2012 at 6:10
  • @zaske - You probably are right that hardware load balancer maybe a overkill. But in case you need to scale up, its easy to do by adding more servers.
    – Chandra
    Commented Jun 22, 2012 at 13:08

I'm currently setting up a similar system (on a professional level) and this is the design I've chosen:

  • Two Nginx loadbalancers (both active, both failover for the other, balanced with DNS round robin)
  • Two MySQL Databases in master master replication mode
  • Two Tomcat instances as a tomcat cluster
  • Two Memcached instances for both caching and session state sharing for the Tomcat cluster

This will achieve a redundant, high availability, scalable solution.

The loadbalancers (on decent hardware) will easily loadbalance a saturated 1gbit line each. This is also a great place for SSL offloading.

You can save your session info in memcached. In case a tomcat instance fails, another tomcat instance can retrieve relevant session information and the clients won't notice a thing. Don't forget to combine this with sticky sessions too. (To keep network traffic down)

Tomcat clustering also has an option to share session information among the cluster in real time, without using memcached. Although I think performance wise, using Memcached will be better.

If you need more power in any of these applications:

  • Nginx: Add more loadbalancers, although I don't think this will be the bottleneck very soon.
  • Tomcat: you can easily increase the size of the Tomcat cluster or add more clusters
  • Mysql: Add some read-only slaves or increase the cluster size (depending on your application, but since you wrote a REST-based application, this should not be a problem)
  • Memcached: Add more nodes, Memcached scales pretty well I believe.

I don't know how your application is build and what the big resources hogs are, but if you see a high database load (during your loadtests!), adding a cache between the application and database could certainly improve performance a lot. But don't forget that not everything is cachable, if your queries are always different, caching won't help (much)

My advice would be to download VMware Workbench (or similair virtualization software) and try to create a simple setup. No loadbalancing or clustering, just the basics and work from there. One by one add more features (balancing, caching, clustering, etc.) and make sure to do some research on each topic, so you'll know you made the right pick.

If you keep running the same performance tests during this process, you can see for yourself if using X is better than using Y in your setup, or what impact caching will have, etc.

In the end, a setup like this really depends on the requirements of your application and its clients, everything can be done in various ways, each with it's own strengths and weaknesses.

Any more questions?

Good luck!


  • hazelcast ? hazelcast.com Commented Jun 26, 2012 at 12:28
  • Do you use a framework for the caching layer, or just a bunch of manual hashes on SQL queries?
    – djechlin
    Commented Oct 7, 2013 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.