11

I am currently researching approaches for moving our application to Docker containers and stumbled upon a question to which I could not find a clear answer.

Our application has several separate databases that are currently hosted in one database server. When moving to Docker should we keep the architecture similar (i.e. one container with all databases) or should we use one container per database?

The latter approach seems more "docker" to me. Similarly to not hosting 2 applications in one container, it seems to make sense to also not host 2 databases in one container.

Are there any established best practices? Does it depend on the parameters of the databases in question (size, access frequency, etc.) or the used database server (SQL server, PostgreSQL, etc.)?

As far as I can tell the "container per DB" approach gives more flexibility (e.g. enforce memory limit per DB) at the cost of more overhead (i.e. the database server overhead is incurred once per database instead of just once in total). Are there any other advantages/disadvantages I should consider?

  • 2
    One container per database and connect them through docker networks if needed. You could wrap the whole logic into a single docker-compose file to ease the process. – Milan Velebit Feb 28 at 12:50
8

Last time I checked it is not recommend to run databases in docker.

Simply put docker is designed to be a stateless container that you can spin up and take down as required. Where as Databases are very state-full indeed!

With a naive docker database approach you would lose all your data if the container crashed. If you span up a new instance you would get a blank database.

This might be ideal for development environments, but its very bad in production.

Now you can do some clever stuff with volumes, but you really have to ask yourself why you are attempting this thing. Databases are generally very mature products, with various backup, fail-over and high availability options built in. Generally you don't want to run them in containers as they already have the concept of containers built in.

| improve this answer | |
  • 1
    The concept of Volumes and data containers has increased in formality and reliability these days. That's one of the mechanisms Kubernetes injects files form Config Maps into a container. Anything that persists data you want later does require mounting a volume. That combined with the way databases are being designed now are much more friendly to containers. However, EBS style volumes are required. – Berin Loritsch Feb 28 at 16:42
  • 13
    There is a difference between hosting the database engine on docker (which should be okay) and hosting the database data on docker (which you shouldn't ever do). A database engine should be fine to be dockerized as long as you keep the actual database data (the .mdf file on SQL Server, for example) on a mounted volume. – T. Sar Feb 28 at 16:43
  • 2
    just as long as you never spin up two looking at the same vol eh? in my view though, even if you can get it to work you havent gained anything. Its no longer containerised – Ewan Feb 28 at 16:47
  • 1
    @JimmyJames Bad things, I suppose. As far as I remember, SQL Server will put a filelock on it, but I've seem cases online where people managed to do eldritch things and attach two engines to the same file, usually causing one of the instances to not work at all with it. – T. Sar Feb 28 at 18:07
  • 1
    @gamma we did a few investigations and had discussions around this a few years back. things maybhave changed. However the answer you link to agrees with me. NOT recommended for production – Ewan Feb 28 at 21:48
5

Containers are ultimately just small wrappers around processes (not machines!) and it is helpful to think about them in terms of that. In this case, each database has its own long-lived master process, and so each probably deserves its own container. This would also help scale to tools like Kubernetes in the future, where the containers could be transparently distributed across a cluster.

Using multiple processes in a container is fine and normal, of course, but usually one process will control the others in the same container. For example, a web server may spawn multiple worker processes, but the root process for the container is responsible for its children.

A corollary is that if you do add multiple database servers to one container, then you will likely have to add extra logic to manage the many master processes. For example, if one Postgres instance dies, you’ll need some way to restart just that one instance. If each database master process has its own container, then Kubernetes or Docker can manage this for free.

Edit: After your clarifying comment, I see that you are referring to database data (the “database” within the application, not the database application!) I think the above reasoning is still helpful framing: a container is just a process separation, and it’s orthogonal from your other storage and partitioning concerns. Any reasons you have for or against using multiple processes for multiple databases apply the same with containers.

I will note that the database data itself should definitely be placed in a volume (and most database images will probably already declare that volume in their Dockerfile.)

| improve this answer | |
  • 1
    Maybe there is a misunderstanding: adding multiple database servers to a single container is not what I was asking. I was asking about multiple databases on one server in the container. – chrischu Feb 28 at 13:53
  • 2
    @chrischu This answer is pretty well aligned to what I would recommend. You can run a server inside a container, but what you really want to do is make that distinction disappear. In other words the container is the server. – JimmyJames Feb 28 at 15:08
  • 7
    Use a mounted volume for the data though. Otherwise if the container dies, so does your data. – Berin Loritsch Feb 28 at 16:44
  • 5
    Please add to this answer that the database data should not be on the docker itself but instead on a mounted volume. It is a very important detail that, if missed, has the potential to end up in pain and tears. – T. Sar Feb 28 at 16:46
3

The database normally use schemas to logically separate unrelated things apart.

I would suggest that you consider moving each schema in its own docker instance with its own persistence volume(s).

Also be aware that Kubernetes may kill the pod without much notice. You need to configure your database accordingly.

| improve this answer | |
0

In answer to the question about "having multiple databases on one server in the container" I would say:

  • As previously mentioned putting a DB in a container is not advised
  • Containers are not persistent, they can be taken down or replicated at any time which creates its own issues
  • A Database per service, micro-service, is desired and helps with encapsulation and security which makes then more modular

So, implement the DB's outside the container environment with One Container/Service talking to One Database

| improve this answer | |
0

From my (granted limited) experience I would examine the setup from four different perspectives:

  • Data (lifetime/priority)

  • Scale

  • Risk

  • Performance

Example: Web shop

Let's assume we have a simple web shop with an inventory (database A), some shopping carts (database B) and a list of orders (database C).

Let's start with the most important data: The list of orders. Here, risk aversion is key, as losing or corrupting this data could potentially ruin the business. At the same time, this might contain PII (personally identifiable information, like addresses or payment data), which also need to be protected.

For this, I'd be conservative and go with a more traditional database server setup, as even with docker volumes a dockerized database server cannot fully control how data is actually stored to disk (too many layers of virtualization involved). Don't introduce any unnecessary risk for vital data storage!

Next up, let's have a look at the shopping carts. The basic concept dictates that this data is volatile; shoppers will add and remove items much more often than finalizing an order, i.e. there will be a lot of insertions, updates and removals. Also, while it shouldn't be a regular occurrence, a loss of this kind of data is at worst a slight annoyance for the user.

Here, I'd pay much more attention to the performance and scale: Since most database operations happen here and the risk of accidental loss of data isn't quite as high, I might be tempted to use dockerized database containers for load balancing purposes, especially at a large enough scale. (This depends a lot on actual requirements, so take this with a grain of salt!)

Finally, the inventory. If it's pretty much just static data that gets updated infrequently (a simple price list), then using dockerized caches that pull from a centralized storage might be interesting if performance becomes an issue. (Of course, the original source for the caches itself would need to be kept in a risk-avoiding manner, similar to orders, for liability reasons.)

However, if the inventory also has to keep track of available stock or is much more dynamic (e.g. real-time stock prices, user content, ...), suddenly scale also becomes a much larger issue. At that point, I'd look towards a traditional database server (or maybe a cluster of them).

TL;DR

There are a lot of factors involved, so there is no easy answer. Generally, I tend to prefer a traditional database server setup, unless the data is either short-lived and has low reliability requirements or pretty much static (caches).

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.