Different ways to provision your servers have different tradeoffs. We have control, performance, isolation, and a focus on complete systems on one end of the spectrum, with convenience, higher-level abstractions, and a focus on individual applications on the other. Xen and Docker are almost on the different ends of this spectrum, so it doesn't make too much sense to compare them. Importantly it's not either–or, but likely both.
Let's look at different ways to manage/install/provision your server systems. We'll reach across bare-metal, hardware-level virtualization, OS-level virtualization (containers), and application-level configuration management.
You could install and configure the server software directly on the server hardware. This requires physical access to the servers, but may actually be preferable and more secure for normal deployments. You have full control of the server. We have one OS per hardware server.
Most server-grade systems come with an integrated management interface in their firmware. This can be used to install a server over a network, without physical access. The server can also be configured to boot from an image stored on the network which makes it feasible to uniformly administer many servers. We still have on OS per hardware server.
Instead of installing a normal operating system we could install a hypervisor like Xen. So when the server is booted, it will actually load the Xen kernel. Xen can run other OSes in virtual machines. This allows us to have multiple isolated OSes per hardware server. Benefits of Xen and other virtualization technologies:
- Because the virtualized OSes are isolated from each other and from the host OS, it is possible to securely share the same hardware server between multiple organizations.
- If each OS will only consume few resources, running multiple small systems on the same hardware server leads to better utilization of the available resources.
- Xen also makes it possible to transfer running virtual machines between physical hosts.
Virtualization can also be used on top of an ordinary host OS, though a full host OS will consume more resources. This may still be sensible since it may be easier to administer for single-server deployments.
Containers do not virtualize the whole operating system, but only the user space. All containers on a system still share the same kernel. While this should still provide sufficiently secure isolation, the shared kernel means this isolation is significantly weaker than with a full virtual machine. As a notable drawback, a single container can't use a kernel that is tuned for its task. Instead, the host OS must use a kernel configuration that balances the requirements of all containers.
So unless you're fiddling with kernel-level stuff, a container may be indistinguishable from a normal virtual machine.
Because containers are managed by the OS they also have many advantages over hardware level virtualization:
- Container images can often inherit from another image, which means that individual images are very small, down to a few MB. So once the base images are installed, containers can be copied around very quickly.
- When each container is specialized for a particular application, non-essential parts of the user-space can be removed. That reduces the footprint compared to a conventional virtual machine.
- Containers can be started and suspended quickly. It can often be avoided to do a full boot of the container's OS.
- The host OS can implement fine-grained quotas, e.g. regarding CPU usage or I/O priorities.
- Versioning, cloning, and copy-on-write: new containers can be created so cheaply that they can be treated as disposable.
- Many container implementations offer convenient, high-level abstractions that are easily scriptable.
Docker is an abstraction layer on top of containers. It makes it easier to create, manage, and run container images. Docker is geared towards packaging a single stateless application per container, in contrast to virtualizing a complete OS. The main advantage of Docker is the large ecosystem of related tools.
At this point we also have the answer to your main question: the convenience of Docker and the restrictions of Docker are two sides of the same coin, without them we have ordinary OS-level virtualization/containers.
When packaging applications, the main focus lies on ensuring that the application runs in an appropriate environment, with the necessary configuration, and all required dependencies. Instead of creating a container where all this is provided, we could also bundle all resources into a self-contained archive and deploy it, e.g. as used by .snap packages or Java EE .war files. Those packages are effectively mounted as a read-only file system, then executed as a normal process. As a similar example, many languages can statically link their dependencies so that no dependencies are necessary at run time.
Instead of bundling all dependencies, the dependencies can be made explicit and get resolved during installation. There's a long list of language- and OS-specific package managers, for example APT on Debian and Pip for Python. Additionally, there are configuration management tools that bring a server system into a particular state. Dependencies are now less fixed. As a notable advantage, this makes it easier to (automatically) install security updates for dependencies.
That was a pretty long list, because this is a large topic with many very good solutions. The point is, these solutions address a spectrum of problems. There is some problem-space overlap for Xen and Docker: running multiple instanced of an isolated, virtualized operating system on the same hardware.
But the proposed solutions are so different that you may reasonably combine them, i.e. run a server which runs Xen which runs an OS which runs Docker which runs your application. You might eliminate Docker from that stack if you don't want Docker-style image deployment, or might eliminate Xen if the server only runs a single OS.
Note that if the problems you are addressing are less about system configuration and more about reliability and hardware utilization, then the abstractions provided by containers are a bad fit. E.g. a production database is a kind of system that benefits from dedicated hardware, and not as much from Docker-style deployment and container isolation.
When you view Docker as a way to package applications, then the comparison to hypervisors like Xen ceases to make any sense. Instead, we now have to compare Docker against solutions like package managers where we have a similar “not either–or but probably both” relationship.