The important thing to understand about virtual memory is that it's, um, virtual.
For example, a process might think it has 20 MiB of virtual memory. However, 5 MiB of that might be part of a memory mapped file that hasn't been loaded from disk, 5 MiB might look like a large area that's full of zeros (but might be the same tiny area full of zeros repeated over and over and over), 5 MiB might be data on the swap partition.
It's all virtual, which means it all looks like RAM is there (even when it's not).
When the process tries to access something that actually isn't there, the kernel is notified (e.g. "page fault") and does whatever it needs to do to maintain the illusion that the RAM was there all along.
For example, if the process reads some data from the area that's actually a memory mapped file, the kernel might load 4 KiB from the file into memory and let the process think the data was always there. If the process writes to that big area full of zeros then the kernel might allocate a new small area (and fill it with zeros) that the process can modify (without messing up other areas of zeros). In the same way, if the process tries to read/write to something that's currently in swap space, the kernel fetches it from swap.
To make this happen, the kernel might need more memory. When it runs out of memory, the kernel can just copy data to disk/swap space and re-use that memory. Of course if a copy of the data is already on disk (e.g. it wasn't modified since it was loaded from disk last time) the kernel doesn't even need to do that - it can just re-use the memory immediately.
So... You might have a process with 2 GiB of virtual space. In that 2 GiB of space the process might only be using 50 MiB of virtual memory. Of that 50 MiB of virtual memory, it might only be 10 MiB of actual RAM (and 40 MiB of "trickery").
Of course (because disk IO is slower than RAM) most of the tricks do effect performance. Kernels try to be smart and try to keep the "most likely to be used" data in memory to minimise the effect on performance. Sadly, predicting the future is hard, so "trying to keep the most likely to be used data in memory" typically ends up being more like "actually keeping the most recently used data in memory".
In any case, the best way to minimise the overhead is to have enough RAM. The more RAM you have, the less trickery the kernel needs to do, and the faster things go. Fortunately, most of the memory that most processes use isn't actually used very often, so (depending on the process) you can get by quite well with only half the RAM.
Also note that (in general) worse performance might be undesirable, but it can be a whole lot better than "process crashed due to out of memory".