True cost of virtual dispatch in C++ - when stop using it?

Question

What I've learned so far as a programmer has lead me to think that the easiest way to write large scale software is to use a lot of interfaces - this makes things very easy to isolate and test. In high performance code, the performance loss due to v-table look-ups is often stressed. I often see this mentioned when discussing game engines, databases ect.

My question is, since virtual functions are so convenient - at what point should we stop using them in order to gain performance? I have seen much divided opinion online on the matter. Can anybody share some wisdom?

Ps. I'm sorry if the question is too vague - from my perspective this is unknown uknown territory, so it's hard to ask the right questions.

Answer 1

Sure, virtual calls have some runtime overhead. But when you use virtual functions, it's usually not a choice but a consequence of the problem you are solving. If you need to call a function on a type-erased value there isn't a real alternative, unless we count some C-style void* approaches.

This runtime overhead is typically just a memory load to retrieve the vtable pointer, and another load to read the vtable entry for the indirect call. There are cache implications, though at least the vtable has a good chance of being cached when performing multiple operations with the same type.

One level of indirection can be removed by using a fat pointer design where the vtable pointer is stored alongside the data pointer, instead of within the data. While this strategy is used by Go and Rust implementations, it's unsuitable for C++ implementations. You could do this yourself (by modelling the vtable as a struct of function pointers) but you do give up a lot of typesafety regarding inheritance.

In some rare cases, usage of inheritance and virtual functions is actually unnecessary in C++. This is a multi-paradigm language, and the OOP paradigm is not necessarily the most suitable. E.g. when we need polymorphism but all types are statically known, a template- or overload-based solution might be more appropriate. But that really depends on the problem, you can't categorically replace virtual calls. There are also significant tradeoffs. While templates tend to allow for better optimizations (can't optimize a across virtual calls), they also tend to lead to bloated and redundant code that is cache-unfriendly.

So: please don't worry about virtual calls. They are a really good feature, and likely a lot faster + safer than DIY fixes. Instead, consider whether you have a suitable data model, e.g. whether it is appropriate to use type erasure at all. Other optimization approaches like profile-guided optimization or link-time optimization are likely to have a much larger overall effect, unless you have a ton of virtual calls in a very very hot loop.

If you are going to descend into performance paranoia, please also consider that a dynamically linked function call has similar performance implications to a virtual call. Yet, I don't see a lot of people arguing that you should stop using dynamically linked libraries for performance reasons.

Answer 2

You suggest to stop using virtual functions at some point, but don't suggest an alternative, so that makes the question of "at what point to stop" hard to answer.

Virtual functions are not simply optional or "nice/fun to use but easily avoidable" — they solve a certain set of design problems, and so should be compared with alternatives that address the same design problem space.

---

There are many aspects to good performance; here's just two of them:

• Cache utilization is one key to large program performance — cache misses are costly, and large programs can thrash the caches.  When that can be avoided performance improves.  You might look into cache friendly and cache aware data structures.

• Another key is eliminating conditional operations — modern processors do better if they run the same operations repeatedly, rather than alternating operations with conditional logic.  You might read up on removing conditional logic and branch prediction.

---

In the game programming industry, they have developed an approach that addresses both of these issues together, called entity-component-system, which seeks to group related data together using composition, over inheritance in class hierarchy.

By using composition instead of inheritance, usage of virtual functions is reduced.  Between the grouping of related data, and separating operational concerns into different systems, they get cache benefits as well as removing conditionals, while still having regular and maintainable architecture.

---

If you get into multi-threading on multi-core processors, performance can be hampered when multiple processors (cores) alternately write to the same memory location (cache line), because a write requires exclusive access to that cache line (see MESI), and transferring exclusive access from one processor to another is costly (like a cache miss).  Performance improves if only one processor writes to any given memory area, so such transfers can be avoided.  So, one approach there is to redesign shared data structures so that each processor has an area of memory to write, and only reads the other processors' write area/data.  For more information on that, see LMAX disruptor, whose team champions this approach.

Answer 3

You should stop using virtual calls if:

1. This leads to a measurable speed improvement.

2. For in-house software: Cost savings due to the speed improvement demonstrably outweigh the development cost.

3. For software that you are selling: Speed improvements lead to increased sales so the profits demonstrably outweigh the development cost.

4. There are no other possible improvements that produce the same speed improvement at lower cost, or more speed improvements at the same development cost.

When you calculate development cost, take into consideration cost for testing, bug fixing, and maintenance.