Do you feel there is a trade-off between writing "nice" object
orientated code and writing very fast low latency code? For instance,
avoiding virtual functions in C++/the overhead of polymorphism etc-
re-writing code which looks nasty, but is very fast etc?
I work in a field which is a bit more focused on throughput than latency, but it's very performance-critical, and I'd say "sorta".
Yet a problem is that so many people get their notions of performance completely wrong. Novices often get just about everything wrong, and their entire conceptual model of "computational cost" needs reworking, with only algorithmic complexity being about the only thing they can get right. Intermediates get a lot of things wrong. Experts get some things wrong.
Measuring with accurate tools that can provide metrics like cache misses and branch mispredictions is what keeps all people of any level of expertise in the field in check.
Measuring is also what points out what not to optimize. Experts often spend less time optimizing than novices, since they're optimizing true measured hotspots and not trying to optimize wild stabs in the dark based on hunches about what could be slow (which, in extreme form, could tempt one to micro-optimize just about every other line in the codebase).
Designing for Performance
With that aside, the key to designing for performance comes from the design part, as in interface design. One of the problems with inexperience is that there tends to be an early shift on absolute implementation metrics, like the cost of an indirect function call in some generalized context, as though the cost (which is better understood in an immediate sense from an optimizer's point of view rather than a branching point of view) is a reason to avoid it throughout the entire codebase.
Costs are relative. While there is a cost to an indirect function call, e.g., all costs are relative. If you're paying that cost one time to call a function which loops through millions of elements, worrying about this cost is like spending hours haggling over pennies for the purchase of a billion dollar product, only to conclude not to buy that product because it was one penny too expensive.
Coarser Interface Design
The interface design aspect of performance often seeks earlier on to push these costs to a coarser level. Instead of paying runtime abstraction costs for a single particle, for example, we might push that cost to the level of the particle system/emitter, effectively rendering a particle into an implementation detail and/or simply raw data of this particle collection.
So object-oriented design doesn't have to be incompatible with designing for performance (whether latency or throughput), but there can be temptations in a language that focuses on it to model increasingly teeny granular objects, and there the latest optimizer can't help. It can't do things like coalesce a class representing a single point in a way that yields an efficient SoA representation for the memory access patterns of the software. A collection of points with the interface design modeled at the level of coarseness offers that opportunity, and allows iterating towards more and more optimal solutions as needed. Such a design is designed for bulk memory *.
* Note the focus on memory here and not data, as working in performance-critical areas for a long time will tend to change your view of data types and data structures and seeing how they connect to memory. A binary search tree no longer becomes solely about logarithmic complexity in such cases as possibly-disparate and cache-unfriendly memory chunks for tree nodes unless aided by a fixed allocator. The view does not dismiss algorithmic complexity, but it sees it no longer independently of memory layouts. One also starts to see iterations of work as being more about iterations of memory access.*
A lot of performance-critical designs can actually be very compatible with the notion of high-level interface designs that are easy for humans to understand and use. The difference is that "high-level" in this context would be about bulk aggregation of memory, an interface modeled for potentially large collections of data, and with an implementation under the hood that may be quite low-level. A visual analogy might be a car that's really comfortable and easy to drive and handle and very safe while going at the speed of sound, but if you pop the hood, there's little fire-breathing demons inside.
With a coarser design also tends to come an easier way to provide more efficient locking patterns and exploit parallelism in the code (multithreading is an exhaustive subject that I'll kind of skip here).
Memory Pool
A critical aspect of low-latency programming is probably going to be a very explicit control over memory to improve locality of reference as well as just the general speed of allocating and deallocating memory. A custom allocator pooling memory actually echoes the same kind of design mindset we described. It's designed for bulk; it's designed at a coarse level. It preallocates memory in large blocks and pools the memory already-allocated in small chunks.
The idea is exactly the same of pushing costly things (allocating a memory chunk against a general-purpose allocator, e.g.) to a coarser and coarser level. A memory pool is designed for dealing with memory in bulk.
Type Systems Segregate Memory
One of the difficulties with granular object-oriented design in any language is that it often wants to introduce a lot of teeny user-defined types and data structures. Those types can then want to be allocated in little teeny chunks if they're dynamically allocated.
A common example in C++ would be for cases where polymorphism is required, where the natural temptation is to allocate each instance of a subclass against a general-purpose memory allocator.
This ends up breaking apart possibly-contiguous memory layouts into little itsy-bitsy bits and pieces scattered across the addressing range which translates to more page faults and cache misses.
Fields that demand the lowest-latency, stutter-free, deterministic response are probably the one place where hotspots don't always boil down to a single bottleneck, where tiny inefficiencies can actually genuinely kind of "accumulate" (something a lot of people imagine happening incorrectly with a profiler to keep them in check, but in latency-driven fields, there can actually be some rare cases where tiny inefficiencies accumulate). And a lot of the most common reasons for such an accumulation can be this: the excessive allocation of teeny chunks of memory all over the place.
In languages like Java, it can be helpful to use more arrays of plain old data types when possible for bottlenecky areas (areas processed in tight loops) such as an array of int (but still behind a bulky high-level interface) instead of, say, an ArrayList of user-defined Integer objects. This avoids the memory segregation that would typically accompany the latter. In C++, we don't have to degrade the structure quite as much if our memory allocation patterns are efficient, as user-defined types can be allocated contiguously there and even in the context of a generic container.
Fusing Memory Back Together
A solution here is to reach for a custom allocator for homogeneous data types, and possibly even across homogeneous data types. When tiny data types and data structures are flattened to bits and bytes in memory, they take on a homogeneous nature (albeit with some varying alignment requirements). When we don't look at them from a memory-centric mindset, the type system of programming languages "want" to split/segregate potentially-contiguous memory regions apart into little teeny scattered chunks.
The stack utilizes this memory-centric focus to avoid this and potentially store any possible mixed combination of user-defined type instances inside of it. Utilizing the stack more is a great idea when possible as the top of it is almost always sitting in a cache line, but we can also design memory allocators that mimic some of these characteristics without a LIFO pattern, fusing memory across disparate data types into contiguous chunks even for more complex memory allocation and deallocation patterns.
Modern hardware is designed to be at its peak when processing contiguous blocks of memory (repeatedly accessing the same cache line, the same page, e.g.). The keyword there is contiguity, as this is only beneficial if there's surrounding data of interest. So a lot of the key (yet also difficulty) to performance is to fuse segregated chunks of memory back together again into contiguous blocks that are accessed in their entirety (all surrounding data being relevant) prior to eviction. The rich type system of especially user-defined types in programming languages can be the biggest obstacle here, but we can always reach around and solve the problem through a custom allocator and/or bulkier designs when appropriate.
Ugly
"Ugly" is hard to say. It's a subjective metric, and someone who works in a very performance-critical field will start to change their idea of "beauty" to one that's a lot more data-oriented and focuses on interfaces that process things in bulk.
Dangerous
"Dangerous" might be easier. In general, performance tends to want to reach towards lower-level code. Implementing a memory allocator, for example, is impossible without reaching beneath data types and working at the dangerous level of raw bits and bytes. As a result, it can help to increase the focus on careful testing procedure in these performance-critical subsystems, scaling the thoroughness of testing with the level of optimizations applied.
Beauty
Yet all of this would be at the implementation detail level. In both a veteran large-scale and performance-critical mindset, "beauty" tends to shift towards interface designs rather than implementation details. It becomes an exponentially higher priority to seek "beautiful", usable, safe, efficient interfaces rather than implementations due to coupling and cascading breakages that can occur in the face of an interface design change. Implementations can be swapped out any time. We typically iterate towards performance as needed, and as pointed out by measurements. The key with the interface design is to model at a coarse enough level to leave room for such iterations without breaking the entire system.
In fact, I would suggest that a veteran's focus on performance-critical development will often tend to place a predominant focus on safety, testing, maintainability, just the disciple of SE in general, since a large-scale codebase which has a number of performance-critical subsystems (particle systems, image processing algorithms, video processing, audio feedback, raytracers, mesh engines, etc) will need to pay close attention to software engineering to avoid drowning in a maintenance nightmare. It's by no mere coincidence that often the most astonishingly-efficient products out there can also have the fewest number of bugs.
TL;DR
Anyway, that's my take on the subject, ranging from priorities in genuinely performance-critical fields, what can reduce latency and cause tiny inefficiencies to accumulate, and what actually constitutes "beauty" (when looking at things most productively).
