# Why don't audio engines already process audio more realistically?

Generally game audio engines use a rather simple system for rendering audio: 1. position an audio source in space 2. apply effects on the audio clip depending on the area the object is in 3. shift volume / phase / pan depending on the angle and distance to the listener

But in real life audio propogates in waves through every medium and is reflected too. For example: If you yell into a wall, you will you much louder than if you yell to the horizon (crude example, I know).

So my idea is that you build an audio engine that works rather like the raytracing in graphics: Each pass (100 times a second --> 441 samples)

1. You have an audio source sending Dry (meaning uneffected) sound in a fixed amount of directions, evenly distributed on the angle sphere.
2. Every ray propogates for a limited amount of time through space.
3. If one hits a wall a new (smaller) set of rays is created and propogates with less of an amplitude (depending on properties set to the wall)
4. When this process is done and all rays reached their max length, each ray is treated as a linear function and the minimal distance between the ray and each listeners channel (for example L/R) is calculated.
5. Phase shift and amplitude of each ray is now calculated in the following was

The amplitude is anti-proportional and the phase shift is proportional to the distance the ray travelled as well as the distance between the ray and the listener while this distance becomes less effective the further the ray has travelled.

I'm sorry if that wasn't clear enough - maybe this pseudocode may help you understand what I mean.

``````// Code example in C# style
// C++ would be more performant but this is easier to get

// Base classes for clarity
class AudioRay
{
Vector3 Source;
Vector3 Direction;
float DistanceOffset;
float MaxLength;
float Volume = 1;

public AudioRay(Vector3 src, Vector3 dir, float off, float maxLen, float vol);

public struct Intersection
{
Vector3 Point;
Vector3 InAngel;
Vector3 OutAngle;
float MaterialMuffling;
}

public Intersection GetIntersection()
{
// Todo - just concept - I don't really know how to calculate that
}
}

static class RayCaster
{
static Vector3 GetNearestPoint(Vector3 targetPoint, Vector3 source, Vector3 direction)
{
// Nearest point on a line to a point...
}

static List<AudiorRay> Cast(AudioRay ray, float maxLength)
{
List<AudiorRay> rays = new List<AudiorRay>();
AudioRay.Intersection intersection = ray.GetIntersection();

if(intersection!= null) // Only cast new rays when the old one intersected with a wall
{
float len = ray.DistanceOffset + (intersection.Point - ray.Source).Length;
if(len < maxLength) // ... and if they are below the max length
{
List<AudioRay> newRays = new List<AudioRay>();

// Add the new rays --> This is only the central one but the others will be evenly distributed
newRays.Add(new AudioRay(intersection.Point, interse.OutAngle, len, maxLenght, vol * intersection.MaterialMuffling));

// Cast the new rays but
}
}
}
}

// Main class
class AudioSource
{
Vector3 Position = new Vector3(0, 0, 0);
Vector3 ListenerPos = new Vector3(10, 20, 30);
int StartRayCount = 162;
int BounceRayCount = 11;
float BounceRayAngle = 25;
float MaxRayLength = 1000;

// The dry wave parameter is not the buffer but the whole file so we can adress
// every sample all over the file.
// I suspect a 512 sample out buffer (~12 ms)
// wavePositionOffset = position in the track
public float[] Pass(float[] dryWave, int wavePositionOffset)
{
List<AudioRay> rays = new List<AudioRay>();

int samples = StartRayCount;
float offset = 2.0f / (float)samples;
float increment = Math.Pi * ( 3.0f - Math.Sqrt(5.0f));

// Create source rays
for(int i = 0; i < samples; i++)
{
float y = ((i * offset) - 1f) + (offset / 2f);
float r = Math.Sqrt(1f - Math.Pow(y, 2f));
float phi = (i % samples) * increment;
float x = Math.Cos(phi) * r
float z = Math.Sin(phi) * r

AudioRay ray = new AudioRay();
ray.Source = Position;
ray.Direction = new Vector3(x, y, z).Normalized;
ray.DistanceOffset = 0f;
ray.MaxLength = MaxRayLength;

}

// Create the out buffer
float[] oB = new float[512]();

float distanceVolumeFalloff = -0.01;    // Complete silence after 100 m
float phaseShift = 20;                   // 20 samples / m shift

// calculate all rays phases and volumes
foreach(AudioRay ray in rays)
{
for(int i = 0; i < 512; i++)
{
Vector3 nearestPoint = GetNearestPoint(ListenerPos, ray.Position, ray.Direction);
float dist = ray.DistanceOffset + (nearestPoint - ray.Position);

// vol can't go below 0
// Respect the material properties stored in RayVolume
float vol =  Math.Max(1 - dist * distanceVolumeFalloff, 0f) * ray.Volume;

int sampleOffset = (int)(dist * phaseShift);

int pos = wavePositionOffset-sampleOffset
if(pos > 0 && pos < dryWave.Length)
oB[i] +=  dryWave[pos] * vol;
}
}

return oB;
}
}
``````

My question here is: Why don't engines support this audio model yet? I don't think there would be too much CPU overhead since reverberation is essentially the same and I "implemented" performance values (for example StartRayCount and MaxRayLenght). The only thing that might be (performance wise) hard to do is the raycasting. But your GPU does this millions of times and were talking maybe several hundreds say 100 times a second. This is still way less than what a 3D video scene does.

Pixels / second: 1920 x * 1080 y * 60 FPS = 124,416,000 only Full HD - Quadrupel for 4K

Rays / second: say about 200 to 1000? * 100 rays/second = 200,000 to 1,000,000

Before anyone cries out loud: I did my research and I've read this.

• Why don't you make one that does this and sell it to game companies? Oct 6, 2016 at 15:30
• It's probably just too much work. If you know anything about graphics, you'll know a lot of it is just an "illusion" and a LOT of shortcuts are taken to make it look the way it is, even if it does look good. You can reuse a textures on different things, but you can't really reuse sound that much in this case: Think of a closed can and an open can. They make radically different sounds, and both would need to be programmed in. You can reuse a can texture and the same model but the sound would need to be unique for every unique element in the scene. Oct 6, 2016 at 15:31
• Sounds like overkill to me Oct 6, 2016 at 17:23
• The ressources of the game developers would be pretty much the same I think. Since there are low poly meshes for physics you can reuse them for audio and as you might have seen in my code above I don't think it's too much overhead. But someone would need to code it at least partially for my because I struggle with raytracing and intersecting restricted planes (tris) with rays in an optimized manner. Oct 7, 2016 at 2:09
• You might find Physical Modeling Synthesis interesting. It's used for music production, but there's no reason it couldn't also be used in games. Jan 27, 2018 at 2:16

There's no technical reason why any of this couldn't be done. Processing of this depth is already done on the images the game displays (although not in quite the same way).

The problem is not so much the processing... it is reproducing sound in a way that is realistic and that the human nervous system will interpret to be authentic.

Consider an ordinary sound system. The theory is that the sound from the left speaker goes to the left ear, and the sound from the right speaker goes to the right ear:

Already your system is compromised, in terms of realism. Why? Because some of the sound from the left speaker reaches the right ear, and some of the sound from the right speaker reaches the left ear.

So you have to use headphones. Headphones isolate the sounds from each transducer, so that the left ear only gets sounds from the left channel, and the right ear only gets sounds from the right channel.

Now you have a new problem: how do you model changes in orientation? With monitor and headphones, you cannot do so realistically. Instead of your head rotating to meet the world, the sonic landscape rotates around you, while the visual landscape remains stationary. Your nervous system will not interpret this as fully authentic.

Systems like Oculus Rift and Microsoft Hololens solve both these problems by putting the transducers (the monitor and headphones) on your head and detecting orientation, so that when you move your head, the imagescape and soundscape rotate realistically around you.

What about sounds from above and below? Your ears can tell where these sounds are coming from because of the shape of your outer ear. So to reproduce up and down, you either have to find some way to mathematically reproduce the sound contours of your outer ear, or make headphones that will provide the sound from the proper direction. The Ossic X attempts to solve this problem by calibrating itself to your anatomy.

So processing is not the problem; we're waiting on the widespread availability of devices that make the experience more immersive and realistic (for both audio and video).

• "The theory is that the sound from the left speaker goes to the left ear, and the sound from the right speaker goes to the right ear" That suggests that we aren't even trying to model the real world. In real life, sounds from one side can be heard with both of our ears. Oct 6, 2016 at 15:42
• @8bittree : right ! and the phase shift between the two ears is essential to help us locating the source. Oct 6, 2016 at 15:49
• @8bittree: Of course, but that modelling already occurs in the sound stream, so the only way to translate that to realistic sound properly is to isolate the left and right channels. Speakers don't do this for a given point in the audio space, because you have two sound sources, not one. Oct 6, 2016 at 15:51
• My original post was originally mean for headphones. But even with Speakers there is no reason why you couldn't have "real" reverberation dependent on the position of the listener and the properties of his surroundings. So I guess you partially misunderstood me: My Idea is dependent on the environmental aspects of the "real"ism. I think the part of stereo/surround separation and binaurality is after that not too hard. Especially if you work with two audio channels (two ears) and model a head with said properties inbetween. The idea is to simulate that. Oct 7, 2016 at 2:00

Actually, there are game sound engines that do what you propose.

https://etiennedeleflie.net/2007/08/30/interview-with-simon-goodwin-of-codemasters-on-the-ps3-game-dirt-and-ambisonics/

and another one here http://www.codemasters.com/research/3d_sound_for_3d_games.pdf

They include directionality, reflection and basically a "vector-based" sound tracing.

Very obviously, focus on "smooth graphics" is typically much higher than audio - On the one hand, not all consoles have (and in the PC world, the users don't own) the proper hardware for real surround sound (see the XBox in the linked article), on the other hand, there's a constant struggle between the sound and graphics guys for CPU - And, graphics apparently "pays better". Customers still seem to be buying games more for visual than audible quality.

• Thanks for that! Yeah I get what you mean.... One can simply advertise images better than audio. But as a developer I think the room effects in most game engines are really just bodges to make life for the engine devs easier. And thats unfortunate Oct 7, 2016 at 2:02