This is a big question, but the algorithms fall broadly under the category of signal processing.
In short, there are a couple of things that make a voice stand out (or any other sound, for that matter, though I will call every sound a voice for simplicity's sake). They are pitch, timbre, and loudness.
Pitch is probably the most familiar, but specifically it refers to the frequencies occupied by a voice. Most people voices have a uniform pitch, which is to say I've never heard of a single person singing a chord.
Timbre is what makes the difference between a saxaphone and a violin playing the same note. It's like the flavor of the sound, and can be affected by the acoustics of the room, the noise (is it breathy or raspy), resonance, etc.
Loudness is like the average pressure of the air's movement. It's surprisingly complicated, but for my explanation a volume knob works.
Okay? Now, to isolate a sound, we can try and trace these three factors. It's pretty rare for a voice to change pitch, timbre, and loudness simultaneously. So we'll call a dramatic change in two of these a different voice.
A fourier transform of your audio signal can give you a "frequency-loudness over time" view of it. So, if you look at a fourier transform (the graph at the bottom of the following image)
you can begin to get an idea of how this can be carried out. Once you know where a voice lies along the spectrum in time, you can apply a moving band pass filter to it to isolate that signal.
Still, this is something that has taken a long time for people to get right. Siri is just the latest in a long line of voice recognition applications of varying ability.
