See also Ron Jeffries's attempt to create a Sudoku solver with TDD, which unfortunately didn't work.
Algorithm requires a significant understanding of algorithm design principles. With these principles it is indeed possible to proceed incrementally, with a plan, like Peter Norvig did.
In fact, for algorithms requiring non-trivial design effort, it is almost always that the effort is incremental in nature. But each "increment", which is tiny in the eyes of an algorithm designer, looks like a quantum leap (to borrow your phrase) to a person who hasn't had the same expertise or knowledge with this particular family of algorithms.
This is why a basic education in CS theory combined with lots of algorithm programming practice are equally important. Knowing that a particular "technique" (small building blocks of algorithms) exists is a long way toward making these incremental quantum leaps.
There are some important differences between incremental progress in algorithms and TDD, though.
One of the difference has been mentioned by JeffO: A test that verifies the correctness of the output data is separate from a test that asserts the performance between different implementation of the same algorithm (or different algorithms vying to give the same solution).
In TDD, one adds a new requirement in the form of a test, and this test shall initially not pass (red). Then the requirement is satisfied (green). Finally the code is refactored.
In algorithm development, the requirement usually doesn't change. The result correctness verification test is either written first, or shortly after a draft (highly confident but slow) implementation of the algorithm is completed. This data correctness test is seldom changed; one does not change it to fail (red) as part of the TDD rite.
However, in this aspect, data analysis is distinctly different from algorithm development, because data analysis requirements (both the input sets and the expected outcomes) are only defined loosely in human understanding. Thus the requirements change frequently on a technical level. This rapid change puts data analysis somewhere between algorithm development and general software application development - while still algorithm-heavy, the requirements are also changing "too fast" to the taste of any programmer.
If the requirement changes, it typically calls for a different algorithm.
In algorithm development, changing (tightening) the performance comparison test to fail (red) is silly - it does not give you any insight about potential changes to your algorithm that would improve performance.
Therefore, in algorithm development, both the correctness test and the performance test are not TDD tests. Instead, both are regression tests. Specifically, the correctness regression test prevents you from making changes to the algorithm that will break its correctness; the performance test prevents you from making changes to the algorithm that will make it run slower.
You can still incorporate TDD as a personal working style, except that the "red - green - refactor" ritual is not strictly necessary in nor particularly beneficial to the thought process of algorithm development.
I would argue that algorithm improvements actually result from making random (not necessary correct) permutations to the data flow diagrams of the current algorithm, or mixing and matching them between previously known implementations.
TDD is used when there are multiple requirements that can be added incrementally to your test set.
Alternatively, if your algorithm is data-driven, each piece of test data / test case can be added incrementally. TDD would also be useful. For this reason a "TDD-like" approach of "add new test data - improve code to make it handle this data correctly - refactor" will also work for open-ended data analytics work, in which the objectives of algorithms are described in human-centric words and its measure-of-success also judged in human defined terms.
It purports to teach a way to make it less overwhelming than trying to satisfy all (dozens or hundreds) of requirements in a single attempt. In other words, TDD is enabled when you can dictate that certain requirements or stretch-goals can be temporarily ignored while you are implementing some early drafts of your solution.
TDD isn't a substitute for computer science. It is a psychological crutch that helps programmers overcome the shock of having to satisfy many requirements at once.
But if you already have one implementation that gives correct result, TDD would consider its goal accomplished and the code ready to be handed off (to refactoring, or to another programmer-user). In some sense, it encourages you not to prematurely optimize your code, by objectively giving you a signal that the code is "good enough" (to pass all correctness tests).
In TDD, there is a focus on "micro-requirements" (or hidden qualities) as well. For example, parameter validations, assertions, exception throwing and handling, etc. TDD helps assure the correctness of execution paths that aren't frequently exercised in the normal course of software execution.
Certain types of algorithm code contain these things as well; these are amenable to TDD. But because the general workflow of algorithm isn't TDD, such tests (on parameter validations, assertions, and exception throwing and handling) tend to be written after the implementation code has already been (at least partially) written.
