Post Made Community Wiki by JeremyKun
source | link

It depends on the area of mathematics he is interested in.

If he is more into Abstract Algebra or Combinatorics, he would probably like Haskell as the complex type system allows for direct programming of a variety of mathematics. ATS might also be an option.

I'd recommend Learn You a Haskell for a Great Good for an intro tutorial and Purely Functional Data Structures for Algorithms.

If he is more into Matrix Analysis, Differential Equations or Dynamical Systems, I'd recommend Matlab, NumPy, or Octave. APL is via Dyalog might also be of interest; though I'd not start him on that. Numerical Recipes, while in C++ can be adapted to serve as a good guide to algorithms.

If he wants to do Statistics then R is the natural choice.