There are a couple of things you could try:
- Try to find some example values that are "obviously correct" (in the sense that any future maintenance programmer can easily see that and why they are correct, and verify their correctness in their heads without using a calculator)
- Look up the paper your algorithm comes from and see whether they have examples listed there, if yes, convert them into test cases (and put references to the paper in the documentation comment of the test method)
- Provide "obviously correct" 1:1 translations of the algorithm from the paper in one or more separate very high-level language(s) (Matlab, Mathematica, Idris, Haskell, Julia, Python, …) and write a script generating test cases using those "obviously correct" implementations. The script should be written in such a way that it can be easily verified that the generated test cases are correct.
- Try to find the simplest, closest, 1:1 translation of the algorithm (or equation) into your test language, and use that to test the actual implementation.
In the latter case, it may very well be that the implementation in the test case and the implementation in the production code end up being identical. However, that should only be a coincidence, not a goal. You should not copy&paste the implementation.
Now, why is such a test case, where the code in the test and the code in the production code are identical, beneficial? Well, quite simply: code changes. Tests usually don't.
If there is one thing that is true about any part of any graphics pipeline, then it is that it is too slow. Now, you're probably thinking: "my code isn't too slow!" And you're probably right. For your use case. For now.
But believe me: sometime, somewhere, someone will have some crazy idea. Maybe they want to use your library for real-time video processing in an eye implant powered by the bio-electric energy of the human eye, and thus with a seriously constrained power budget (and thus a very slow processor). Maybe it's an art project and they want to render real-time video onto the entire Chinese Wall and thus need to be able to render trillions of pixels at 60Hz.
And then you will have to uglify your algorithm. Put in shortcuts. Manually unroll loops. Offload it to the GPU. Rewrite it in assembly. Maybe even implement it in hardware.
However, your unit test will still be the same straightforward, easily reviewable, easily understandable, piece of code, acting as a safety net for all the crazy, ugly things you need to do to your production code, because, well, real-life constraints happen.