It is tempting to write the code and thethen see if the result "looks right", but, as you rightly intuit, it's not a good idea.
When the algorithm is hard you can do a number of things to make the manual calculation of the result easier.
Use Excel. setSet up a spreadsheet that does some or all of the calculation for you. Keep it simple enough so that you can see the steps.
Split your method up into smaller testable methods, each with itstheir own tests. When you are sure the smaller parts work, use them to manually work through the next step.
Use aggregate properties to sanity-check. E.g.For example, say you have a probability calculator; you might not know what the individual results should be, but you know they all have to add up to 100%.
Brute force. Write a program that generates all possible results, and check that none are better than what your algorithm generates.