A possible solution is to reduce the complexity of tests. For instance, with property-based testing (eg, ScalaCheck, QuickCheck) a single test could replace several unit tests as special cases. Then the whole testing hierarchy is flattened. Property testing fits maths/numerical applications particularly well, but at a computational cost.
A complementary approach, in line with the OP's hunches, is to use a testing framework amenable to a hierarchy of tests. This goes towards behaviour-driven development (BDD). You could have a high-level function (= BDD "Feature") and low-level supporting functions ( = BDD "Scenarios"). BDD testing tools easily accommodate this hierarchy. Some of them have nice GUIs visually organising, if not diagramming, it. In addition, they can serve as living documentation.
Some BDD references/search terms: "BDD In Action", Dan North, cucumber, serenity tool.