I'm currently working in a project that aims to implement automatic testing of a software package. You can imagine this software is a bit like Excel in that it has a workspace that contains all the data, and a user interface that executes code that works on this workspace data. We are primarily focused on testing "core functionality", rather than the user interface in itself. Previously, a lot of testing has been manual and poorly documented.
One of the problems we're facing is that essentially anything that you might want to test requires a "complete" data set. Due to inherited architectural complications it is not really feasible to unit test anything - the mocking required is too extensive.
Another problem we're facing is that the code is in several different languages - mostly Python and C++. Some of the Python code is being run inside a Python interpreter running in the software, giving it access to C++ code that is otherwise less accessible.
Also, the current level of manual testing is deemed "good enough", and we aim to get at least "the same coverage". However, since manual testing inevitably goes through the UI and since the current manual cases are extremely poorly documented, who knows what we're currently covering with the manual tests?
I'm having a hard time understanding how to achieve good coverage (or even just how to measure coverage intelligently).
A lot of the standard answers I've come across so far don't really apply here. E.g., "write testable code" is a great tip when you start writing code, but refactoring millions of lines of code accumulated over 25 years is not in scope for this project, not a manageable task for a small team, and a political impossibility given the circumstances.
I'm looking for any and all suggestions on how to achieve good test coverage, how to measure coverage, and generally how to tackle the transition from poorly documented manual testing to some sort of sensibly comprehensive automatic testing.
I'm not an expert, so there may be low hanging fruit that I've overlooked, in particular if I happen to not know the relevant search terms - if that's the case, a gentle prod in the right direction may very well be a good start.