This boils down to time management and a measured approach to risk. First consider where your stakeholders stand -- While not universally true, it's typically the case that stakeholders prefer the Pareto principle by looking to find the 20% effort which delivers 80% of the desired outcomes.
This crops up often in complex software - many activities can suffer from a sort of contradiction whereby they are impossible to clearly define or estimate with any confidence yet also suffer the law of diminishing returns with regards to developer productivity and overall benefit of time taken.
This is a problem because all activities need a priority and a decision about whether they are worth starting at all, as well as whether other activities or goals might need to be pushed out of the way.
Activities which snowball out of control can easily lead to a lot of poorly-spent time wrecking the team's ability to meet their overall delivery commitments, as well as the team drowning under the weight of expectation for undeliverable promises, especially if a developer or team are regularly taking on a lot of those unbounded activities.
A common approach to time management for such seemingly unbounded tasks is to consider Timeboxing some investigation work which simply has the goal of sufficient investigation for a more confident range of estimates -- at least a "worst case" vs "best case".
Before starting investigation, plan out how to make the best use of the timebox - remember the goals are discovery of unknowns and estimation so your concerns are at a higher-level rather than deep-dive detail. Consider some of the following tasks:
- If the features you're touching are unfamiliar, turn to the product owner, BA and QA testers to learn about those and users' data.
- Seek architectural information/docs around that area of the code from other developers who have worked on the system before.
- Identify the most important architectural boundaries, structures and interfaces which surround those features
- Attach a debugger, enable full logging then try hitting the relevant parts of the code from a user perspective.
- Check existing tests for that code.
- Introduce some new tests to try to check any of your most important assumptions.
- Look at the output of static analysis tools to get an approximate idea of the code quality.
Once a timebox ends, work stops and the developer reports back to the team (including Product Owner) on how far they were able to get in the timebox, to try to talk about the worst/best case estimates. The outcome might just be that the problem is still nowhere near to something that anyone can define or estimate. Then it should be the role of the Product Owner and/or technical & project leadership to use the findings to decide whether/how to progress, whether to do more investigation, or even to put the work on-hold.
The amount of time to timebox is somewhat risk-dependent; it should be enough time to have a realistic attempt at discovering the unknowns but it is ultimately nothing more than a discovery task. Risk is a context-dependent subject, but there are a some common factors to think about:
- Importance of the task to the team delivering on its commitments (e.g. how heavily dependent other work might be).
- Implications for security/privacy, health/safety, legal/regulatory compliance and contractual arrangements.
- Quality and complexity of the code.
- Automated testing and tooling that surrounds the code.
- Extent of the code and features under investigation (if it's a very large portion of the code then that'll naturally require a longer timebox)
- The organisation's collective understanding of the software (poorly understood code which nobody has touched in years generally needs some investment to unearth its secrets)
- Availability of others who could share knowledge and support/assist in the investigation
Ideally the organisation and stakeholders should be actively managing risk - while they may not understand anything of the technical aspect, it's important for the team to communicate upwards about why some tasks are shrouded in unknowns because it feeds into their expectations and perception of the team, as well as business planning, resourcing, budgets, commitments to customers, etc.
Ultimately this should be something to approach as a whole team; while you may personally find it hard to manage time and risk, teams exist to work towards collective goals and support each other, so it may be that your team as a whole could benefit by changing how risky/unknown tasks are planned and elaborated.