We have a working proof of concept that implements the sunny day scenarios, but we need to start handling the exception paths. We are contractors and need to put together detailed estimates to discuss this next phase with the client before they renew the contract.
Up till now, we have mostly focused on sunny day scenarios to put the proof of concept together. Now that that's finished, we need to bring the library to maturity and support rainy day scenarios in the different areas of the library including:
- bad data provided through the API by sdk users
- error paths in our code
- legal error responses from the browser API and third party libraries
- network issues (jitter, bandwidth limits, temporary disconnection, etc.)
Question: What process can we follow to generate a reasonable task list of rainy day scenarios that we don't but ought to support. In other words: How can I figure out what's missing between (1) my proof of concept application and (2) a hypothetical mature, robust application that implements the same features
I am also interested in retrospective type suggestions about what we could have done while implementing the proof of concept to make it easier to itemize this work.
What I would do in a vacuum
Without any better advice, I will be doing a static analysis of the code I've written, and the specs for the third party libraries and browser apis, and identifying possible error cases. Starting with the whole, I would divide it into components, subcomponents, etc.. When I've reached enough granularity, I would perform the static analysis or analyze the spec and itemize ways it could fail (legal error output, reasonable exception cases, etc.). Then I would aggregate those failure modes for each subcomponent, then component, and back to the whole and I would have a list of rainy day scenarios that I need to support.
Issues with this:
- how can I avoid missing something major?
- is there a way to avoid analyzing code and spec line by line?
- we'll need to prioritize these rainy day scenarios - how do we divide the list into more important and less important? (likelihood that it would occur?)
- how to aggregate the list of particular error cases for particular methods/apis into something high level enough for the project leads to make decisions about what/how much/when? (This is closely related to the last question about prioritization)
Update: I've done an FMEA (Failure Mode Effects Analysis) which was helpful in analyzing the system boundaries - rainy day scenarios related to networking, consumers of the public API, and (to a lesser degree) errors at the third party api layer. The FMEA process suggests using a diverse team to generate the list of possible failures, which would help with my issue #1 (how to avoid missing something major). It also deals with #3 (how to prioritize) by giving a priority metric based on the severity of a failure, it's likelihood of occurring, and the likelihood that it will be detected. It didn't help me (at this point) to come up with a rainy day scenario list for the code we've written, and it didn't address my issues #2 and #4.