From the Agile Manifesto:
We follow these principles:
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
I have a requirement where we are doing cross database joins with multiple partner teams in many different data sources. Some are traditional RDMBS, some are columnar, and one is more of an API than a database for security reasons (so I can't run arbitrary
SELECT ... JOIN against it).
Our solution is to set up ETL jobs against the various sources and try to get snapshots of relevant data in a single place, and use these local tables to do the cross db joins and eventually generate reports.
The problem is that if I need to use some new column as a join key (a new data source is introduced), or if a new business question is being asked, and I haven't pulled that data, then I need to modify the schemas and then rerun all the extracts again, and in some cases change the business calculations in the transforms which can take hours and requires a lot of operational overhead.
I keep hoping that requirements won't keep changing, but honestly that hope is not Agile - I should be able to respond to additional business questions on the fly. Another part of Agile is that the PMs should try to plan for these sorts of changes as early as possible - is that where my process is breaking down? Alternatively, how can I structure dataflows and related processes to be just as agile as other types of code?