I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that loads the entire raw data and recomputes features.
I'm looking for a framework to think about how we can incrementally update our features as new data arrives without recomputing it from scratch.
Note that the features we use tend to have the following properties:
- They are usually counts, averages and ratios.
- Some of the features are computed for last n days. n is usually 7, 30 or 60.
I want to know two things.
- Is an incremental approach viable, where we only run an ETL to load the data that came in since the last ETL?
- If yes, how do we go about storing the meta-data that's required for the same?