I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that loads the entire raw data and recomputes features.

I'm looking for a framework to think about how we can incrementally update our features as new data arrives without recomputing it from scratch.

Note that the features we use tend to have the following properties:

  1. They are usually counts, averages and ratios.
  2. Some of the features are computed for last n days. n is usually 7, 30 or 60.

I want to know two things.

  • Is an incremental approach viable, where we only run an ETL to load the data that came in since the last ETL?
  • If yes, how do we go about storing the meta-data that's required for the same?

For the quantities you mentioned, incremental updates are pretty simple, especially when your time windows are always calculated in full days.

  • I think it is obvious how to incrementally update a count of some record over a certain number of days - just make sure you have the counted values stored for every day, then when moving the window by one day, just subtract the count of the former first day and add the count of the new last day of the window

  • a sum of values can be updated in an analogous fashion

  • an average is calculated as a sum of some records divided by the number of those records - just keep sum and the record counters, which can be updated incrementally as mentioned before

  • and I guess by "ratio" you mean the quotient of two other incrementally calculatable values - which works similar as the former average calculation

If you have to deal with other quantities where it is not so obvious how to update them incrementally, just describe them, the community here may tell you how to deal with them.

  • Thanks a lot! Do you have any suggestions on how to deal with max and min of variables for last n days? We can't do it with just data from the previous first and last days. – spoderman Dec 19 '18 at 14:14
  • @spoderman: sure. Just store the maximum value over all records of each day (lets call it M_1, ..., M_n). The total maximum over n days is then equal to the maximum over M_1,...,M_n. So when the time window changes by one day, you have to recalculate the maximum over M_2, ... M_(n+1), which should be no issue for small values of n (and n = 30 or even n=365 is very small) – Doc Brown Dec 19 '18 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.