I am currently responsible for feeding data to an STT model. My backend is written in Python/Django and is primarily responsible for managing a Postgress database. The database has a table that saves data records; each instance, or record, will at some point end up being fed to the model. Each data record has a bunch of metadata that makes it unique in some way.
Currently, I have an endpoint that returns a list of
N IDs for each datarecord. The endpoint is usually queried by another service that deals much more closely with the model. Right now, the endpoint will simply return the
N newest IDs, but our business needs have changed a bit and our model needs to be better at addressing new data.
I have noticed that, overwhelmingly, the data fed to our model is not unique enough, and it is largely because of this strategy of returning the, "newest data." As I mentioned before, each record has a bunch of metadata that defines its uniqueness; things like
age- etc. A whole myriad of things that don't really matter too much. What does matter is that, for the most part, a lot of the data I receive is very similar. I would like to feed the model more diverse datasets. "Diverse," here, I define as "different from the data that was already fed to the model."
The database has a field for consumed and unconsumed data, so I know which records need to be fed to the model. The problem is, I cannot identify a proper-sounding strategy for this problem. The strategy I have come up with is:
Identify the categories (metadata fields) that are relevant to this problem (for example,
agemight be relevant, but
gendermight not be).
Assign weights to each relevant category (for example, maybe category
xcan be given a weight of 50,
z20, to make a total of
Give a score to a record based on how unique its category is (for example, maybe a record has for its
xfield a value of
x_valueand that gives you a score of 30/50, a
ygives you a score of 25/30, and a
zgives you 10/20, which means it has a total of 65/100 points).
Return a list of
Nrecords with the highest score.
This strategy seems... alright to me, but the main problem is how often I would need to make updates to each record. It seems really expensive to me. With each update, I would also need to recalculate how each value in a category scores (because they get points based on how often the model has encountered them amongst all other possible values).
The whole operation seems pretty expensive for me, and it definitely seems like something that must have been solved. I am looking for some advise on how this can be solved from an algorithms and/or systems point of view.
I hope my problem is clear enough. If you have any clarifying questions, please feel free to ask them.