I need to choose records from a database to feed to a model for training, but would like to weigh records beforehand. What's a good strategy for this?

Question

I am currently responsible for feeding data to an STT model. My backend is written in Python/Django and is primarily responsible for managing a Postgress database. The database has a table that saves data records; each instance, or record, will at some point end up being fed to the model. Each data record has a bunch of metadata that makes it unique in some way.

Currently, I have an endpoint that returns a list of N IDs for each datarecord. The endpoint is usually queried by another service that deals much more closely with the model. Right now, the endpoint will simply return the N newest IDs, but our business needs have changed a bit and our model needs to be better at addressing new data.

I have noticed that, overwhelmingly, the data fed to our model is not unique enough, and it is largely because of this strategy of returning the, "newest data." As I mentioned before, each record has a bunch of metadata that defines its uniqueness; things like location, gender, age- etc. A whole myriad of things that don't really matter too much. What does matter is that, for the most part, a lot of the data I receive is very similar. I would like to feed the model more diverse datasets. "Diverse," here, I define as "different from the data that was already fed to the model."

The database has a field for consumed and unconsumed data, so I know which records need to be fed to the model. The problem is, I cannot identify a proper-sounding strategy for this problem. The strategy I have come up with is:

1. Identify the categories (metadata fields) that are relevant to this problem (for example, age might be relevant, but gender might not be).

2. Assign weights to each relevant category (for example, maybe category x can be given a weight of 50, y 30, and z 20, to make a total of 100).

3. Give a score to a record based on how unique its category is (for example, maybe a record has for its x field a value of x_value and that gives you a score of 30/50, a y_value for category y gives you a score of 25/30, and a z_value for category z gives you 10/20, which means it has a total of 65/100 points).

4. Return a list of N records with the highest score.

This strategy seems... alright to me, but the main problem is how often I would need to make updates to each record. It seems really expensive to me. With each update, I would also need to recalculate how each value in a category scores (because they get points based on how often the model has encountered them amongst all other possible values).

The whole operation seems pretty expensive for me, and it definitely seems like something that must have been solved. I am looking for some advise on how this can be solved from an algorithms and/or systems point of view.

I hope my problem is clear enough. If you have any clarifying questions, please feel free to ask them.

Answer 1

As am I not too in STT expertise, the only idea that could come to me would be to run some clustering (potentially full np-)algorithm in order to identify groups of similar data and those that are differents, allowing to determine how many distinct class you have. Of course you would need for that to determine how to compute the "distance" between each "point" of your cluster.

Which mean you need to compute some score that is pertinent enough to feed "diversity" to your model. For me that could be a score on a vocal/musical analysis that would return the distribution of the voice or some stuff like that. You should probably ask more on a more specialized stackexchange for this.

Because as I understand now, you want some diversity, but you don't even know how to mesure the said diversity and neither the number of class you should aim for in order to have a realistic distribution.

So to resume :

• Determine what "distance" you need to mesure to be efficient with your model (unless some specialist in the field pass by there you will need to ask another SE forum).
• Run some Neareast neighbours algorithm or whatever clustering algorithm you need in order to identifify the said groups. Once you have identify your classes, when you receive new dta, you compute the distance to the various classes, affect to the neasrest class and then can decide if your model has been fed enough of them or not.