I need to think of a way to model how one thing is more "changy" than another.
Say I need to index a news website's different news sections with some web crawler. I want to prioritise indexing those sections (e.g. live news or sports) that receive more news items than those that don't (e.g. health or culture).
A crude way that I thought up is to initially have equal probability to pick each news section. I then index them one page at a time (I have this restriction) and each time I index a section, I sum how many news items have been removed, added or updated in that section compared to last time (think of it as a delta) and I use that to adjust the probability of that section being picked for indexing next (e.g. by some weight).
Over time, the live news section will have a bigger weight than something like health or culture, since it will have had more news items added to it or updated (I'm using weighted random choice to randomly select items to pick with some weight consideration) and so will be indexed more frequently to keep up with any new changes.
But I'm not sure how to think of this numerically. What initial weights, factors and ranges make sense? Can I prevent more dynamic sections from completely dominating less dynamic ones for which section gets picked next for indexing (I can only visit one page at a time)? Are there established algorithms that deal with these kinds of problems? Ideally I'll have something that gives a sufficient priority to "changier" sections whilst still allowing less dynamic sections to get indexed.