How to index a massive, randomly selected, uncontrollable, constantantly changing dataset?
Imagine you want to index all of the snow particles in a giant snowglobe that is constantly being shaken. Each snow particle represents a piece of lengthy text (say a book) that needs to be indexed, however, the particles are constantly moving and cannot be grouped into categories of indexed and not indexed. On top of this, snow particles disappear from the globe (either through disintegration or removal) and new particles are added. It is also possible for the text of the book to be changed or updated. There is a desire to keep the database as up to date as possible, so older index data may need to be updated periodically.
a. Is there an algorithm that can prevent having to look up a particle each time to see if it has already been indexed? On a large enough dataset, this will become very burdensome. I imagine there may be more overhead associated with such an algorithm and may be cumbersome for smaller datasets, but would prove faster with larger datasets as the index grows.
b. How to handle the updating of the index in terms of making sure the text is still the same and still exists? I don't really want to check if the data is the same in the case I randomly pick up an already indexed item to prevent unecessary loops (i.e. picking indexed items multiple times instead of spending less time indexing new data). As long as the first problem is solved, i.e. each examined item is a new item, then I suppose an update scheme which goes through after a certain period of time checking for consistency will work. Any other thoughts?
Please see my comments below which might help clarify my question and its motivation.
I will also add another example in case the snowglobe bs isn't clear:
Imagine you are tasked with independently cataloging and indexing the entire Library of Congress, but without the help of the library staff, so you must personally go to each book rack and begin indexing them individually on your own. Also, they will not help you by holding checked out books at the counter for your approval before reshelving. There is no clear place to start, so you begin with authors last names beginning with 'A'. By the time you get to section 'B', it is likely that someone has returned a book into the 'A' section after you completed indexing it and have no way of knowing which one it was, if there was one put back up at all. Now, let's imagine that the LOC is extremely busy, and so not going back to check the 'A' section would result in your index missing 10-20% of the dataset. In addition to that, not only are checked out books being returned by patrons, but new books are being published and stocked, as well as the release of updated editions to existing indexed items.
This is obviously not an ideal situation for actually indexing the LOC, but given the problem I'm certain there are better solutions than others. How to solve while minimizing time and cycles wasted?