How to save a list of strings which might grow too large but old data is not useful

Question

I am developing an application which sends certain notifications to the user as read from a read-only external service. The user might dismiss notifications, and those should not appear again.

I cannot ask the server to give me only entries newer than than my last query because I am particularly interested in a value that changes over time. I have to give entries a chance for at least a week or so. Because of that, the queries to the service might return data which were already retrieved before, and I need to filter the ones already dismissed. I can do that by looking at the IDs of the received entries, which appear to be SHA hashes.

I can save those IDs to the Preferences as id -> boolean pairs, or in a SQLite database, but surely they will reach some limit sooner or later.

Also, I do not really need to check the older entries. I could put a hard limit on, say, the 100 latest entries and that should be more than enough.

How should I approach the disposal of old entries to ensure I don't go over the limits?

---

EDIT: As requested, more information about the problem which might be useful:

• My query currently is of the form "the latest 1000 entries, from newest to oldest, if they are newer than 2 weeks". 1000 is a number so high that is effectively infinite, for the purposes of my application. 2 weeks is a time interval so long that the user should not want to be notified about that information anymore, as it is highly unlikely to become relevant by that time.

• All entries have a "created" timestamp. They also have an "updated" timestamp, which, if exists, should be treated as the "created" date for the purposes of my application. I do not expect answers to account for this technicality, though.

• All entries have an "importance coefficient", which is the value I am tracking. I notify the user only of entries with this coefficient higher than a set threshold. Since this value changes over time, I cannot simply ignore entries I have already fetched before and found to not be relevant. Changes in this value do not affect the "updated" field.

• If the user dismisses the notification about an entry, its notification should be filtered out the next time a query happens. Comparing the IDs is enough for that.

Answer 1

Just an idea, maybe I misunderstood some parameters of the problem. I'll offer my solution anyway, as a start to work towards something functional.

At retrieval time:

• retrieve from read-only database only entries that are either newer than your last query or newer than 1-2 weeks
• filter out the entries that have already been dismissed comparing their ID's with the ones you saved in your SQLite database

In addition, if you are afraid to run out of space in your SQLite database, or anyway want to forget about the old dismissed notifications:

• once every week, check which ones of the entries of your SQLite database are older than 2 weeks (either running a cross-check with the read-only database, or just saving their entrance date in your SQLite database), and delete them

Answer 2

Using complement operation from set theory. And assuming recent notifications limited to 100.

Set<Notification> updateNotifications(Set<Notification> persisted, Set<Notification> recent) {
    Set<Notification> newNotifications = complement(recent, persisted);
    save(newNotifications);
    persisted = persisted.addAll(newNotifications);

    if (persisted.size() <= 100) {
        return persisted;
    }

    Set<Notification> obsoleteNotifications = complement(persisted, recent);
    delete(obsoleteNotifications);
    persisted = persisted.removeAll(obsoleteNotifications);

    return persisted;
}

---

class Notification {
    /* Fields provided by the service */
    boolean dismissed;
}

Notification class presumably deserialized from JSON and dismissed set to false by default. After each service fetch call updateNotifications to save new notification and delete obsolete ones. Only new notifications are saved, so dismissed flag on previously persisted notifications is preserved.

---

With a timestamp you can implement complement and delete more efficiently.

Answer 3

Write a notification class or struct with properties that cover all fields in your result set plus an extra "Dismissed" property with a default value of false.

At application startup read your stored sorted list of notification objects (which may be empty the first time). It should be sorted by timestamp. Trim it to your limits. Then periodically

• execute your query.

• loop through the records in the result set, for each:

  • look up the object with the same id;
  • if its Dismissed property is true, continue/next. Else create a notification object for the record and aďd it to your sorted list.

• Present the not yet dismissed objects in your collection to the user. Allow the user to read and dismiss an object. When he does dismiss it, set the Dismissed property to true and remove the notification from view.

At application close, persist your notification collection.

If the notification view is open and your poller kicks in, it should either skip a round or refresh the view after the notification collection has been updated.

Answer 4

As per what you have mentioned and as per my understanding,

1. Fetch results from the server.
2. Update record if entry exists otherwise insert.
3. Set an expiry date of record (add two weeks to the created or updated date, whichever is higher/ depending on your requirement).
4. Delete the entries expired.

And when user dismissed the notification for an entry, then you can set a flag on the record.