Web page: randomised collection of paged items retrieved via rest api call

Question

Using .Net web api

I have a web page that renders a collection of items depending on filter values applied via an ajax call to an api endpoint.

The items are paged so that if the user scrolls to the bottom of a page then another ajax call to the api with the required page number of the next set of items is made.

I have just had a request to randomise all of these items daily.

The obvious issue here is the ajax paging - each call retrieves the next set of items which at the moment is easy as the order of those items does not change. So if we are randomising, I cannot simply randomise each call.

So I guess I need some sort of caching...

These are thoughts so far:

1. At midnight, all items are loaded into the cache with some sort of date-based version key (based on server time).
2. Any client calls that are page 1 items will always get the cache's latest version using the date-based key. The page 1 response will return the cache key to the client (along with the data) so that any calls for pages greater than 1 will require the cache version key.
3. To deal with the possibility of a user making a page 1 request to the api at 23:59:59 and then a page 2 request after midnight at 00:00:30 we could expire cache versions after 36 hours. However, new cache version is created every 24 hours at midnight.

This is the general direction I'm considering.

Does anyone have any wise words?

Answer 1

So I guess I need some sort of caching...

While your cache idea isn't impossible, I suggest taking an easier route. Caches with custom logic are often hard to implement correctly, or debug when something is going wrong.

---

I have just had a request to randomise all of these items daily.

The "daily" part caught my attention. If you were to always randomize the list, you'd end up with some complicated logic (as you've already discovered).

However, given that the shuffle occurs on a relatively slow cycle, there is a simpler way to do it.

Given that they asked for a daily shuffle, I infer that they specifically want to maintain the same order during the same day, for every user. This simpler method means less overhead in terms of making sure that everyone works with the same shuffled order on the same day.

---

1. Add a ShuffledOrder int property to your class

public class Person
{
    public string Name { get; set; }
    public int ShuffledOrder { get; set; }
}

2. Create a job that runs every midnight, which shuffles the order. The idea is that you always end up with a ShuffleOrder sequence ranging from 1 to N (where N is the amount of entities), e.g.:

  Name  |  ShuffledOrder
==========================
   Tim  |      3
   Tom  |      2
   Joe  |      1
   Bob  |      4

A simple example of a shuffling algorithm:

var rnd = new Random();
var people = db.People.ToList();
int shuffleOrder = 1;

foreach(var person in people.OrderBy(p => rnd.Next())
{
    person.ShuffledOrder = shuffleOrder++;
}

db.SaveChanges();

Note that this is a simplified example to showcase the shuffling logic. Performance considerations (e.g. not loading a potentially massive table in memory) have been omitted to keep the example simple.

3. When retrieving items from the table, always sort them by their ShuffleOrder

var pagedList = db.People
                       .OrderBy(p => p.ShuffledOrder)
                       .Skip(pageIndex * pageSize)
                       .Take(pageSize)
                       .ToList();

Summary

• Setting the shuffled order in stone ensures that you maintain the shuffled order for as long as you wish.
• This approach is not negatively affected by any additional filter logic you might have.
• I expect this to be more performant than redoing the randomizer for every call (even if the randomizer is seeded with today's date)
• It avoids caching, which tends to complicate development and debugging.

Answer 2

You could include some unique key for each element already loaded and post this with the new Ajax request and then select some random items except the ones in the list received. Something like

SELECT top 10 * from myTable where Not Id in ('id1','id2'...) order by myRandomizer

Answer 3

Don't make your DBE redundant doing its job (caching) and do not overcomplicate client with implementation details (cache keys).

Your DBE already has the tools to manage a good cache (eventually even for cache invalidation), create a view sorted using current date as criteria and do pagination versus the view instead of the table.

DBE caches and indexes will do the rest. You may first try with:

• simple ORDER BY in the query when reading the view, but not every DBE might be efficient enough for this use case.
• if supported then use a display_index in the view, calculated using a random number seeded with current date over a query already sorted by another stable column (for example id).

In any case start with the simplest one and measure (and tune your index!!!)