2

Using .Net web api

I have a web page that renders a collection of items depending on filter values applied via an ajax call to an api endpoint.

The items are paged so that if the user scrolls to the bottom of a page then another ajax call to the api with the required page number of the next set of items is made.

I have just had a request to randomise all of these items daily.

The obvious issue here is the ajax paging - each call retrieves the next set of items which at the moment is easy as the order of those items does not change. So if we are randomising, I cannot simply randomise each call.

So I guess I need some sort of caching...

These are thoughts so far:

  1. At midnight, all items are loaded into the cache with some sort of date-based version key (based on server time).
  2. Any client calls that are page 1 items will always get the cache's latest version using the date-based key. The page 1 response will return the cache key to the client (along with the data) so that any calls for pages greater than 1 will require the cache version key.
  3. To deal with the possibility of a user making a page 1 request to the api at 23:59:59 and then a page 2 request after midnight at 00:00:30 we could expire cache versions after 36 hours. However, new cache version is created every 24 hours at midnight.

This is the general direction I'm considering.

Does anyone have any wise words?

1
  • Really interesting question! It's a simple problem statement with a less-than-trivial solution.
    – Flater
    Sep 20, 2018 at 6:30

3 Answers 3

2

So I guess I need some sort of caching...

While your cache idea isn't impossible, I suggest taking an easier route. Caches with custom logic are often hard to implement correctly, or debug when something is going wrong.


I have just had a request to randomise all of these items daily.

The "daily" part caught my attention. If you were to always randomize the list, you'd end up with some complicated logic (as you've already discovered).

However, given that the shuffle occurs on a relatively slow cycle, there is a simpler way to do it.

Given that they asked for a daily shuffle, I infer that they specifically want to maintain the same order during the same day, for every user. This simpler method means less overhead in terms of making sure that everyone works with the same shuffled order on the same day.


  1. Add a ShuffledOrder int property to your class


public class Person
{
    public string Name { get; set; }
    public int ShuffledOrder { get; set; }
}


  1. Create a job that runs every midnight, which shuffles the order. The idea is that you always end up with a ShuffleOrder sequence ranging from 1 to N (where N is the amount of entities), e.g.:


  Name  |  ShuffledOrder
==========================
   Tim  |      3
   Tom  |      2
   Joe  |      1
   Bob  |      4

A simple example of a shuffling algorithm:

var rnd = new Random();
var people = db.People.ToList();
int shuffleOrder = 1;

foreach(var person in people.OrderBy(p => rnd.Next())
{
    person.ShuffledOrder = shuffleOrder++;
}

db.SaveChanges();

Note that this is a simplified example to showcase the shuffling logic. Performance considerations (e.g. not loading a potentially massive table in memory) have been omitted to keep the example simple.


  1. When retrieving items from the table, always sort them by their ShuffleOrder


var pagedList = db.People
                       .OrderBy(p => p.ShuffledOrder)
                       .Skip(pageIndex * pageSize)
                       .Take(pageSize)
                       .ToList();


Summary

  • Setting the shuffled order in stone ensures that you maintain the shuffled order for as long as you wish.
  • This approach is not negatively affected by any additional filter logic you might have.
  • I expect this to be more performant than redoing the randomizer for every call (even if the randomizer is seeded with today's date)
  • It avoids caching, which tends to complicate development and debugging.
2
  • Note that your shuffleOrder doesn't even need to be exact. You can use a random integer value instead of an incrementing one, and the results will most likely be good enough.
    – Duroth
    Aug 16, 2019 at 12:32
  • @Duroth: Only if the order value is a hidden value. For a shown value, people tend to expect a sequence with no gaps.
    – Flater
    Aug 17, 2019 at 15:35
0

You could include some unique key for each element already loaded and post this with the new Ajax request and then select some random items except the ones in the list received. Something like

SELECT top 10 * from myTable where Not Id in ('id1','id2'...) order by myRandomizer
1
  • Note that this would reshuffle the order for every new list that is loaded, which conflicts with having a daily shuffle. Unless the randomizer itself uses the current date as a seed, but I suspect you're going to run into performance issues by redoing the random-but-fixed-for-today shuffle for every query. Also, if the user can apply any filter to the list, your randomizer (even when seeded with today's date) will yield different results when the filter is changed, since an order by is applied after a where
    – Flater
    Sep 20, 2018 at 6:25
0

Don't make your DBE redundant doing its job (caching) and do not overcomplicate client with implementation details (cache keys).

Your DBE already has the tools to manage a good cache (eventually even for cache invalidation), create a view sorted using current date as criteria and do pagination versus the view instead of the table.

DBE caches and indexes will do the rest. You may first try with:

  • simple ORDER BY in the query when reading the view, but not every DBE might be efficient enough for this use case.
  • if supported then use a display_index in the view, calculated using a random number seeded with current date over a query already sorted by another stable column (for example id).

In any case start with the simplest one and measure (and tune your index!!!)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.