1

In my WebApi controller I have the following (pseudo) code that receives update notifications from Instagrams real-time API:

[HttpPost]
public void Post(InstagramUpdate instagramUpdate)
{
    var subscriptionId = instagramUpdate.SubscriptionId;
    var lastUpdate = GetLastUpdate(subscriptionId);

    // To avoid breaking my Instagram request limit, do not fetch new images too often.
    if (lastUpdate.AddSeconds(5) < DateTime.UtcNow)
    {
        // More than 5 seconds ago since last update for this subscription. Get new images
        GetNewImagesFromInstagram(subscriptionId);
        UpdateLastUpdate(subscriptionId, DateTime.UtcNow);
    }
}

This won't work very well if I receive two update notifications for the same subscription almost simultaneously, since lastUpdate won't have been updated until after the first request has been processed. And even if I update lastUpdate directly, there is a "lag" because of the round trip to the database.

What would be the best way to tackle this problem? I'm thinking of using some kind of cache (that would be faster than the database roundtrip), but I'm not sure how. Is there some kind of best practices for these kind of things? I'm guessing it's a common problem: "receive notification, do something if something hasn't been done recently..."

  • Are you trying to implement Web API throttling? – rwong Mar 20 '15 at 8:18
  • @rwong, I might :) Not sure. For example: I might receive 5 notifications within a couple of seconds, saying that subscription with id = 5 has been updated. I only want to act on one of those (preferably the last) and get updates from Instagrams API for that item. Could throttling help me here? – Joel Mar 20 '15 at 8:41
  • What happens if you receive a notification, then another notification 4 seconds later, and then no notification for 10 minutes? Then the new images will not be fetched for 10 minutes, defeating the purpose of using the realtime API. (And I do not think that throttling would help here.) – jhominal Mar 20 '15 at 10:07
  • @jhominal I know. That is another situation I'm thinking about... :) But I just can't respond to every update notification or I will break my rate limit towards instagram pretty fast. – Joel Mar 20 '15 at 10:23
2

I have another suggestion for the mechanism: when an update arrives, mark a subscription as having to update, and wait for your delay before making the update. That is, there is no need for a cache, you only need to store some state for each subscription id as "having to update" or not.

I will use async because it makes the code pretty straightforward.

For example:

private struct Empty {}
private ConcurrentDictionary<int, Empty> subscriptionsToUpdate = new ConcurrentDictionary<int, Empty>();

[HttpPost]
public async Task PostAsync(InstagramUpdate instagramUpdate)
{
    var subscriptionId = instagramUpdate.SubscriptionId;

    if (subscriptionsToUpdate.TryAdd(subscriptionId, default(Empty)) {
        await Task.Delay(TimeSpan.FromSeconds(5));

        if (subscriptionsToUpdate.TryRemove(subscriptionId)) {
            GetNewImagesFromInstagram(subscriptionId);
            UpdateLastUpdate(subscriptionId, DateTime.UtcNow);
        }
    }
}

Note that the implementation here only considers requests that are handled by a single Web API process. (E.g. if you have multiple servers, that code would potentially make one request, every five seconds, for each server that you have. If you want to avoid that, you would have to use some kind of shared store instead of a ConcurrentDictionary. (I would recommend something like a key value store, like Redis or memcached.))

  • Thank you. I went with this approach, although I used MemoryCache (msdn.microsoft.com/en-us/library/…) – Joel Mar 29 '15 at 9:38
  • @Joel: Take care if you are using MemoryCache - as far as I know, items can be removed from MemoryCache without you doing that explicitly. – jhominal Mar 30 '15 at 9:27
  • Thanks. I'll keep that in mind. However, in my case that is no problem since I'm just using it to limit the number of requests to Instagram. – Joel Mar 30 '15 at 9:30
1

How about something like this?

private static readonly object _syncRoot = new object();

[HttpPost]
public void Post(InstagramUpdate instagramUpdate)
{
    var subscriptionId = instagramUpdate.SubscriptionId;
    var lastUpdate = GetLastUpdate(subscriptionId);

    // To avoid breaking my Instagram request limit, do not fetch new images too often.
    if (lastUpdate.AddSeconds(5) < DateTime.UtcNow)
    {
        lock (_syncRoot)
        {
           lastUpdate = GetLastUpdate(subscriptionId);

           if (lastUpdate.AddSeconds(5) < DateTime.UtcNow){
               GetNewImagesFromInstagram(subscriptionId);
               UpdateLastUpdate(subscriptionId, DateTime.UtcNow);
           }
        }
    }

}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.