I am trying to create a web page with very fast real-time football match updates. The user gets today's matches (some of them Live) according to his/her timezone (the sets of matches are different for different timezones). I got to 600ms for page load. But I want it faster.

Overview of my current solution:

  1. Console app pulls live data from third-party API
  2. Stores it in DB
  3. If there was a change in live data in any match, the match pushed via message queue to the web app server
  4. Web app server pushes the update to UI
  5. Web app server serves data to UI (on page load) from DB filtered by date and sorted by league name

pros: simple implementation

cons: not very fast, many DB connections, in rare cases page reload can cause older data load than was previously updated by pushing

Other solution I have tried:

  1. Store matches as Hash in Redis cache

  2. Store match ids with score (match date converted to double) in SortedSet

  3. Update match in Hash when there is a change from console app

  4. Get matches for today for a specific timezone (web server)

    a) get ids from SortedSet between scores

    b) get all matches from Hash and filter by ids from SortedSet

    c) sort by league name

  5. In this case matches skip Live status in DB (update from not started straight to finished) because the data always served from Cache

pros: A bit better performance 450ms

cons: complex implementation, still not fast enough, because of frequent Hash retrieval from Cache, sorting and filtering

I am wondering, is there a better solution? Faster and more elegant?

  • How about storing all the data in memory in the web server process? I assume there isn't a ton of data - megabytes at most.
    – user253751
    Jun 29 at 9:10
  • @user253751 What advantage does this approach have over storing in Redis? Just different kinds of cache... I maybe I miss something?
    – Dazhush
    Jun 29 at 12:34
  • indeed it is a different kind of cache, but I wonder if you are overcomplicating it. What does Redis do for you that Dictionary doesn't? Aren't you wasting time by using a completely separate server like Redis?
    – user253751
    Jun 29 at 16:22

600ms is pretty good. But its not about the page load time, or how you store the score information It's how to push updates to all users.

To get a fast update each user has to have an open connection, each connection needs to have the score sent separately. So if the score packet is say 1Kb.

If you have 1000 users, you have 1000 open connections to your servers and have to send 1Mb in < 1 sec. Sounds do-able.

If you have a million users you have 1,000,000 open connections and have to send 1Gb < 1 sec. Harder.

And that's just one goal in one match. In reality you have dozens of live matches and multiple event types goals, substitutions, fouls, throw-ins etc. What you need is broadcast networking, which the internet doesn't do very well.

You could do do deals with ISPs so you can host a server in their datacenter and your users get a broadcastable IP address on connection, or sign up to a provider that has this kind of network already in place.

  • Thanks for the answer. This is another problem that I may have in the future if I (hopefully) will get a lot of users. But currently, I am more interested in optimizing the API request for the page load and the right way to structure the flow.
    – Dazhush
    Jun 23 at 12:23
  • have you tried web sockets?
    – Ewan
    Jun 23 at 12:47
  • I am using .net core Signalr solution. It uses web sockets.
    – Dazhush
    Jun 23 at 12:48
  • im surprised you don't see faster updates then, caching shouldn't make a difference as you only read once, to determine if there is a change in score?
    – Ewan
    Jun 23 at 12:50
  • I guess it is not very optimized. O(log(N)) to retrieve from SortedSet, then O(N) to get the Hash, then filtering and sorting on the server. Plus hashed matches constantly retrieved, changed, and placed back by another process. Other cached responses, tomorrow matches, for example, are pretty fast, but they do not use hash.
    – Dazhush
    Jun 23 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.