Because we need to keep response times low, we get tons of requests, and we need to basically process ALMOST the same data (which I'll refer to as X) each request (the inputs are different though, so we can't cache responses), we are using a technique where we grab a new copy of X every 90 seconds from the database and store it locally in memory as a python list of dictionaries, on our application servers (we are using uwsgi).

The kink in the machine: There are temporary analytics that we need to keep track of in those 90 seconds to adjust our data each iteration, and each iteration is dependent on what we calculate from the last iteration.

The trouble with this is, we have multiple application servers that are storing the same data, X, in memory and each of those servers need to refresh X at the same time to keep calculations consistent for the next interval. I've tried some techniques, like broadcasting a message after each calculation to reload each server's X, but it hasn't been as effective as I would hope, and it just makes things more complicated.

I should say, the reason we haven't used memcached or something similar is because we don't want to sacrifice any speed if we can. Maybe I am ignorant on how fast we can retrieve and load the list into python objects from memcached.

I understand my explanation isn't the greatest, and will answer any questions to give a better picture of the situation.

Edit: we are at about 5000 request/second, the size of the data we process is about 2MB at the moment but will continue to grow, so we'd like to avoid sending it over the wire for each request.

  • 1
    Did you try memcached and found that it was slow or are you just guessing? Because it's really fast. Like, really. You should definitely profile it. Commented Apr 16, 2013 at 21:07
  • I'll give it a real test later today and report back with my results.
    – tonyl7126
    Commented Apr 16, 2013 at 23:21
  • Sorry about the slow response. In my initial test, just getting the data from memcached (pylibmc driver) was too slow (25 ms (used "timeit")) due to the size of the list. I know that is fast by most standards, but I cant waste time just getting the data. It was actually more like 2 megabytes, I misinterpreted sys.getsizeof in my previous statement. If I want to use memcached, I will need to break the list up, which means changing the application logic
    – tonyl7126
    Commented Apr 19, 2013 at 0:01
  • 1
    Did you profile your code? Maybe the hot spot is relatively compact, and you could rewrite it using Cython? It might buy you enough time to access memcached or Redis.
    – 9000
    Commented Apr 19, 2013 at 3:16
  • Did you ever implement this? Interested in what the final result became. Commented Oct 4, 2013 at 8:53

2 Answers 2


Your question is interesting, thought to give a quick answer but it is more complicated.

You have a set of data (please supply size and amount of requests indication!). It should be equal at all machines at a given moment. That is interesting because it makes distributed systems less use-full. You get to a more transaction based system where you can really trust the data.

You state it is a requirement (and so important).

First thing to ensure yourself of is: Do I really need to distribute this data? If you use a central storage you don't have all the synchronisation and replication issues at all. So if there is no problem you don't have to fix it. (In this case hardware might be a much cheaper solution).

And I assume based on your story that there is no reason. Why? Because it is possible for you to get every 90 seconds "all" data. You could even do that on a broadcast. So your central storage seems to be able to deliver that instant.

Though you have multiple servers so it seems needed. Now they store the data, do calculations on it, reset itself to the new data and start servicing again. That means the expectation is that the response delay will increase significantly when you ask the data from you central storage instead of your local Python dictionary.

Sounds reasonable. Did you measure this? Are you really sure that's not possible? Really really sure?

Ok, let's say it is impossible: The you need to have a distributed system which guarantees that the data will be there always when you start with a new session. I personally think you should forget about that 90 sec. period anyway. So, in that case you get into realtime, near-realtime, solutions. This is possible but I think you make it yourself very hard.

Generally all no-sql solutions which are distributed weakened the guarantees for syncing data. So find yourself an easier way out or create the almost real-time cluster. Those are your best bets.

  • I appreciate all of the advice, could you possibly go into a little more detail on a potential real-time cluster solution? Or give a real life example?
    – tonyl7126
    Commented Apr 16, 2013 at 18:35
  • If you have your figures ready we can move on to more concrete advise! Waiting for that, let us know when you have the figures! Commented Apr 17, 2013 at 14:16

I think a good approach would be to split checking for data and actually getting it. Why? Memcache is really fast... But as you explained, there is some latency due to the big chunk of data you need to recover. So let's just adjust your design around this topic.

That is, try to mimic the HEAD + GET methods of HTTP...

Each of your servers gets a little value from memcache. This tiny value acts a HEAD, just returning the date+time when the big 2MB data was last calculated.

If the server sees it's the same as its own copy, just goes on with the local cached version of the 2MB blob.

When the data is recalculated, both 2MB data and tiny semaphore are updated on memcache. So, next time a server gets the tiny date, it knows the data has to be reloaded into its local cache.

  • The question states that caching is not an option because the vary depending on inputs. On top of that implementing caching done correctly with a variable about of inputs is quite a lot of work. Also keeping the cached value as a state of the server goes against the benefits of using memcached.
    – pllee
    Commented May 20, 2013 at 22:11
  • Sorry if I didn't explain it clear enough. As the question states, the results are used for 90 seconds, then recalculated. That's a 90 seconds cache, as far as I know. Commented May 21, 2013 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.