I'm looking at the design of dropbox's edgestore. It manages several thousands of MySql instances, where all dropbox's metadata resides (users, filenames, etc.). I understand why sharding is necessary, and I understand why it's very useful to have the client talk with a edgestore core that routes the request to the correct MySql shard (edgestore engine). What I don't understand is what's the benefit of adding the following complexity:

Since our workload is read-heavy, Cores use a Caching Layer to speed lookups. The caches are also partitioned and replicated for high-availability. Edgestore provides strong-consistency by default, which requires invalidating caches on writes. If the workload can tolerate stale reads, clients can request eventual consistency.

What's wrong with just relying on the LRU cache each MySql provides by default? Is the application caching so much more efficient? how? extra points for showing me how its so much more efficient that it justifies maintaining such a complex mechanism.



A few things I can think of

  • If the data is relatively static (think daily or hourly ETL) the cache can save the cost of repeating a slow query in between data changes.
  • Database servers are expensive. High-end cores, fast disk, lots of RAM & mega-buck licenses. Moving work off these boxes can prevent (or at least delay) a very spendy upgrade.
  • In distributed systems having the data close to the user reduces latency. Caches can be close to users while the database system of record is centralised and ACID.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.