How to scale an application where every read results in a write?

Question

So I am trying to design a simple URL shortener application where every time a URL is queried, it is going to update the number of times it has been queried.

I'm thinking of using MongoDB and I am thinking of some sample schema like this

{
id : ...., //Mongo-generated ID
originalUrl: .... // The original URL
shortUrlKey: .... // The shortened URL key
createdAt: ....
updatedAt: .....
hitCount: ...... // Number of times the document has been queried.
}

I want to make sure that every time a particular URL is queried, the request increments the hitCount field by one and then returns the entity.

Now I read somewhere that writes on MongoDB result in locking that particular document.

So I have the following questions:

1. Since every read here is going to update the document, how best can I design my application so that it can be scaled efficiently?

2. Also, I want to serve the URLs from Redis cache once the hitCount crosses a certain number. But while I'm serving the URL from the cache, I still want to update the hitCount field. How do I do that?

It makes sense to have an asynchronous (fire and ignore the result) call to update the field once the URL starts getting served from cache because at that point keeping hitCount synchronised doesn't matter, but until that point, how can I sync the document without degrading the performance or losing the chance of scalability?

Answer 1

Document databases work better with a write little/read many approach. This is even more true of databases backed by Lucene indexes (which I believe Mongo still is). The constant churn on the index can easily fragment and potentially corrupt your data store. I would steer clear of storing that information in Mongo.

The technical details of why this is bad has to do with how database manufacturers have to handle deletes:

• Flag the record to be purged
• Wait until you can commit the change
• Rebuild that index without the flagged records

As an optimization, some database vendors will fragment the index.

I recommend storing the count someplace else. Redis is designed for high churn data, as are many name-value pair databases.

Answer 2

I don't think MongoDB is a good use case. In a document database, you typically incur a read/modify/write cycle for the whole document. Your schema is small, there is no variance in the records, you don't have sparse records, I think your schema is also not going to change much. Document databases are a "store first, figure out schema later" approach where you can store different data types and then later figure out access patterns and which properties to index. You also don't need to normalize data and reconcile it with a schema.

But that schema would help a relational database to know exactly where hitcount is located on disk, how to read and write to it. If you put hitcount in it's own table, you'd have a very high cache efficiency since it can hold exactly the data that's frequently written to in memory.

One other thing to consider is whether or not hitcount needs to be factually accurate or just roughly accurate. If roughly accurate is enough, a lot of performance is to be had by writing the increments to a queue and coalescing the writes (i.e. adding up multiple increments for the same URL and updating the database in a single transaction)