Designing a hierarchical structure with lots of reads and writes?

Question

I am in the process of working on a video on demand system part of it involves the management of a hierarchical tree structure (think windows explorer) which allows users to upload videos, move folders around, create new folders etc. User action will be allowed depending on which permissions they have.

In addition to managing the structure (creating folders and uploading videos), subscribers will be viewing content (read access). The number of reads will be significantly more than the writes.

My question (and it is big one) is should I store the data in a database (for the writes) and also have some sort of cache which will be used for the reads? or do I use two databases? or is there a better solution?

Also I will have to resolve concurrency issues which I think optimistic locking on the database will resolve.

I have a fair bit about CQRS over the last few months but not sure if this is the way to go.

Any ideas?

Answer 1

If it's a system with any sort of scale involved (i.e. multiple users), you are almost certainly going to need a database. You should only need one. Databases are already designed to handle concurrency.

Caching will come into play if, and when, you become the next Netflix. In other words, YAGNI (You ain't gonna need it, at least not yet). Caching is intended to be bolted on to an existing system so, if your software is architected sensibly, you should be able to add it later.

Is CQRS "Command Query Responsibility Segregation?" If you pick a sensible architecture like MVC, you should get those sort of best practices for free.

Answer 2

I'd do the simplest possible thing that would work, which (for me) would be to keep everything in a database. I'd probably store the media outside the database, just to reduce the load on the connection pool (transferring several GB of data can tie up a DB connection for a long time), but other than that just keep it simple. Write every configuration change to the database, re-read the configuration every time the UI needs to be updated, get all the features working then profile it to get some numbers. You may find out it's "fast enough" as it stands, so you don't need to change anything. If it is slow, you'll be able to quantify how slow it is and where, so you can start optimizing the parts that really matter as opposed to the things you suspect matter. It's also nice to have a baseline implementation you can test against, so you can verify that your performance improvements haven't broken anything.