I am in the middle of an architectural decision that down the line will be important.
I have a system where I use ATS (Azure Table Storage) as the store for simple and very little data. It's not written to, too often and it's not read to, too often.
Now I need to read/write data based on these ATS data. A lot of data. But I will not read the data in ATS before I write the "other" data.
My concern is that I cannot read the data fast enough from ATS, based on the needs I have. E.g I need to count rows fairly fast to give the users feedback, and count is not a function within ATS. I could gain something from the CQRS "pattern" but I would still need to count rows! And I am not looking for a solution where I need to add complexity to overcome a very simple thing on another platform (SQL).
So my thought is to save these data inside a SQL database where I also have all the data manipulation-features I need. But I would miss out on the scalability and end up maintaining that instead of an easy scalable datastore like ATS. And the data I am writing here is very simple, but it's a lot.
I would like to stay with ATS only but I would need a pattern to get around these limitations. Any ideas ?