1

I have a simple use case:

  • Newsfeeds aggregated based on posts that are linked to geopoints
  • Users can comment on feed posts
  • Users can like feed posts and comments

I'm certain that Redis as a cache can provide me with what I need if this app were to come under high load, however I am a bit puzzled about the following options:

Write-through cache

Java app -> Redis -> DB

Seems like the best option data consistency-wise, in that the request will only return when the database has been updated. However the disadvantage implies that no batch updates are possible.
If the app were to come under heavy load, this doesn't seem to be the best option.

Write-behind cache

Java app -> Redis -> Java app (batch config) -> DB

Seems like the best option performance wise. In case of heavy load, data is transferred from the cache to the DB through Java (configuring batch sizes, delay etc).
However, the disadvantage seems that if there is a system failure on Redis, all in-memory data is gone (?)!

My question is simple:

  • Does a high-availability cluster with write-behind cache solve all of my problems?
  • Does high-availability write to disk (Redis), guaranteeing that no data will be lost?
  • Even with write-behind cache, is it the safest course of action to add a message queue (eg RabbitMq) or is this generally not necessary even under high load?

Missing posts in a user app like this in case of failure is a big no-no, so I wonder what is your view on this, and what is the best resolution of the problem.

4
  • I haven't seen this kind of setup before, but it seems unnecessary to me. I'll need to ask some questions to understand: Is this to write data into the db? Where is "java app"? Is redis on the server side? – Todd Dec 29 '20 at 10:29
  • @Todd DB directly is a bottleneck if the load is high. Imagine lots of comments / likes in a short timeframe, using a cache to write, and batch updates (or queue) seems necessary to avoid too many writes to the db. The Java app is the backend, taking care from the REST endpoint to the cache ->.db. – Trace Dec 29 '20 at 10:37
  • 2
    You might be right, but it's possible that you are pre-optimising. Make sure you measure and improve.Do you have data to back-up: "batch updates seems necessary to avoid too many writes to the db". Again, you might be right, but no one knows until they benchmark. Generally, it's cheaper to upgrade the hardware, and optimise the database itself; than to complicate the stack further (which costs more and leads to new bugs). There is also a lot of good db tuning that you can do for writes - you can find good DBAs online that will work hourly. – Todd Dec 29 '20 at 13:10
  • @Todd Thanks for the comments. You may be right about the premature optimization. I'm going to keep this simple and start with a simple Spring Boot - MongoDB system and perform some tests first. Perhaps the extra cache really isn't worth it initially. Cost is an issue, as this my own pet project. – Trace Dec 29 '20 at 13:20
3

Your question is specific, so I'll answer it head-on first.

I anticipate that "write-behind" cache will be better. "write-through" might help with occasional spikes in ingress data, but won't alleviate the problem of disk throughput (and transaction throughput).

Of course, this would need to be verified in real-life with benchmarking.


My recommended implementation

I strongly recommend that you don't overengineer your architecture. Less is more.

YAGNI: Start simple and tunable, then update your architecture as needed.

1. Bypass the Java and Redis, and write directly to the database

  • see https://colossal.gitbook.io/microprocess/definition/data-web-gateway. This approach lets you write directly to the database over HTTP.
  • Choose MariaDB(MySQL) as the database and choose the InnoDB engine; I am quite sure that it generally has higher throughput than PostgreSQL, and supports a higher number of concurrent connections (~200k vs 500) - but you should measure that yourself using the target hardware/VM that you plan to use. (see [Compatible Database Performance Comparisons] in https://colossal.gitbook.io/microprocess/database-system/introduction)
  • Have a table with the available Database-Web-Gateway endpoints (that you can add to in the future) and have your client connect to a random one; select * from View_DataWebGateways order by RAND() limit 1;
  • Use a writeable view so that you can point straight to the posts table today, and be able to change that in the future. Query something like: insert into WriteView_Posts (...) values (...). The same goes for PostComments and PostReactions
  • (Install MySQL and the Database Web Gateway on the same VM)

2. Optimise DB and Hardware

Right off the bat, you need a DBA and SysOps person who you can call on to scale up your DB and VM/Hardware and stay ahead of your growth curve. You'll probably find that starts happening in say 2 years anyway. If it happens sooner, your business is doing great and you can now hire more programmers yay!

3. Logical Sharding

With posts, you have a natural shard topic - the post. You won't have PostComments that belong to two posts, they will have an affinity for the Post. You also have Geo affiliated data, so I am guessing that people close to those locations will want that data more - with Sharding you can master that data in a closer data center.

Create a "Management" database to hold a Shards table, as well as the DataWebGateways table. This will be a replicated database with one master, and each new shard will have a read-replica of this.

For each new shard:

  • Create a new VM, create a new Database with a numbered scheme "Posts_1", "Posts_2". (there is NO need to replicate these databases)
  • Make sure you have a read-replica of the Management database to this MariaDB instance.
  • Insert the relevant Shard and DataWebGateway records into the master instance (manually I guess).

Now, you can make the View_DataWebGateways view a bit smarter. Let the client get all of the records, so that it can randomly choose one to use, while pinging the others to see which is closest (by latency), then switch to that.

4. Staging Table

It's probably best that you just create another shard, but if you do testing, and find that batching greatly improves insert performance, then you can do the following:

  • Change WriteView_Posts to point to PostStaging instead of Posts table.
  • Benchmark different MariaDB engines, and select the right engine and configuration for the PostStaging table that optimise for Insert performance. This table can be configured to a dedicated Disk Volume that is also benchmarked to perform best for the simulated load (sequential reads).
  • Ensure you don't have any constraints or triggers on this PostStaging table. (While Posts can)
  • Create a Timed-Interval Microprocess which batches inserts into Posts from PostStaging every X seconds.

Now, if the power goes out, you don't have to lose your "cached" posts. This also helps because you can also redirect posts to the appropriate shard by geography at this point.

5. "High load" probably means Reads not writes

  • Add more VM/hardware: create Read-Replicas of Shards within the same datacentre, and add DataWebGateways records with ReadOnly flag for the client to use. The client will do all reading from read-replicas, and writing to the single write master DWebG.

6. Curate feeds with FeedViews

Avoid caching at all costs - https://colossal.gitbook.io/microprocess/building/caching

If you would like to curate specific geographic feeds (eg. Germany, France, ...). While a geography shard server is great, Germans might want a mix of posts from around the world, while the French only want French posts - that's up to you to figure out.

To make this work:

  • Have a crude view View_CountryFeed and use it like select * from View_CountryFeed order by post_id desc limit 10;. This will only work on a single shard to begin with.

When that is reaching its limits, improve your algorithm and also introduce manual materialisation with the following:

  • Log interest in posts per country
  • Run your algorithms as microprocesses
  • The algorithm parameters will be shared on the Management replicated database
  • The algorithm will run in isolation on each Shard grouping to Country IDs
  • The Management database will be configured on which shard to store the feed
  • A microprocess will pull country feeds from other shards to aggregate. Each shard will pull the right CountryIDs as configured. "Pulling" is done on the resource-sensitive side so this can be tuned per-shard to minimise resource usage.
  • Update View_CountryFeed: having PostID, PostBodyJSON, ShardID. The ShardID is important because that's where post interactions will be sourced from and published, while the PostBodyJSON will enable efficient rendering of the post by the client.

7. Realtime Notifications

Database Web Gateway will be able to help here to in the future. see https://colossal.gitbook.io/microprocess/database-system/feature-gaps. When the CountryFeed table is updated, a trigger will fire and notify the DWebG which will rerun subscribed Views and send new records to the subscribed web clients.

2
  • Thanks for your reply. I was actually thinking about using MongoDB for horizontal scalability, and it also allows geo indexing. – Trace Dec 29 '20 at 14:28
  • 1
    Yeah, MongoDB should be doable @Trace – Todd Dec 29 '20 at 14:32
3

There are already very good answers. Here are my comments:

Does a high-availability cluster with write-behind cache solve all of my problems?

The design sounds reasonable.

One thing worth testing in your benchmark, is the how it performs when it serves heavy write and read at the same time, since Redis is single thread.

If the data size is huge, caching all of them would be harmful. In this case, you may consider expire the Redis key (for example, configure Redis to delete items on an LRU manner).

Your data schema is important too, since complex data structures/queries may be less efficient to cache in Redis. With this consideration, it is possible that

  • design the strategy that prioritizes some requests, or
  • make the caching mechanism serve for some queries, while database reading serve for the others.

Does high-availability write to disk (Redis), guaranteeing that no data will be lost?

Note data could be lost in some rare case. Assume you are using Redis cluster and write to master, if the master then dies before the new data gets populated to slave, the recent written data will be lost.

Even with write-behind cache, is it the safest course of action to add a message queue (eg RabbitMq) or is this generally not necessary even under high load?

If the data size is not huge and you are persisting all the data in Redis, an alternative option is RedisGears. It supports write-behind pattern to synchronize Redis data structures to a backing data store (Oracle, MySQL, Cassandra, and Snowflake are currently supported) with RGSync recipe. MongoDB is not included yet though.

Regarding the database, maybe also worth considering Elassandra, which is a combination of Cassandra and ElasticSearch, so it is good at horizontal write scaling and failure-takeover (Cassandra), and geo-spatial query (ElasticSearch).

1
  • I've given it some thought and decided to (at least initially) skip the Redis cache altogether. I don't know if the database will actually a problem in terms of performance. I might investigate a strategy to archive old feed items per place / user to keep the collections relatively small and enhance performance. Regarding the database to use, it's a dilemma currently. I have to take into account complexity of general devops (I'm a Java guy), pricing and existing cloud infrastructure possibilities (I'm going to the Cloud for all services rather than managing everything myself). – Trace Jan 1 at 13:51
0

I very much agree with Todd with the advice:

I strongly recommend that you don't overengineer your architecture. Less is more. YAGNI: Start simple and tunable, then update your architecture as needed.

Specific Answers

To answer your specific questions but reordered slightly:

Missing posts in a user app like this in case of failure is a big no-no, so I wonder what is your view on this, and what is the best resolution of the problem.

First, we need to be clear about the problem. Implicit in your question is the assumption that write throughput is a big problem to solve. Buffering will happen in Redis when the write-throughput has exceeded the single master write-path of a MongoDB cluster. You are then considering what might happen if Redis crashes. This is all good thinking. Yet is optimising write-throughput the major architectural challenge in your service?

You describe your service as news aggregation. We might expect that reads might dominate. Writes are be many orders of magnitude smaller for news. If it is indeed news you are publishing you can consider buffering writes at the client under high write (e.g., browser local storage). If you load-shed writes you can add simple logic at the client can try again using an exponential back-off give your backend time to catch up (else heal from an outage). A tiny write message queue at every client is easy to implement. You can expect it to be reliable and easy to test. Only if your service is not accepting writes, and the user deletes their client app (else clears browser local storage), will they lose work.

If your service is a bit more like an Uber or Lyft ride-sharing service then write throughput will be critical. With ride-sharing there are a huge number of jobs published constantly and searched for in a geospatial manner. Write throughput and graceful outages are critical to those services. In which case you can research how those companies solve the write-path problem.

Below I will propose two solutions that I would investigate. One that optimises for write throughput and another that might be more suitable for a low budget "news aggregation" service that can scale up later.

Does a high-availability cluster with write-behind cache solve all of my problems?

Operational complexity and recovering from failures is likely to give you the biggest real-world business problems. Studies of big systems indicate that it is "grey failures" are the most tricky sort of problems that lead to extended outages and data loss. The moment you integrate two products on your write path you can expect complex interactions under failures and load. In theory, yes, a highly-available cluster with a write-behind solves all your problems. In practice, I would hesitate to go with a HA Redis Cluster in front of a MongoDB Cluster.

Does high-availability write to disk (Redis), guaranteeing that no data will be lost?

Well, Redis is an excellent product. In the past, it has had some hiccups with data loss. I have used Redis myself as cache in a few architectures. Would I use Redis as a silver bullet to optimise the write-throughput a geospatial based web-scale system? No.

IMHO opensource products grow up to fulfil the core needs of their community. They then bolt on additional features that serve as wide a set of needs as possible. People then get into trouble when they push those additional features to the extreme. You then encounter bugs that haven’t been found and fixed by the wider community as you are the outlier. Redis excels at being a low-latency cache optimising the latency of the read path. Using it to optimise the write-path of a very high write throughput geospatial system would be something I would be very nervous about in practice. I think Redis is great and I will continue to deploy it to optimise the read-path. Yet I would go with a specialist solution to optimise the write-path.

If write-throughput is the major challenge than I would look at Apache Kafka or an alternative. It has an architecture that is very different from traditional massage brokers so that it can scale to "IoT" levels of writes. You can then have consumers that update your main data store and possibly a separate geospatial index. If a consumer was buggy/lossy you can easily replay the data stream to “fix-up” your geospatial index service or main store after you have fixed the bug in your consumers. I would then have a single product on the durable write path. If the secondary writes to the main database or index service fail the ability to easily replay a stream of events will be invaluable. Engineering to make it easy to recover from the unexpected is money better spent than trying to eliminate the unexpected. A large number of real-world outages are caused by human error so designing for failure recovery is critical.

Even with write-behind cache, is it the safest course of action to add a message queue (eg RabbitMq) or is this generally not necessary even under high load?

Well, RabbitMQ is an excellent product. I have used it myself. If you can buffer the writes at the client under high-load, and do load shedding in-front of RabbitMQ, then it might be a great fit. It would be low-latency and allow you to update the main database and possibly a separate geospatial index. Using a message queue to decouple and independently scale microservices is a great strategy. Yet it will add operational complexity at go-live when it might not be needed until the service starts to take-off. So I would be tempted to start without it and then use it as a way to break apart a simple initial architecture into a more complex architecture at a later date.

My Two Suggested Approaches

(1) For a "mega-scale" Geospatial System with both high writes and high read:

If write-throughput is a big challenge I would evaluate both Apache Kafka and Apache Pulsar as the initial durable write-path. Those are very highly scalable messaging engines. Kafka is often used as the transport in asynchronous microservices architectures so we might expect it to have satisfactory latencies at high throughput. I would recommend having an edge "news publish" microservice that exposes RSocket over WebSockets to the clients. Clients would push the new news message over WebSockets as a request-response interaction. The RSocket server would simply apply security (validate the user), validate the payload, write it into Kafka/Pulsar, and respond. I would add logic at the client to hold the message in local storage and periodically retry if it got timeouts or errors. I would exponentially back-off in the retry logic to allow the service to recover.

A news aggregation service will need scalable reads. Your proposal is to use a MongoDB cluster. This is because it does geospatial queries. That would work. If you were looking to scale reads to a much higher level you could consider using a more scalable main database such as ScyllaDB and deploy separate geospatial query service such as Elastic Search. The consumers that read new news from Kafka can first write to the main ScyllaDB database. They can then update a geospatial index in Elastic Search. Elastic Search is often used as a secondary index for both free text search and geospatial indexing. As you are a news aggregation service deploying a dedicated free text search index may also be useful.

(2) For a "start-up budget" system with initial low writes and modest reads:

While we all want to build a service that can scale to huge loads from launch often the reality is that the main threat to a business at launch is over-engineering. It makes business sense to initially focus all engineering effort on the user experience while using a simple and cheap backend. Facebook was a college toy at go-live. Amazon started off selling books from a single workstation computer that beeped whenever an order came in. Demand scales up over time and only after you have a great product deployed. If it is a great business idea and a compelling product then a few hiccups as you grow and scale the architecture will be fine.

PostgreSQL is a traditional relational database. It now does binary JSON storage where you can index into the JSON fields. It also supports geospatial indexes. It also supports free-text searching. It is very easy to run locally to develop against. Better yet major cloud providers support it with lots of automation, high-availability, automated backup/restore, point-in-time restores, and monitoring dashboards. Amazon RDS lets you run PostgreSQL and they can seriously scale it up for you with a few clicks. You can start very cheaply and then add HA easily later then scale up to bigger and bigger database servers as you grow. That gives you time to then fix the real performance bottlenecks rather than guessed problems.

You can start with an edge microservice that will load-shed rather than do too many writes into PostgreSQL. The client can buffer the write and try again later. Then start off with simple code that does all of the writes against PostgreSQL for the main document and geospatial indexing. Spend the engineering effort on the frontend and business features. Later you can put RabbitMQ between the edge microservice and PostgreSQL. Then you can either break up or swap out PostgreSQL.

At a later date, you might create a separate geospatial index in Elastic Search. Later yet you might choose to move actual documents out of Postgres and into ScyllaDB. Do we know which things we should do in which order today? No, we cannot. Instead plan to evolve the architecture. Maybe just splitting from one PostgreSQL server into three servers, one dedicated to each of geospatial indexing, binary json, and free text search might work? That sounds like a great intermediate step before swapping in Elastic Search or ScyllaDB. We don't know but we can be flexible.

9
  • Thanks for profound answer. I'm a bit puzzled as for why Todd and you are very much in favour of using Postgres over Mongo, since Postgres is more difficult to scale and feeds typically have a document-like format. Full text search isn't really a concern of mine which would appear to be Elasticsearch strongest point; these feed posts are just guestbook like messages that people can leave in Google places through mobile app. Thanks for sharing your experience! (Next comment an additional question) – Trace Dec 29 '20 at 21:21
  • My main concern is not necessarily with feed items only, but also 'likes' and 'comments' which may be a lot... I will follow your suggestion to start with simple use case, I thought simply REST -> Java app -> Mongo. Can't go simpler than that I guess (btw I have zero experience with Scylla therefore would like to avoid it initially). If I know that I can solve my problems initially by just throwing more resources against it, that would work for me. Do you think that Kafka is something I would preferably implement on an initial launch? – Trace Dec 29 '20 at 21:23
  • Lastly, I would also like to avoid web sockets if I can for the same reason that I don't have a lot of experience with scaling them and from what I've heared, this can be quite painful. SSE seems to be an easier road to take. – Trace Dec 29 '20 at 21:25
  • Just read a bit more on the use cases of Kafka, and I think that I will add it, for the sake of having peace of mind. I think geospatial indexed queries in Mongo (or Postgres GIS for that matter) will do initially. Thanks again for your advice. – Trace Dec 29 '20 at 21:41
  • 2
    having used mongo in production for a few years in a large system i will never be using it again. i hope that you find it reliable to operate at scale. certainly, it is easy to get started with and ease seems to be your key focus. keep things simple and be ready to replace parts if they let you down. good luck! – simbo1905 Dec 30 '20 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.