Design considerations for data shared across multiple users: is Redis a good candidate?

Question

I have an app that has an assets table like the following:

user_id	code	current_price
1	ALUP11	12.5
2	ALUP11	11.9

user_id and code are unique together and current_price is updated if the user triggers a task that fetches the values from another service. I'm also using CQRS so when this task finishes it updates the read model asynchronously through messaging communication. So if an user has N assets, N + 1 messages will be dispatched to the broker when he/she triggers the task that updates current_prices.

I'm starting to think that this was bad design because the current_price of an asset isn't something that should be bound to an user but rather something global, shared across all users.

My initial idea is to move this data to Redis and query from it to construct the aggregations that needs this data when updating the read model. If I go for it, I see a trade-off:

1. On one hand it'll be expensive to update the read model because when updating the current_price of the assets of an user I need to perform the aggregation on a small subset of entities, while moving current_price to a global DB forces me to update all assets of all eligible users.
   • Thus when this global tasks runs, it will publish M * N - where M is the number of users and N is the number of assets for each user - messages to the broker, which doesn't seem scalable at all to me;
   • I can think in a way to publish only M messages to the queue, i.e., instead of having a task that will update only one asset of a given user, I could have a task that updates all assets of a given user.
2. On the other hand, the load on that other service will reduce significantly because the number of task invocations will reduce dramatically. This services fetches the data from an external and paid API so some costs reduction will also happen here.

I'm inclined towards 2. because I think it'll be simpler and more effective to scale (considering that I can change the task logic as described in the second point of 1.).

Real-time accuracy is NOT crucial and it's "OK" to lose the granularity of updates for individual assets.

NOTE: this is not a perceptive problem YET, but I think this will hurt myself in the future.

Have you experienced something similar? Is my train of though in the right direction?

Answer 1

Here's what I heard you say:

1. We need to know "recent", but not up-to-the-second, prices of commodities like this one.
2. Displaying "stale" prices carries some cost.
3. Each query to refresh a price incurs some small API cost.
4. Commodity prices do move, but they move "slowly".

You can cache prices in an RDBMS or in Redis, whichever you're more comfortable with, as both can offer sensible indexing.

The big things you need to do are

• quantify the cost of (2.) stale price quotes, and
• choose a refresh strategy.

We can evaluate (4.), slowness of price movement, against historic quotes per commodity, or averaged across all commodities. And (3.), the API cost, comes straight from your API contract.

So now we need to write down the cost of (2.) staleness, of delivering a (t1, price1) quote at time t2 when the API would reveal a code was actually trading at price2. Historic price movement will suggest a relationship between Δt and Δprice. Knowing t2 - t1 lets you estimate whether Δprice likely is "large".

Balancing that against the known API costs suggests a refresh strategy.

---

Whenever you ask the API about code, you should definitely record the tuple (code, price, write_time, write_time), showing that at write_time it traded at price.

When displaying the price of a commodity, update that last element with read_time, so we can track "popular" commodities that are frequently read by many users and should be frequently updated.

We might additionally tack on an estimate of "error" or "uncertainty" after the "read time". This has the advantage of being indexable, for efficient queries of the most badly out-of-date commodities. Which brings us to....

refresh strategy

You can query the API

• from a background daemon, and/or
• from the process displaying results to the user

Imagine your vendor's API is "slow" at delivering price results, too slow for an interactive web user to wait on. Then you would choose to have a daemon issue all queries, letting each user process consult the local cache and hope for the best.

If the vendor's API is "fast", then each user process might choose to send it queries, cache results, and display results. That brings us to another choice.

You can

• display only cached results to the user, or
• issue queries, await results, then finally display them to user.

The first leads to predictable UX response times. If you prefer the second, you still have the option of making a kafka pub-sub request to a daemon then awaiting response, or doing the API query in the context of the user process.

background daemon

The daemon is responsible for minimizing the (2.) staleness cost function of anticipated display requests, subject to (3.) constraints on API spending.

Suppose that all commodities are equally volatile at all times, and that all users are equally important. Then we pick some "acceptable" staleness, perhaps a threshold of ten minutes, and the daemon simply asks the local RDBMS / redis cache for commodities that are going stale. It cycles through them, issuing API requests and updating the cache. If it has trouble keeping up we can run more than one instance of the daemon.

Now suppose that some commodities are more popular than others, perhaps following a Zipfian distribution. Use the read_time mentioned above to prioritize commodities. If ALUP11 is popular, we will cache price1 at time t1, and then display that same price1 to users at times t2, t3, t4.... Your design challenge is to figure out by which time t5 should the daemon have already consulted the API and cached an updated price.