3
  • We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications.
  • This data changes rarely. Let's say it changes once every two weeks.
  • This data is also relevant for other applications in other domains.
  • We now want to make this data available to these other applications in our distributed system.

What is the best way to make such rarely changing data available to other applications?

Is using a messaging system overkill for rarely changing data? Would a REST interface enforce too much coupling onto other applications?

4
  • 6
    How soon do the other applications need the change? Or conversely, how long can they be out of date? The frequency of the changes tends to be less important than the urgency of having the updates.
    – JimmyJames
    Jan 17, 2023 at 14:55
  • @JimmyJames The other applications can be out of date for some minutes. Jan 17, 2023 at 14:59
  • Do these changes happen on some sort of schedule? Is there a system managing those updates such as a scheduling system?
    – JimmyJames
    Jan 17, 2023 at 18:04
  • @JimmyJames No, there is no regular schedule. Jan 17, 2023 at 19:40

2 Answers 2

4

In this answer, I'm going to use SLA to mean the maximum amount of time that a downstream system can be behind the primary source.

There are basically two options here:

  • Downstream systems poll for data on a frequency smaller than the SLA (e.g. 2 times per SLA period)
  • You detect changes and alert the downstream systems

The first option is definitely the simpler one. It's also the easiest to make robust. Unless you have a really large number of downstream systems, I can't recommend doing anything else without some other mitigating circumstance that you haven't mentioned. As @DocBrown notes in the comments on @Hans' answer, you can implement a HEAD operation which provides the last time the data was updated. This will minimize the cost of these polling calls. The clients then, just need to keep track of which update they have successfully processed. It's important that you don't change the last update date-time until you have fully processed it. This will make your updates more robust. Downstream applications should also have account for being repeatedly unable to handle a given update. I recommend at least 2 checks in each SLA time period. If you only check once, even a simple networking hiccup could result in missing the SLA.

If the volume of these checks is too high (unlikely) then the only thing that really changes is that you need to notify clients. There are many options, they aren't that hard to implement if you've done it before. The trick isn't getting them to work, it's making them robust to errors. The most likely failure mode is that they stop receiving updates. I'm going to guess that you don't need this and if you decide you do, the details are outside the scope of this question.

5
  • In comments, you asked OP if the data change on a schedule. OP said "No," but what if they could negotiate that constraint? What if they could get the actor who drives the change to buy in to the idea that downstream applications will not see and apply the change it until;..., midnight, or the top of the hour, or the next whole multiple of five minutes, etc.? Might that not make everybody's job a little bit easier? Jan 17, 2023 at 23:46
  • Long polling is reasonably robust and simple - if your platform supports it.
    – user253751
    Jan 18, 2023 at 14:21
  • @SolomonSlow Yes, that's exactly why I asked. If the changes were driven on a schedule, there may be other options.
    – JimmyJames
    Jan 18, 2023 at 14:29
  • @user253751 I'm not a huge fan of long polling in general. It's kludgy, IMO. If the SLA were a lot tighter, it might be worth considering, though.
    – JimmyJames
    Jan 18, 2023 at 15:41
  • I'll add that many data stores have some sort of notification option, like PostgreSQL's NOTIFY - there is no need to decouple the data from the notifications.
    – jaskij
    Jan 18, 2023 at 21:21
2

For low-frequency updates a message broker indeed feels like overkill.

You can emulate a publish/subscribe behavior on top of REST endpoints:

  1. The information provider offers the data through some ordinary endpoint. Consumers can use that endpoint at any time to get the data, and if they don't need to get timely updates that's all they might want to do.
  2. The information provider has an endpoint /subscriptions where information consumers can register and deregister interest using POST and DELETE actions. A subscription request would include the URL of an endpoint belonging to the consumer which is notified when new data arrives.
  3. Consumers will be notified about new data through the endpoint they registered with the provider. They can then use the provider's data endpoint to access the current data.

By using a /subscriptions endpoint, you reduce coupling between producer and consumers, as the producer does not need to know in advance who their consumers will be. Consumers need to know about the producer anyway, so requiring them to subscribe seems reasonable.

Of course, the management of subscriptions introduces some additional overhead, including the handling of dysfunctional subscribers (should notifications be buffered? for how long? should alerts be sent? etc.). You also open some possible attack surfaces if these interfaces aren't entirely within your organization, so you might need authentication checks and DoS protection etc, which you probably have anyway but which make the system bulkier.

5
  • 1
    This looks like a good start, however, I guess there should be some pull mechanism for consumers which are not online during the latest update, or consumers which don't want to listen actively. And maybe a pull mechanism is fully sufficient for the OP.
    – Doc Brown
    Jan 17, 2023 at 16:35
  • 1
    By using push subcriptions you increase complexity a lot. How about long polling instead?
    – user253751
    Jan 17, 2023 at 16:36
  • 2
    @DocBrown: The first point describes a pull mechanism - data would be accessible at all times. But polling every few minutes for an update that may come after two weeks sounds somewhat excessive. Jan 17, 2023 at 16:48
  • 1
    @user253751 using long poll would be possible but also feels like some kind of busy waiting, both the producer and all consumers would be somewhat active while nothing really happens. Jan 17, 2023 at 16:52
  • 2
    @Hans-MartinMosner: the required pull frequency depends on the time between the arrival of the new data and the time when it needs to be forwarded for the consumers. If the OP is right with "should be not more than just a few minutes", than that's the frequency constraint. However, the API could offer a mechanism to let the consumers check the timestamp of the last update before pulling the whole dataset. So consumers can check that timestamp every two minutes, and just pull the full dataset when that changes, so ~ every two weeks.
    – Doc Brown
    Jan 17, 2023 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.