Questions tagged [pubsub]

Publish/subscribe is a method for wiring software components together. One component publishes data, and subscribers receive events containing the data. This decouples the components so they do not need to know about each other; and it allows for configuration to wire components together without recompiling code.

Filter by
Sorted by
Tagged with
45
votes
3answers
18k views

Ways to share DTO across microservices?

My scenario is as follows. I am designing a system designed to receive data from various types of sensors, and convert and then persist it to be used by various front-end and analytics services later....
30
votes
3answers
27k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
11
votes
2answers
2k views

How is the publish-subscribe pattern different from gotos?

My understanding is that Goto statements are generally frowned upon. But the publish-subscribe pattern seems to be conceptually similar in that when a piece of code publishes a message, it performs a ...
9
votes
2answers
6k views

Relationship between RESTful URIs and PubSub topics

I am designing a system which fits quite nicely into a RESTful architecture. Users can navigate a resource hierarchy from a root node, each resource links to other resources, resources have URIs etc. ...
9
votes
1answer
2k views

Diagramming messages on a service bus

I'm looking for a way to clearly diagram how multiple applications communicate via a service bus. The best I've come up with so far is a sequence diagram, but I really don't like that. Sequence ...
7
votes
1answer
7k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
6
votes
2answers
3k views

Simple, permanent queue system with pub/sub for Node.js?

Highly related https://stackoverflow.com/questions/6021938/nodejs-many-clients-requests-through-one-socket I'm investigating a robust message passing system for this application of mine. The easyest ...
5
votes
3answers
1k views

Are imperative events an anti-pattern?

Are events which are intended to tell listeners to carry out explicit actions an anti-pattern? Imagine a worker service which raises the event WorkStarted at its beginning and the event WorkFinised ...
5
votes
1answer
1k views

pubsub with multiple instances of each consumer

I'm currently looking into setting up a publish subscribe messaging infra structure for our microservice based platform. The new setup is meant to replace our current kafka based one, with something ...
5
votes
4answers
2k views

Design Patterns for Coordinating Change Event Listeners

I've been working with the Observer pattern in JavaScript using various popular libraries for a number of years (YUI & jQuery). It's often that I need to observe a set of property value changes (e....
3
votes
3answers
426 views

What scenarios are implementations of Object Management Group (OMG) Data Distribution Service best suited for?

I've always been a big fan of asynchronous messaging and pub/sub implementations, but coming from a Java background, I'm most familiar with using JMS based messaging systems, such as JBoss MQ, HornetQ,...
3
votes
1answer
118 views

Pub-Sub pattern and instance replication

I am porting an event-driven architecture to a container environment (Docker) and I wonder how to take advantage of replication while using pub-sub: if I just increment replicas I end up with many ...
3
votes
1answer
92 views

Run a job every hour with Lambda or something else?

I am working on a SaaS project that will have a trial when the trial is ending I get a webhook notification when 3 days are remaining. I do some stuff with this and one of the things is I update user....
3
votes
0answers
232 views

Looking for design pattern to represent shared state in music sequencer

I like to give lots of context, but also to state the basic question up top as concisely as possible. Feel free to read a little or a lot. I'm working on a music sequencer, and have been happy so far ...
2
votes
3answers
176 views

Is this an inefficient use of the Subscriber (PubSub) pattern?

I have a Publisher object that raises an event at regular intervals (think of it something like a clock tick). I then have 100s (or 1000s) of subscriber objects (all instances of a single type or two) ...
2
votes
2answers
1k views

Publish / Subscribe via HTTP Callbacks?

My team is tasked with creating a publish/subscribe system for incoming REST messages. 99% of the time, this system will be used for notifications between different processes on the same cpu, but we ...
2
votes
1answer
84 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
2
votes
2answers
161 views

Pros and cons of consuming events in batches

Suppose I have a message broker that allows consumers to specify a batch size. What are the advantages and disadvantages of consuming events in batches compared to consume them one by one? The ...
2
votes
2answers
176 views

Name of this pattern

I create an object that holds the status of a job. The job has many stages, some concurrent. Worker threads constantly monitor the status object to see if it's ready for the next stage to begin, if it ...
2
votes
1answer
511 views

Is publish–subscribe pattern suitable for my case?

I'm planning to use publish–subscribe pattern with Azure Service Bus in my logistics application, but I'm confused whether publish–subscribe pattern is suitable for my case. My scenarios: Users will ...
2
votes
1answer
230 views

Good pattern(s) for pointing to mutable state [closed]

I asked this question once here, but believe the question was unclear. However I'm having a hard time extracting the general problem from my specific case. UPDATE: I've answered my own question ...
2
votes
1answer
442 views

Should API gateway wait until all micro services are done?

for our project, we are using micro services with API gateway pattern. One of the features that the client do is to generate a report. Client, fills a form and uploads a file and our system will ...
2
votes
1answer
79 views

Publisher-Subscriber architecture with central registry

Problem description Publisher-Subscriber architecture with a central registry where agents can either promote their capabilities or search for a given capability. The project must be developed with ...
2
votes
0answers
38 views

Data change event publishing by microservice vs separate application - best practice

Suppose there is a microservice that has an restful HTTP API for CRUD operations on a database - nothing fancy, but there is a journal table of all changes recorded for audit purposes. Suppose also ...
2
votes
1answer
301 views

Architecture pattern for temporary message queue subscriptions with websocket clients

We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
1
vote
1answer
206 views

How do I set up short-lived queues?

This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
1
vote
1answer
3k views

What is a good diagrammatic way to represent async event communication between two systems?

What is a good diagrammatic way to represent asynchronous event communication between two systems (the pub sub model)? I am not looking for tools but more of a representation. Is it a sequence diagram,...
1
vote
2answers
63 views

How to agree on message schema in a Publish–subscribe pattern

I'm working on a project that uses PubSub(GCP), my question is not specific to GCP, it's more regarding to the architectural pattern(I'm used to statically typed languages, and I have a hard time ...
1
vote
1answer
61 views

What is a good strategy for managing PubSub subscriptions

I am working with Reddis but this could apply to any PubSub implementation. Essentially we are trying to implement an event system using Reddis's PubSub capabilities. As an example UserA creates a ...
1
vote
1answer
405 views

Publish and Subscribe (data transfer) with permanently offline nodes. Are message queues a good fit?

The general question is what kind of mechanism can I use to transfer data to and from publishers and subscribers where publishers or subscribers can be permanently offline? Can message queues be used ...
1
vote
0answers
49 views

Using a message queue or pub / sub solution for connecting frontend to backend instead of REST APIs

I own a dell r620 server at home and want to make more use of it. Typically its my staging environment as my internet is currently ipv6 with dslite so I can’t port forward. But I can easily call and ...
1
vote
0answers
78 views

Flowchart symbol or pattern for Publisher/Subscriber or Listen/Notify

We have a system consisting of Detectors Converters/Transformers Business Rules Engines and State Managers Views/Displays/Applications There are several potential ways these may interact and share ...
1
vote
0answers
73 views

Event-based communication between microservices

I'm currently looking into NestJS to use as the framework for a new e-commerce project. The goal is to build multiple loosely-coupled services, which can communicate with each other using the pub/sub ...
1
vote
1answer
100 views

Event sourcing access by reference

In an event sourcing architecture, what is the typical pattern for passing information about related objects (aggregates)? For example, in a order processing system, should OrderCreated event (...
1
vote
1answer
125 views

Handling application-wide notifications from various layers of the application?

I have recently started updating portions of our corporate platform, and I've been trying to find ways to make the entire platform more efficient, better structured, and just cleaner in general. With ...
1
vote
0answers
246 views

Lazy loading collection data through granular subscriptions and OOP (Meteor.js)

I am transforming my Meteor collection to a class I've created: People = new Mongo.Collection('people', { transform: function(doc) { return new Person(doc); } }); Person has a bunch of ...
0
votes
1answer
951 views

Design pattern for modifying state and notifying the other modifiers without a loop

The idea is that I have a set of properties that represent possible options in the app. These options can be modified through a variety of different ways, such as from the UI, remotely, third party, ...
0
votes
2answers
2k views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
0
votes
5answers
189 views

How to deail with a hypothetical situation in which a pub/sub cycle gets into an unending recursive loop?

Let me explain what I mean. Imagine A subs to event b. In such case A pubs event a. B subs to event a. In this case B subs b. This is a full-blown circle. How does a pub/sub cycle deal with such ...
0
votes
1answer
184 views

Which message pattern should I use?

I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
0
votes
1answer
49 views

Pubsub model - publisher broadcast to remote subscribers

With a single publisher(go-routine) and multiple subscribers(go-routine) on same machine, below message hub help create pubsub model: type PublisherHub struct { subscribers map[*subscribmediator....
0
votes
1answer
234 views

How can an event subscriber catch up with missed events?

At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time. For example, a service is running and subscribes to events on ...
0
votes
1answer
41 views

What types of Google Cloud application should I consider for something running as a continually available service that subscribes to a pub/sub?

I want to create an application that will run as a service that subscribes to a Google Cloud pub/sub topic, and delivers emails based on 'alerts' that it receives through the pub/sub subscription. It ...
0
votes
1answer
2k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
238 views

Consuming messages directly from a queue which belongs to a third party system

We are working on a project which interacts with multiple third party systems and they need each other's data to function. Now, for some data which needs to be in sync in multiple systems, we are ...
0
votes
1answer
338 views

Strategy vs Publiser-subscriber pattern

Let's say I have a channel endpoint where I receive messages. Because I want to process messages in a different way depending on its Header property, rather than create a massive switch I create ...
0
votes
0answers
39 views

Broker queue and topic naming conventions

I'm in the process of adding a message broker (ActiveMQ Artemis) to a system currently and while I have my own notions on how I can divide up the namespace, it seems like a good time to get opinions ...
0
votes
0answers
35 views

WebApp architecture with multiple client

I'm investigating how I could achieve a project I've been thinking of for a while. Let me explain the project (I will not go into too much details, but I think what I'll write will be enough). So, I ...
0
votes
0answers
30 views

How to narrow time window for published data

I have a system organized in microservices: we can simplify the structure with backend server, frontend server and client. Data are quite limited (hundreds of redis keys) and each client can recevie ...
0
votes
0answers
40 views

Architecture for Data Pipeline (ETL) on GCP

I want to create a Data Pipeline on GCP, after a lot of research I decedide to use Pub/Sub to send the data and Dataflow (Apache Beam) to subscribe to that Topic in Pub/Sub and save the data in the ...