Questions tagged [pubsub]

Publish/subscribe is a method for wiring software components together. One component publishes data, and subscribers receive events containing the data. This decouples the components so they do not need to know about each other; and it allows for configuration to wire components together without recompiling code.

Filter by
Sorted by
Tagged with
0
votes
5answers
178 views

How to deail with a hypothetical situation in which a pub/sub cycle gets into an unending recursive loop?

Let me explain what I mean. Imagine A subs to event b. In such case A pubs event a. B subs to event a. In this case B subs b. This is a full-blown circle. How does a pub/sub cycle deal with such ...
-3
votes
0answers
19 views

Scaling in P2P Kafka vs JMS/MQ

Consider using Kafka as a traditional point-to-point queue (by using just one consumer group). If you have, say 4 partitions of your Kafka topic, you can have a maximum of 4 consumers in your consumer ...
0
votes
0answers
21 views

Architecture for Data Pipeline (ETL) on GCP

I want to create a Data Pipeline on GCP, after a lot of research I decedide to use Pub/Sub to send the data and Dataflow (Apache Beam) to subscribe to that Topic in Pub/Sub and save the data in the ...
0
votes
1answer
78 views

Event sourcing access by reference

In an event sourcing architecture, what is the typical pattern for passing information about related objects (aggregates)? For example, in a order processing system, should OrderCreated event (...
2
votes
1answer
65 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
3
votes
1answer
105 views

Pub-Sub pattern and instance replication

I am porting an event-driven architecture to a container environment (Docker) and I wonder how to take advantage of replication while using pub-sub: if I just increment replicas I end up with many ...
0
votes
0answers
35 views

Using a message queue or pub / sub solution for connecting frontend to backend instead of REST APIs

I own a dell r620 server at home and want to make more use of it. Typically its my staging environment as my internet is currently ipv6 with dslite so I can’t port forward. But I can easily call and ...
2
votes
1answer
67 views

Publisher-Subscriber architecture with central registry

Problem description Publisher-Subscriber architecture with a central registry where agents can either promote their capabilities or search for a given capability. The project must be developed with ...
-4
votes
1answer
47 views

application can not handle certain amount of RPS when 3k+ sessions connected

i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it. first i will explain the application structure. the ...
1
vote
2answers
56 views

How to agree on message schema in a Publish–subscribe pattern

I'm working on a project that uses PubSub(GCP), my question is not specific to GCP, it's more regarding to the architectural pattern(I'm used to statically typed languages, and I have a hard time ...
0
votes
0answers
28 views

Is it ok to have a mix of PubSub and Request/Reply in a WebSocket server (focus on unit-test)?

I'm using Node.JS to build a system where the data get consumed by WebSocket requests, instead of classical REST API calls. WebSockets were used for realtime bidirectional communication, but then were ...
-1
votes
1answer
36 views

pub/sub events to/from microservices in different languages

I'm starting a brand new project implementing microservices with domain driven design. We will have microservices written in different languages like C#, Python, and Node. I'm thinking about hosting ...
0
votes
0answers
19 views

How to scale raft state to dozens of followers?

I'm looking into developing a distributed pubsub with p2p messaging akin to ROS where instead of being brokered, messages are transported directly from producer to consumer. XY problem - each node ...
0
votes
0answers
13 views

How to design event processing on cloud with write backs to primary data source

I have an application which mainly deals with business workflow management. In this, there are going to be many events which will be processed async. The application raises the event and these events ...
1
vote
0answers
46 views

Flowchart symbol or pattern for Publisher/Subscriber or Listen/Notify

We have a system consisting of Detectors Converters/Transformers Business Rules Engines and State Managers Views/Displays/Applications There are several potential ways these may interact and share ...
2
votes
0answers
34 views

Data change event publishing by microservice vs separate application - best practice

Suppose there is a microservice that has an restful HTTP API for CRUD operations on a database - nothing fancy, but there is a journal table of all changes recorded for audit purposes. Suppose also ...
1
vote
0answers
53 views

Event-based communication between microservices

I'm currently looking into NestJS to use as the framework for a new e-commerce project. The goal is to build multiple loosely-coupled services, which can communicate with each other using the pub/sub ...
0
votes
0answers
39 views

Preferrable way to send push messeges?

I am making a chat-app using the XMPP protocol and ejabberd as XMPP server. I want to build a push notification functionality for the app. the is similar to whatsapp. I came to know 3 diffrent ...
2
votes
2answers
114 views

Pros and cons of consuming events in batches

Suppose I have a message broker that allows consumers to specify a batch size. What are the advantages and disadvantages of consuming events in batches compared to consume them one by one? The ...
-1
votes
2answers
70 views

Using Apache Kafka for rare events [closed]

There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
0
votes
0answers
75 views

Are message identifiers and offsets the same thing in Kafka?

I am reading up on Kafka and encountered the following on one of the tutorials. The producer sends the following to the broker - unique message identifier, timestamp and a binary payload. The ...
3
votes
0answers
61 views

Run a job every hour with Lambda or something else?

I am working on a SaaS project that will have a trial when the trial is ending I get a webhook notification when 3 days are remaining. I do some stuff with this and one of the things is I update user....
36
votes
3answers
15k views

Ways to share DTO across microservices?

My scenario is as follows. I am designing a system designed to receive data from various types of sensors, and convert and then persist it to be used by various front-end and analytics services later....
1
vote
1answer
60 views

What is a good strategy for managing PubSub subscriptions

I am working with Reddis but this could apply to any PubSub implementation. Essentially we are trying to implement an event system using Reddis's PubSub capabilities. As an example UserA creates a ...
0
votes
1answer
170 views

How can an event subscriber catch up with missed events?

At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time. For example, a service is running and subscribes to events on ...
0
votes
1answer
38 views

What types of Google Cloud application should I consider for something running as a continually available service that subscribes to a pub/sub?

I want to create an application that will run as a service that subscribes to a Google Cloud pub/sub topic, and delivers emails based on 'alerts' that it receives through the pub/sub subscription. It ...
0
votes
1answer
176 views

Which message pattern should I use?

I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
0
votes
1answer
224 views

Consuming messages directly from a queue which belongs to a third party system

We are working on a project which interacts with multiple third party systems and they need each other's data to function. Now, for some data which needs to be in sync in multiple systems, we are ...
0
votes
2answers
2k views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
1
vote
1answer
121 views

Handling application-wide notifications from various layers of the application?

I have recently started updating portions of our corporate platform, and I've been trying to find ways to make the entire platform more efficient, better structured, and just cleaner in general. With ...
0
votes
1answer
2k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
2
votes
2answers
175 views

Name of this pattern

I create an object that holds the status of a job. The job has many stages, some concurrent. Worker threads constantly monitor the status object to see if it's ready for the next stage to begin, if it ...
0
votes
1answer
1k views

How to load test Publisher/Subscribing queue?

I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second). Problem statement : The pub sub ...
2
votes
1answer
287 views

Architecture pattern for temporary message queue subscriptions with websocket clients

We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
1
vote
1answer
190 views

How do I set up short-lived queues?

This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
1
vote
1answer
360 views

Publish and Subscribe (data transfer) with permanently offline nodes. Are message queues a good fit?

The general question is what kind of mechanism can I use to transfer data to and from publishers and subscribers where publishers or subscribers can be permanently offline? Can message queues be used ...
2
votes
3answers
169 views

Is this an inefficient use of the Subscriber (PubSub) pattern?

I have a Publisher object that raises an event at regular intervals (think of it something like a clock tick). I then have 100s (or 1000s) of subscriber objects (all instances of a single type or two) ...
1
vote
1answer
2k views

What is a good diagrammatic way to represent async event communication between two systems?

What is a good diagrammatic way to represent asynchronous event communication between two systems (the pub sub model)? I am not looking for tools but more of a representation. Is it a sequence diagram,...
4
votes
1answer
1k views

pubsub with multiple instances of each consumer

I'm currently looking into setting up a publish subscribe messaging infra structure for our microservice based platform. The new setup is meant to replace our current kafka based one, with something ...
2
votes
1answer
441 views

Should API gateway wait until all micro services are done?

for our project, we are using micro services with API gateway pattern. One of the features that the client do is to generate a report. Client, fills a form and uploads a file and our system will ...
30
votes
3answers
25k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
8
votes
2answers
6k views

Relationship between RESTful URIs and PubSub topics

I am designing a system which fits quite nicely into a RESTful architecture. Users can navigate a resource hierarchy from a root node, each resource links to other resources, resources have URIs etc. ...
2
votes
1answer
502 views

Is publish–subscribe pattern suitable for my case?

I'm planning to use publish–subscribe pattern with Azure Service Bus in my logistics application, but I'm confused whether publish–subscribe pattern is suitable for my case. My scenarios: Users will ...
2
votes
2answers
1k views

Publish / Subscribe via HTTP Callbacks?

My team is tasked with creating a publish/subscribe system for incoming REST messages. 99% of the time, this system will be used for notifications between different processes on the same cpu, but we ...
0
votes
0answers
160 views

Is NSNotificationCenter an implementation of a PubSub model?

I am trying to understand the subtleties of Publisher-subscriber and similar notification mechanisms. I was checking Cocoa NSNotificationCenter, but it's unclear to me if it can be defined as a PubSub,...
1
vote
0answers
243 views

Lazy loading collection data through granular subscriptions and OOP (Meteor.js)

I am transforming my Meteor collection to a class I've created: People = new Mongo.Collection('people', { transform: function(doc) { return new Person(doc); } }); Person has a bunch of ...
4
votes
3answers
1k views

Are imperative events an anti-pattern?

Are events which are intended to tell listeners to carry out explicit actions an anti-pattern? Imagine a worker service which raises the event WorkStarted at its beginning and the event WorkFinised ...
11
votes
2answers
2k views

How is the publish-subscribe pattern different from gotos?

My understanding is that Goto statements are generally frowned upon. But the publish-subscribe pattern seems to be conceptually similar in that when a piece of code publishes a message, it performs a ...
5
votes
4answers
2k views

Design Patterns for Coordinating Change Event Listeners

I've been working with the Observer pattern in JavaScript using various popular libraries for a number of years (YUI & jQuery). It's often that I need to observe a set of property value changes (e....
0
votes
1answer
326 views

Strategy vs Publiser-subscriber pattern

Let's say I have a channel endpoint where I receive messages. Because I want to process messages in a different way depending on its Header property, rather than create a massive switch I create ...