Questions tagged [pubsub]
Publish/subscribe is a method for wiring software components together. One component publishes data, and subscribers receive events containing the data. This decouples the components so they do not need to know about each other; and it allows for configuration to wire components together without recompiling code.
67 questions
1
vote
2
answers
89
views
Subscribing to a range of topics per instance of a service [closed]
If I have a system where my distributed service is sending live scores of 1000 s of football games from some hypothetical event to millions of clients. The service subscribes to the games from a ...
- 244
0
votes
1
answer
230
views
architecture design: In Pub Sub, how are publishing and subscription servers connected?
I was reading Google's Architectural overview of Pub/Sub and I was curious how
publisher and subscription servers were connected.
From what I understand:
When a message is published, it is stored, ...
- 129
0
votes
0
answers
103
views
Good reason not to do this: Pub/Sub with only 1 topic and publishers are also subscribers
Looking for a good reason to avoid this solution.
Problem
Multiple microservices/SAAS, each storing the same data that needs to be synced (example is customer contact details).
1st is an OAuth ...
- 159
0
votes
1
answer
115
views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101
0
votes
0
answers
91
views
Event based update of In memory Hashmap in multiple pods of kubernetes
I have a service (S1) that is deployed on kubernetes having multiple replicas (pods).
S1 requires some data from database (Cosmos Cassandra Api) for calculation.
This data is constantly updated by ...
- 101
2
votes
2
answers
7k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 131
2
votes
3
answers
472
views
What should we do in face of a failing sub in pub-sub?
I'm working on a microservices application that is implemented from the ground up with MediatR em CQRS. We have a list of domain events that will be published via MediatR [simple pub-sub library that ...
- 510
0
votes
2
answers
206
views
Prevent missing updates using a message broker after creating a task
I'm trying to design a data updates mechanism in my micro-services architecture.
For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
- 109
3
votes
1
answer
5k
views
Event bus vs PubSub
I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
- 179
0
votes
0
answers
112
views
Design : How to divide the responsibility in micro services
I have two micro services built using spring boot,
One micro service is: "payment" Which handles actions related to payments.
Like creating a payment link.
Getting the latest status of ...
- 101
1
vote
2
answers
218
views
Make infinite loop handleable via delay?
I have a Node.js micro-service architecture-based back-end. Services communicate with each other via PubSub. E.g. upon a request to service A, service A messages either service B or service C via ...
- 255
0
votes
1
answer
91
views
What kind of bottlenecks are possible when implementing a distributed workflow using a central point of control?
I was going through https://docs.microsoft.com/en-us/azure/architecture/patterns/choreography, the document mentions in the drawbacks of having a central orchestrator that it can become a performance ...
- 303
0
votes
0
answers
787
views
gRPC client - to - client communication between other Lan network
I have a question about gRPC communication.
I know gRPC has 4 models communication: unay, server-streaming, client-streaming and bi-direction. And the terms "client" and "server" ...
- 101
0
votes
0
answers
137
views
Redundancy approach for Pub-Sub application with events replication
I'm new to the microservices field.
I'm looking for the right approaches regarding how to deal with the challenge of redundancy of a pus-sub application.
Assume my application has has the below ...
- 101
3
votes
0
answers
762
views
Broker queue and topic naming conventions
I'm in the process of adding a message broker (ActiveMQ Artemis) to a system currently and while I have my own notions on how I can divide up the namespace, it seems like a good time to get opinions ...
0
votes
1
answer
80
views
Pubsub model - publisher broadcast to remote subscribers
With a single publisher(go-routine) and multiple subscribers(go-routine) on same machine, below message hub help create pubsub model:
type PublisherHub struct {
subscribers map[*subscribmediator....
- 2,305
0
votes
5
answers
529
views
How to deail with a hypothetical situation in which a pub/sub cycle gets into an unending recursive loop?
Let me explain what I mean. Imagine A subs to event b. In such case A pubs event a. B subs to event a. In this case B subs b. This is a full-blown circle. How does a pub/sub cycle deal with such ...
- 27
3
votes
1
answer
225
views
The publish-subscribe pattern: garbage-collecting old subscriptions
I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
- 725
-4
votes
1
answer
146
views
application can not handle certain amount of RPS when 3k+ sessions connected
i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it.
first i will explain the application structure.
the ...
- 21
1
vote
2
answers
134
views
How to agree on message schema in a Publish–subscribe pattern
I'm working on a project that uses PubSub(GCP), my question is not specific to GCP, it's more regarding to the architectural pattern(I'm used to statically typed languages, and I have a hard time ...
- 21
3
votes
1
answer
286
views
Publisher-Subscriber architecture with central registry
Problem description
Publisher-Subscriber architecture with a central registry where agents can either promote their capabilities or search for a given capability.
The project must be developed with ...
- 139
-1
votes
1
answer
270
views
pub/sub events to/from microservices in different languages
I'm starting a brand new project implementing microservices with domain driven design. We will have microservices written in different languages like C#, Python, and Node. I'm thinking about hosting ...
- 185
1
vote
0
answers
458
views
Flowchart symbol or pattern for Publisher/Subscriber or Listen/Notify
We have a system consisting of
Detectors
Converters/Transformers
Business Rules Engines and State Managers
Views/Displays/Applications
There are several potential ways these may interact and share ...
2
votes
0
answers
110
views
Data change event publishing by microservice vs separate application - best practice
Suppose there is a microservice that has an restful HTTP API for CRUD operations on a database - nothing fancy, but there is a journal table of all changes recorded for audit purposes.
Suppose also ...
- 2,036
1
vote
0
answers
238
views
Event-based communication between microservices
I'm currently looking into NestJS to use as the framework for a new e-commerce project. The goal is to build multiple loosely-coupled services, which can communicate with each other using the pub/sub ...
- 111
0
votes
0
answers
124
views
Preferrable way to send push messeges?
I am making a chat-app using the XMPP protocol and ejabberd as XMPP server. I want to build a push notification functionality for the app. The app is similar to WhatsApp. I came to know three ...
2
votes
2
answers
1k
views
Pros and cons of consuming events in batches
Suppose I have a message broker that allows consumers to specify a batch size. What are the advantages and disadvantages of consuming events in batches compared to consume them one by one?
The ...
- 249
4
votes
1
answer
374
views
Pub-Sub pattern and instance replication
I am porting an event-driven architecture to a container environment (Docker) and I wonder how to take advantage of replication while using pub-sub: if I just increment replicas I end up with many ...
- 695
-1
votes
2
answers
199
views
Using Apache Kafka for rare events [closed]
There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
- 111
3
votes
1
answer
136
views
Run a job every hour with Lambda or something else?
I am working on a SaaS project that will have a trial when the trial is ending I get a webhook notification when 3 days are remaining. I do some stuff with this and one of the things is I update user....
- 161
1
vote
1
answer
968
views
How can an event subscriber catch up with missed events?
At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time.
For example, a service is running and subscribes to events on ...
- 405
1
vote
1
answer
77
views
What is a good strategy for managing PubSub subscriptions
I am working with Reddis but this could apply to any PubSub implementation. Essentially we are trying to implement an event system using Reddis's PubSub capabilities.
As an example UserA creates a ...
0
votes
1
answer
60
views
What types of Google Cloud application should I consider for something running as a continually available service that subscribes to a pub/sub?
I want to create an application that will run as a service that subscribes to a Google Cloud pub/sub topic, and delivers emails based on 'alerts' that it receives through the pub/sub subscription. It ...
- 401
0
votes
1
answer
231
views
Which message pattern should I use?
I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
- 11
1
vote
1
answer
176
views
Event sourcing access by reference
In an event sourcing architecture, what is the typical pattern for passing information about related objects (aggregates)?
For example, in a order processing system, should OrderCreated event (...
- 149
0
votes
3
answers
5k
views
Maintaining order of events with multiple consumers in a typical pub sub scenario
I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
- 153
1
vote
1
answer
205
views
Handling application-wide notifications from various layers of the application?
I have recently started updating portions of our corporate platform, and I've been trying to find ways to make the entire platform more efficient, better structured, and just cleaner in general. With ...
- 1,377
1
vote
1
answer
3k
views
Processing only once the same message produced by two producers
If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed?
Is the only way to have an ...
- 43
2
votes
2
answers
190
views
Name of this pattern
I create an object that holds the status of a job. The job has many stages, some concurrent. Worker threads constantly monitor the status object to see if it's ready for the next stage to begin, if it ...
- 198
0
votes
1
answer
2k
views
How to load test Publisher/Subscribing queue?
I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second).
Problem statement :
The pub sub ...
- 103
2
votes
1
answer
436
views
Architecture pattern for temporary message queue subscriptions with websocket clients
We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
- 29
0
votes
1
answer
481
views
Consuming messages directly from a queue which belongs to a third party system
We are working on a project which interacts with multiple third party systems and they need each other's data to function. Now, for some data which needs to be in sync in multiple systems, we are ...
55
votes
4
answers
35k
views
Ways to share DTO across microservices?
I am designing a system to receive data from various types of sensors, and convert and then persist it to be used by various front-end and analytics services later.
I'm trying to design every service ...
1
vote
1
answer
332
views
How do I set up short-lived queues?
This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
- 1,100
4
votes
1
answer
10k
views
What is a good diagrammatic way to represent async event communication between two systems?
What is a good diagrammatic way to represent asynchronous event communication between two systems (the pub sub model)? I am not looking for tools but more of a representation. Is it a sequence diagram,...
1
vote
1
answer
653
views
Publish and Subscribe (data transfer) with permanently offline nodes. Are message queues a good fit?
The general question is what kind of mechanism can I use to transfer data to and from publishers and subscribers where publishers or subscribers can be permanently offline? Can message queues be used ...
- 141
2
votes
3
answers
207
views
Is this an inefficient use of the Subscriber (PubSub) pattern?
I have a Publisher object that raises an event at regular intervals (think of it something like a clock tick). I then have 100s (or 1000s) of subscriber objects (all instances of a single type or two) ...
- 3,342
5
votes
1
answer
3k
views
pubsub with multiple instances of each consumer
I'm currently looking into setting up a publish subscribe messaging infra structure for our microservice based platform. The new setup is meant to replace our current kafka based one, with something ...
- 161
2
votes
1
answer
471
views
Should API gateway wait until all micro services are done?
for our project, we are using micro services with API gateway pattern. One of the features that the client do is to generate a report. Client, fills a form and uploads a file and our system will ...
- 123
9
votes
2
answers
7k
views
Relationship between RESTful URIs and PubSub topics
I am designing a system which fits quite nicely into a RESTful architecture. Users can navigate a resource hierarchy from a root node, each resource links to other resources, resources have URIs etc. ...
- 281