Questions tagged [pubsub]

Publish/subscribe is a method for wiring software components together. One component publishes data, and subscribers receive events containing the data. This decouples the components so they do not need to know about each other; and it allows for configuration to wire components together without recompiling code.

Filter by
Sorted by
Tagged with
2
votes
2answers
61 views

Pros and cons of consuming events in batches

Suppose I have a message broker that allows consumers to specify a batch size. What are the advantages and disadvantages of consuming events in batches compared to consume them one by one? The ...
0
votes
0answers
18 views

Pub-Sub pattern and instance replication

I am porting an event-driven architecture to a container environment (Docker) and I wonder how to take advantage of replication while using pub-sub: if I just increment replicas I end up with many ...
-1
votes
2answers
63 views

Using Apache Kafka for rare events [closed]

There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
0
votes
0answers
19 views

Are message identifiers and offsets the same thing in Kafka?

I am reading up on Kafka and encountered the following on one of the tutorials. The producer sends the following to the broker - unique message identifier, timestamp and a binary payload. The ...
2
votes
0answers
51 views

Run a job every hour with Lambda or something else?

I am working on a SaaS project that will have a trial when the trial is ending I get a webhook notification when 3 days are remaining. I do some stuff with this and one of the things is I update user....
0
votes
1answer
62 views

How can an event subscriber catch up with missed events?

At a conceptual level, what ways can a subscriber catch up with events it has potentially missed but needs to know about at a later time. For example, a service is running and subscribes to events on ...
1
vote
1answer
55 views

What is a good strategy for managing PubSub subscriptions

I am working with Reddis but this could apply to any PubSub implementation. Essentially we are trying to implement an event system using Reddis's PubSub capabilities. As an example UserA creates a ...
0
votes
1answer
33 views

What types of Google Cloud application should I consider for something running as a continually available service that subscribes to a pub/sub?

I want to create an application that will run as a service that subscribes to a Google Cloud pub/sub topic, and delivers emails based on 'alerts' that it receives through the pub/sub subscription. It ...
0
votes
1answer
134 views

Which message pattern should I use?

I'd like to develop a microservice architecture application that sends text messages from one communication provider to another and I'm just considering changing architecture would be best suited to ...
0
votes
1answer
51 views

Event sourcing access by reference

In an event sourcing architecture, what is the typical pattern for passing information about related objects (aggregates)? For example, in a order processing system, should OrderCreated event (...
0
votes
2answers
734 views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
1
vote
1answer
117 views

Handling application-wide notifications from various layers of the application?

I have recently started updating portions of our corporate platform, and I've been trying to find ways to make the entire platform more efficient, better structured, and just cleaner in general. With ...
0
votes
1answer
1k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
2
votes
2answers
173 views

Name of this pattern

I create an object that holds the status of a job. The job has many stages, some concurrent. Worker threads constantly monitor the status object to see if it's ready for the next stage to begin, if it ...
0
votes
1answer
762 views

How to load test Publisher/Subscribing queue?

I am trying to load test a queue that works on a pub sub pattern and can't figure out how response time is to be calculate and the TPS(transactions per second). Problem statement : The pub sub ...
2
votes
1answer
196 views

Architecture pattern for temporary message queue subscriptions with websocket clients

We use Google Pub/Sub for event streams which we want consumed by transient websocket clients. What is a good pattern for creating subscriptions and cleaning them up when the client is no longer ...
0
votes
1answer
202 views

Consuming messages directly from a queue which belongs to a third party system

We are working on a project which interacts with multiple third party systems and they need each other's data to function. Now, for some data which needs to be in sync in multiple systems, we are ...
28
votes
3answers
8k views

Ways to share DTO across microservices?

My scenario is as follows. I am designing a system designed to receive data from various types of sensors, and convert and then persist it to be used by various front-end and analytics services later....
1
vote
1answer
150 views

How do I set up short-lived queues?

This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-...
1
vote
1answer
1k views

What is a good diagrammatic way to represent async event communication between two systems?

What is a good diagrammatic way to represent asynchronous event communication between two systems (the pub sub model)? I am not looking for tools but more of a representation. Is it a sequence diagram,...
1
vote
1answer
249 views

Publish and Subscribe (data transfer) with permanently offline nodes. Are message queues a good fit?

The general question is what kind of mechanism can I use to transfer data to and from publishers and subscribers where publishers or subscribers can be permanently offline? Can message queues be used ...
1
vote
3answers
144 views

Is this an inefficient use of the Subscriber (PubSub) pattern?

I have a Publisher object that raises an event at regular intervals (think of it something like a clock tick). I then have 100s (or 1000s) of subscriber objects (all instances of a single type or two) ...
4
votes
1answer
918 views

pubsub with multiple instances of each consumer

I'm currently looking into setting up a publish subscribe messaging infra structure for our microservice based platform. The new setup is meant to replace our current kafka based one, with something ...
1
vote
1answer
407 views

Should API gateway wait until all micro services are done?

for our project, we are using micro services with API gateway pattern. One of the features that the client do is to generate a report. Client, fills a form and uploads a file and our system will ...
6
votes
2answers
5k views

Relationship between RESTful URIs and PubSub topics

I am designing a system which fits quite nicely into a RESTful architecture. Users can navigate a resource hierarchy from a root node, each resource links to other resources, resources have URIs etc. ...
2
votes
1answer
481 views

Is publish–subscribe pattern suitable for my case?

I'm planning to use publish–subscribe pattern with Azure Service Bus in my logistics application, but I'm confused whether publish–subscribe pattern is suitable for my case. My scenarios: Users will ...
1
vote
0answers
45 views

Securing an Azure Service Bus topic and subscriptions

We are working on a solution that will publish messages on an Azure Service Bus topic but we don't know yet which other team or even third party might be interested in subscribing to those messages in ...
1
vote
2answers
923 views

Publish / Subscribe via HTTP Callbacks?

My team is tasked with creating a publish/subscribe system for incoming REST messages. 99% of the time, this system will be used for notifications between different processes on the same cpu, but we ...
1
vote
0answers
160 views

MVC WebApplication as MQ Subscriber

We have a Web (.Net MVC5) application that we are looking to start offloading some of the heavy processing off the Web Application to a distributed services layer. As part of this process we are ...
0
votes
0answers
142 views

Is NSNotificationCenter an implementation of a PubSub model?

I am trying to understand the subtleties of Publisher-subscriber and similar notification mechanisms. I was checking Cocoa NSNotificationCenter, but it's unclear to me if it can be defined as a PubSub,...
1
vote
0answers
235 views

Lazy loading collection data through granular subscriptions and OOP (Meteor.js)

I am transforming my Meteor collection to a class I've created: People = new Mongo.Collection('people', { transform: function(doc) { return new Person(doc); } }); Person has a bunch of ...
4
votes
3answers
862 views

Are imperative events an anti-pattern?

Are events which are intended to tell listeners to carry out explicit actions an anti-pattern? Imagine a worker service which raises the event WorkStarted at its beginning and the event WorkFinised ...
0
votes
1answer
255 views

Strategy vs Publiser-subscriber pattern

Let's say I have a channel endpoint where I receive messages. Because I want to process messages in a different way depending on its Header property, rather than create a massive switch I create ...
0
votes
1answer
780 views

Design pattern for modifying state and notifying the other modifiers without a loop

The idea is that I have a set of properties that represent possible options in the app. These options can be modified through a variety of different ways, such as from the UI, remotely, third party, ...
2
votes
1answer
224 views

Good pattern(s) for pointing to mutable state [closed]

I asked this question once here, but believe the question was unclear. However I'm having a hard time extracting the general problem from my specific case. UPDATE: I've answered my own question ...
2
votes
0answers
206 views

Looking for design pattern to represent shared state in music sequencer

I like to give lots of context, but also to state the basic question up top as concisely as possible. Feel free to read a little or a lot. I'm working on a music sequencer, and have been happy so far ...
11
votes
2answers
2k views

How is the publish-subscribe pattern different from gotos?

My understanding is that Goto statements are generally frowned upon. But the publish-subscribe pattern seems to be conceptually similar in that when a piece of code publishes a message, it performs a ...
7
votes
1answer
6k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
8
votes
1answer
2k views

Diagramming messages on a service bus

I'm looking for a way to clearly diagram how multiple applications communicate via a service bus. The best I've come up with so far is a sequence diagram, but I really don't like that. Sequence ...
29
votes
3answers
23k views

How to implement a message queue over Redis?

Why Redis for queuing? I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or ...
0
votes
2answers
1k views

Is this RabbitMQ / sockjs pub/sub connection setup a comet solution?

My company is attempting to set up a pub/sub connection using RabbitMQ and sockjs, on Windows servers. I have two questions: My supervisor is calling this setup a comet solution, I was wondering if ...
5
votes
4answers
2k views

Design Patterns for Coordinating Change Event Listeners

I've been working with the Observer pattern in JavaScript using various popular libraries for a number of years (YUI & jQuery). It's often that I need to observe a set of property value changes (e....
6
votes
2answers
2k views

Simple, permanent queue system with pub/sub for Node.js?

Highly related https://stackoverflow.com/questions/6021938/nodejs-many-clients-requests-through-one-socket I'm investigating a robust message passing system for this application of mine. The easyest ...
3
votes
3answers
403 views

What scenarios are implementations of Object Management Group (OMG) Data Distribution Service best suited for?

I've always been a big fan of asynchronous messaging and pub/sub implementations, but coming from a Java background, I'm most familiar with using JMS based messaging systems, such as JBoss MQ, HornetQ,...