2

This really sounds like something you could easily put in a AWS Lambda. We do something similar at my company using Azure Functions. You could even chain them together. One lambda determines who to send to put them into a queue. Then another calculates and processes the data and puts it into another queue and then the last formats and sends the emails.


2

We try to make sure all changes to a message contract are both forward and backward compatible. In those rare cases where there is truly a breaking change in the message contract, we consider that a different message (a new kind of message). We use the Parallel Change pattern to manage such breaking changes. In cases where we have control over the ...


1

First choice is better for security and independence between tenants, like the choice about database.But you have to build tools to handle it. Failure of one queue, don't expose failure to other tenants. First is not simple technically, but have functionals advantages : limited failure independence between tenants/confidentiality, eg: data cannot be ...


1

can a queue have multiple consumer ? Yes it's possible that multiple consumers take messages from the same queue. But every message is only consumed once by one consumer is it possible for the queue to retain data until after a consumer has consumed it No the messages are not visible for the other consumers if one consumer started consuming it. I ...


1

At work I have worked in a similar requirement that you have proposed here. I'd like to answer your questions and give some more hints here. First of all for starters please study this link in order for apprehending the Exchange Types available in Rabbit: https://www.cloudamqp.com/blog/2015-09-03-part4-rabbitmq-for-beginners-exchanges-routing-keys-bindings....


1

Your receiver should be idempotent and deal with duplication. Look at Idempotent Receiver Pattern. As you mentioned Apache Camel implement some features to deal with duplication.


1

The idea of a message bus is to run things asynchronously. Message exchange is not supposed to take place while a client is waiting for your response. It should happen in the background, so for example if your ProductService needs some data about users, it should listen to events from UserService about user changes, and apply them to its own local copy of ...


1

So what is the use of persisting the events as a sequence when we are unsure about their order? Well, we are sure about the order of events -- they are sequenced in the order that they are written, which is determined by the order in which the commands finish processing. (If you are using a single command handler, that is equivalent to the order in which ...


1

Depends on the duration of these long running tasks (validations) and desired client (producer) interaction. If the tasks are not too time consuming and clients would prefer a result of the upload in realtime. You can implement this using Servlet 4.0 (Async servlet) service implementation. This would eliminate the need for MQ message broker and service can ...


1

The main thing you should be designing for here is the order you want tasks to be processed: If all tasks should be processed strictly in the order they are queued, you want one queue with all messages on it. The language would then be flagged in the message contents. However, to scale effectively you will still want multiple consumers of this queue, and ...


1

You're right in general about throughput that you would get with a low-level message broker solution vs an iPaas - however you should consider whether you really need that throughput. Some iPaas solutions are quite fast and may be 'fast enough' for your use case. We deployed an integration platform successfully to connect real-time systems (IoT/control ...


1

I'm not sure I agree with the premise of the question - and here's why. Mathematical Theory According to Queuing Theory, in any First-In-First-Out stochastic process, you have an arrival rate for requests and an average time the process takes to service requests. The process then operates according to the rules of a Poisson Process, and we can tell ...


1

I think the pattern you outline is the common one. (In that you program your own 'routing worker') But you could condense it down, moving the routing logic (red) into the worker. For example, say instead of a worker listening to a single queue, I add code that is aware of the users. I can then fire up a thread per user queue in the same worker service and ...


1

The proper term for the issue you are addressing is 'poison message processing'. Dead letter queues are typically used by messaging systems for undeliverable messages. A poison message is one that a client application cannot successfully process. A 'poison message queue' is a queue that is configured to receive these messages in order to prevent a message ...


1

I understand your concerns about this, but I would say that yes, you do need to extract the dead messages from the queuing system. My sole reason for saying this is this sentance: maybe even reject some valid messages until we talk with the support team A dead letter queue is great for exceptions or handling a break down of a component, but the ...


1

How are your workers picking up work? Are they polling redis or do they subscribe to a channel? You're right that this isn't something Redis is well suited for. The fact that it's durability guarantees aren't that great should also give you something to worry about if at least once delivery it's something you care about. For reference, large scale message ...


1

Distributed Queue-based solutions have a number of advantages for systems that are asynchronous especially when they are throughput focused. I would not recommend using distributed queues as part of synchronous transactions. Pros Volume can be leveled out. Processing will continue at it's maximum rate and overflow will be managed on queues. If designed ...


1

My team did a meeting to discuss this question. We understand that the Binding and Queue on Consumer is the better place. My question already presents the main advantages, so I will not copy in the answer here. In the beginning of the meeting some team members "feel" that the right place was in the Producer when doing RPC calls. I understand that this ...


1

Option A sounds like the way to go as you can set workers it the API independently instead of a huge tasks that only a single worker can manage. I have a very similar scenario using kombu and Celery: We get a message posted to RMQ by some integration to a RMQ queue We have a kombu consumer draining events When event is received we execute the callback (...


Only top voted, non community-wiki answers of a minimum length are eligible