We’re rewarding the question askers & reputations are being recalculated! Read more.
28

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely that not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database-based queue system for a middleware solution. I agree ...


25

Imagine that you have a web service that can accept many requests per second. You also have a accounting system that does a lot of things, one of which is processing the requests coming from the web service. If you put a queue between the web service and the accounting system, you will be able to: have less coupling between the two applications, because ...


22

I've run into some difficulties thus far I'd like to document here. How do you handle reconnect logic? This is a hard problem and an especially hard problem in designing and implementing a message queue. Messages must be able to queue up somewhere when consumers are offline, so a simple pub-sub is not strong enough, and consumers need to reconnect in a ...


17

Everything I have read on the internet suggests that using a database for a Message Queue isn't a scalable solution. The reality frequently ignored by the "don't use X because it doesn't scale" (link contains language some may find objectionable) crowd is that scale isn't always important. I'd go so far as to say that if you look at every application on ...


16

In addition to providing a buffer between a web service and another backend service, message queues can be used for more advanced scenarios. Rabbit MQ (and other mature Message Queue products also referred to as MOM -- Message Oriented Middleware) can be configured to route and distribute messages according to different rules. For example the Pub-Sub ...


16

Switching to a solution involving databases or the queuing systems mentioned by Ewan would create dependency on a new, complex system in both backend and frontend introduce unnecessary complexity and a sh*tload of new points of failure increase cost (including cost of ownership) Moving/renaming files within a single volume is guaranteed to be atomic on all ...


14

In short: This is a hard problem. Don't reinvent the wheel. There are many technologies that solve the message queue layer. They include ZeroMQ RabbitMQ Apache Kafka Redis, with BLPOP or PUBSUB (I've asked how to do this here). Other AMQP implementations besides RabbitMQ I think it's out of scope for me to discuss the drawbacks of each, not the least ...


12

Actor Model The actor model is computer science strategy for building applications that handle lots of concurrent computation and stateful processing. It's not the only strategy but it's a very well tested, simple, and reliable approach that moves computation into actors, which communicate through messages that they process one-at-a-time and in order. Akka ...


10

I don't think either solution is inheritly a bad practice, so answering which is the best practice may be difficult. I don't believe the YAGNI principal applies here if you're dealing with scale. "Working" is relative, if you have a strong potential for catastrophic data loss, and little ability to scale, I wouldn't really consider that working. I'm not ...


9

Message Queues really come into their own when you have many of them and route messages between them, fanout to more than one consumer etc. If you just have a single 'job queue' of stuff you want to process 'off line' then an SQL table will do just fine. Don't forget to ensure you have some way of marking jobs in-progress, cleaning out old ones and ...


8

As the queue is an important component in the sequence you are presenting, it should most definitely be present with a lifeline. As the consumer explicitly listens for events from the queue, I would start the diagram with the listen call from the consumer to the queue. After that, the producer can insert its event into the queue (possibly with an indication ...


8

My suggestion is that instead of worrying about whether to use an external or internal cache, your first concern should be that your booking-service does not care whether or not your are using an external service. That is to say, your booking-service should be caching against an interface with the concrete implementation injected in; it would not know or ...


7

I've had to use WF before in an integration scenario based on a Pub-Sub messaging system... and we found that using WF was a pain in the ass. Firstly, it's awkward to code with. At first you'll be tempted to code all your logic with activities in the Designer, but this is a very bad thing. Even simple things like object instantiation turns into either a ...


7

Implementing Bart's answer in the wonderful PlantUML could look like this, (ab)using an entity to distinguish the queue from the participating threads: The corresponding source code: participant Producer entity Queue Consumer->Queue : dequeue activate Consumer ... Producer->Queue : enqueue activate Producer Queue-->Consumer deactivate Consumer ...


7

Don't view the result as the outcome of a request, but as the followup of an event. The question doesn't specify much so I can only speak in abstract terms. I believe the following to be the best approach. Task, Input from user: If input from queue exists, do the task: create result Else, preserve the input to be used in "input from queue" Task, Input ...


5

Since you are a startup I would say it depends on two things. Do you have any time constraints on this? Will building your own messaging system give you an advantage? I don't know what kind of business your startup is in but in general its best to focus on great user experience in the beginning. The code running in the back may be awful but the user will ...


5

You may be interested in Comet, a programming model that allows the server to push data to a client. There are implementations for JSF, look at this blog post about Richfaces integration for an example. HTML 5 specifies WebSockets that can be used for server push, but not all browsers support it. The simple alternative is to simply poll the server at ...


5

If you want to use Redis for a message queue in Node.js and you don't mind using a module for that then you may try RSMQ - the Redis Simple Message Queue for Node. It was not available at the time this question was asked but today it is a viable option. If you want to actually implement the queue yourself as you stated in your question then you may want to ...


5

It looks like Autobahn fits nicely with what you're trying to do. There are other tools available as well. Check out the Windows Azure Service Bus (which has client frameworks for Java, .NET, PHP, Python, NodeJS, and Ruby). While the built in rest messages are useful. You'll find that your application will outgrow basic CRUD operations. For instance if your ...


5

If you have the connectivity, go with a message queue - although you have to define your own protocols (hardly a difficult task!) to send messages of a particular structure and format. The problem with maintenance is that typically the client and server are built separately so you need to be careful to keep both ends using the same message definitions, but ...


5

Communications do not usually only go one way. Parties usually communicate using request-response pairs, which are clearly not one way. I presume that you already know this, so what you are probably thinking when you say "one-way" is not how information flows, but who initiates the requests. So, yes, the way we usually do things is that only one of the ...


5

Systems such as RabbitMQ (or in your case maybe look into Kafka) can offer persistence, or guaranteed delivery, as well as configurable TTL on messages. However, they are not designed as a long-term persistent storage solution, and if we're talking about days, I would actually store the job in a DB as you suggest.


5

While the good 'ol save a file and copy it to a done dir is a staple of many communication layers esp. with older main frame systems and the like. The 'anti' guys do have a point; in that it has many problems and edge cases. Which are hard to deal with if you need 100% reliablitiy, and occur more often as you scale up the frequency and volume of files. If ...


5

As per the wikipedia page you linked to regarding message queues: Message queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time. Messages placed onto the queue are stored until the recipient retrieves them. (Bolded emphasis mine). ...


5

Kafka/Kinesis is modelled as a stream. A stream has different properties than messages. Streams have context to them. They have order. You can apply window functions on streams. Although each item in a stream is meaningful, it may be more meaningful with the context around it Because streams have order, you can use that to make certain statements about the ...


5

Kafka deals in ordered logs of atomic messages. You can view it sort of like the pub/sub mode of message brokers, but with strict ordering and the ability to replay or seek around the stream of messages at any point in the past that's still being retained on disk (which could be forever). Kafka's flavor of streaming stands opposed to remote procedure call ...


5

One way to deal with this is to use pull-based listeners instead of push-based. Each listener keeps track of its last read message and can request "messages since X". You could use either polling or a notification when a new event comes in or both to trigger message requests. No more queues, but the listener will need to store its last processed message id. ...


4

We faced a similar problem, and used RabbitMQ for message transport, and formatted the messages using Google Protocol Buffers. RabbitMQ is excellent, it just works. Protocol buffers format the data compactly (far more so than XML or JSon) and is a known schema allowing interoperability between different languages and machine architectures. On the Java ...


Only top voted, non community-wiki answers of a minimum length are eligible