Distributed transactions and queues, ruby, erlang

Question

I have a problem that involves several machines, message queues, and transactions. So for example a user clicks on a web page, the click sends a message to another machine which adds a payment to the user's account. There may be many thousands of clicks per second. All aspects of the transaction should be fault tolerant.

I've never had to deal with anything like this before, but a bit of reading suggests this is a well known problem.

So to my questions. Am I correct in assuming that secure way of doing this is with a two phase commit, but the protocol is blocking and so I won't get the required performance? It appears that DBs like redis and message queuing system like Rescue, RabbitMQ etc don't really help me a lot - even if I implement some sort of two phase commit, the data will be lost if redis crashes because it is essentially memory-only.

All of this has led me to look at erlang - but before I wade in and start learning a new language, I would really like to understand better if this is worth the effort. Specifically, am I right in thinking that because of its parallel processing capabilities, erlang is a better choice for implementing a blocking protocol like two phase commit, or am I confused?

Answer 1

Based on the comment, you do not have a two-phase commit. You have two different normal one-phase-commit issues.

A simple (but likely wrong) solution would be to have the web site, when clicked, insert a record in the database. This can be done transactionally during the web processing, so there is a high probability that the user will know their click was taken. But at that point they will not know that the payment was made.

A separate system could poll the db (I said it was likely wrong) looking for entries. If it finds an entry, it can transactionally delete that row, and amend the payment record. This makes a simple restartable system.

If you want it to be a little more real-time, you could use a queue, with transaction semantics at both ends, and a durable (i.e. disk-based) queue in the middle.

The web click would transactionally enqueue something. The queue manager would take care of persisting it to disk if that is appropriate. At the payment end, it can dequeue an item and log the payment. No two-phase commit required (more-or-less) because you are never really dealing with distributed resources. If your dequeue server crashed, you should somehow be able to detect if the latest item in the queue is already in the database, but that should be solvable with some kind of request ID in the message.

Unless of course I have completely misunderstood the question. As I don't understand what two resources you are trying to do the two-phase commit between...

Answer 2

As long as your message queue middleware is restartable and can guarantee Exactly Once processing, you will be fine. Your apps should drop their requests onto the queue, and the middleware tool will take care of persisting that and managing the retry activity. At the "data layer" end, you just need to make sure that your transaction to the DB and your acknowledgement to the middleware occurs correctly.

This can be handled by a proper XA 2 phase commit tool, or you can work around it.

For example, perhaps, save the unique message ID to the DB when you record the payment, and before you write a payment you do a check that the unique message Id has not been seen before.

Answer 3

I've used Redis with message queues for a small 24-node distributed system. It is very nice, high performance, and easy to use, with lots of language bindings. I use it with Python and PHP. It also has a feature that serializes every operation to disk, and in case of a power outage you can replay the log. Maybe you could try prototyping a solution to your problem and see if it does the job. The learning curve is not as steep as Erlang. :-)