How can you effectively use web services in an enterprise environment if you can't use transactions?

Question

The place I'm working at is trying to establish some ground rules, and the debate we're having now is local libraries vs web services for code reuse. Web services seem to be the popular pick in most companies, and that's what most of the developers here are leaning toward.

I just can't see how you can effectively use web services for any serious work. How can I safely execute multiple service calls if I can't use a transaction?

Let's say I have a cron job that grabs customers from our database who meet a certain condition that they need to be notified of. They are sent a fax, an email, and a ticket is created to track the issue internally. That is 3 different service calls that would happen for each customer in a for loop.

If an error occurs anywhere in there, it's possible that, for example, a fax and email is sent to the customer, but a ticket is not created. Or worse, this cron job could contain a bug on that causes it to fail at the same point every time, and it repeatedly emails the same customer. If the libraries were all local, everything could just be wrapped in a transaction, and none of that would happen. But we're using web services in this example.

Note that the email and fax methods actually insert the data into database-backed queue tables, which in turn are handled by a separate cron job process. So the calls to the "send email" and "send fax" service methods could be aborted side-effect free if needed.

An option is to put this entire chunk of code in the web service itself, so the web service itself would call the email, fax, and ticket creating methods in a transaction. But then we're creating a web service method just for the use of a transaction; there is no valid reason we would ever actually need to call this method from anywhere except this one cron script.

How would you generally handle this method?

Answer 1

What you are describing is in fact a distributed transaction implementing two-phase commit. Some enterprise messaging platforms include transaction managers to support that kind of thing, but concrete products are platform / language dependent. I don't have concrete experience with such tools, but hope these pointers help.

Answer 2

It's interesting that as I'm participating in this particular Q&A, there is a similar thread about services supporting multiple platforms on the DDD/CQRS mailing list that I'm participating in. I can re-iterate some of my advice here.

One option for supporting transactions in a heterogeneous environment is using a transport mechanism that supports transactions and is supported on all platforms from which it will be used. The Advanced Message Queue Protocol (AMQP) does support transactions and there is a native API for almost every language that is commonly used today. RabbitMQ is a server that implements AMQP and has been vetted in the industry as a robust solution.

Leveraging a RabbitMQ based system puts you on a path to having a full ESB should you need to grow into it. You publish messages to a channel and subscribe to a queue. Where that gets really powerful is that between the channel and queue, you can perform a lot of interesting things. A channel can feed multiple queues (pub/sub) a queue can be fed by multiple channels, you can route messages to different queues based on content, etc. etc.

I was just reading about alternatives to transactions (which come with overhead and turn an async operation into a blocking op). RabbitMQ supports what's called a publisher confirm. Basically it allows you to register a callback for a published method to handle a failed transaction. In your case it could undo the email/fax requests and delete the ticket.

Of course the rabbit hole (pardon the pun) goes even deeper from there. You can use Rabbit to do complex orchestrations with both internal and external web services.

For your public facing webservices, it becomes dead simple. Your service (be it SOAP, REST, or JSON) just publishes a message to the appropriate service queue and let your internal system handle it from there.

There's also functionality for creating a request/response message for those scenarios where you expect information back quickly.

Answer 3

The keywords you are looking for are 'web service choreography'.

Check out Wikipedia's article about it.

Answer 4

The way I handled this in a services app I wrote was to create a wrapper to handle the necessary transactions. In my case, the user request, made by web site, desktop app or Windows service, had to query a web service and, based on the result and the user's options, it had to update a local DB and, optionally, a remote one via a web service. Then it had to generate a report to be returned immediately, emailed and/or faxed. I had control over the local DB, email and report generation but none on the web services or fax server.

Creating a wrapper allowed better transaction control and error handling. It also allowed better security by controlling access to internal network services from outside sources. In general, I see the need for transactions and management of the service as a valid reason to create an appropriate wrapper for a single solution as long as code is reused properly (no cut-n-paste coding).

Answer 5

I just can't see how you can effectively use web services for any serious work. How can I safely execute multiple service calls if I can't use a transaction?

You can't.

The question you should be asking is: how do I implement transactions with web service framework X? Right now, you're just assuming that it's impossible.