Since rest deals with documents I'm not sure how those would translate into operations executed on a domain model, given that documents are anemic and domain models are not.
The main thing to keep in mind is the point of a web api is to adapt your domain for the web; which is to say, to provide the illusion that your resources are really just entries in a document store.
In a sense, a consumer should be able to drive your API using a web aware document editor.
Coarse grained approach
The save action will result in a PUT request (PUT /orders/3) containing the entire order including order lines. The drawback is that it's hard to figure out what has changed (i.e. which order lines should be added / updated / deleted?).
This is a perfectly reasonable approach to take - you tell the API how you want the resource to look, and the API has to figure out how to get there.
Since shopping carts don't typically call for collaboration among multiple editors, you might get by building a new data model for the resource from the snapshot you have been given, then overwriting the existing model.
If you, for whatever reason, instead need to discover finer grain command messages from the diff; yeah, that can be awkward. In a case like this one, it's not too bad -- you can detect change easily enough, and any valid change is likely drawn from a small vocabulary.
It's also worth noting that you don't need to support PUT on any arbitrary representation of the resource, but can instead pick and choose a representation that makes sense (and, hopefully, one where it is easy to identify the correct changes).
Fine grained approach
The delete order line action will result in a DELETE request (DELETE /orders/3/lines/5).
Any URI you want to use is fine as far as REST is concerned; but conceptually it might be better to think in terms of removing the entity by id, rather than by position in the list.
The add order line action will result in a POST request
Could also be done as a PUT.
This implies the app now needs to keep track of a lot of changes, and execute the changes in the correct order when the user decides to save.
Yes; I believe you run into a similar situation with an occasionally connected client. Command messages get queued up until the target system becomes available.
Another approach to consider would be to manipulate the history of the resource, rather than the resource itself. Effectively, the history becomes an append only collection of changes.
I'm just wondering how I can avoid duplicating business rules on the client. For example, if an order line cannot be deleted under certain circumstances, nothing prevents the client from doing so.
There are a couple of answers here, based on the idea that you signal that information in the representations.
One answer is that you simply annotate that the order line should not be deleted. The business rules that decide whether an order line can be deleted live on the server, the client just sees that the flag has been set.
You can communicate that information out of band to the clients.
In the fine grained approach, the usual answer is that you use hypermedia controls. Rather than clients sending a message to a pre-determined endpoint, the client is expected to look in the representation and "follow a link". So you can communicate the fact that an order item can't be deleted by removing the delete item link from the representation.
In a course grained approach, I suppose you could try distinguishing the modifiable parts of the resource from the rest, and providing editable representations of only the parts that the client is allowed to edit. There's liable to be some confusion there.