We are dealing with a third party API that doesn't handle concurrency, and we don't have access to the database directly. Our client application is deployed in a cluster environment and has multiple worker nodes sending update requests to this API. Our goal is to wrap this API in another REST API, so we could add a service layer on top to control the concurrency so that all requests will be synchronized in a pessimistic lock fashion. It will certainly work if we deploy our wrapper API to a non-cluster environment. Will this approach still work if we deploy our wrapper API to a cluster environment as well? My concern is if the pessimistic lock will be shared across all worker nodes?
I would advise against that. The problem is that you need to serialize all requests, not grind your application to a halt.
A better mechanism is to use a Message Queue. Whether you roll your own, or (preferably) use an existing purpose built system, you can have your API push messages on to the Queue, and your wrapper pop messages off to submit.
The problem with locking the last step in an asynchronous by design construct is that you run the very real risk of deadlock. Even if you manage to avoid deadlock, your wrapper API will spend a very long time resolving lock contention.
If you separate the request from when it is processed using a Queue, you avoid the deadlock or lock contention issues that will impact your application. If you rely on the response, it does complicate things a little in that you have to wait until you are notified that the response was received. You can use the message queue for that purpose as well.