answer of the first question is, as you said, it will either timeout or get some error messages from server. it depends on how you design your program.
I suppose some of nodes in the cluster are working as api servers as it's a reasonable configuration. If you don't have an api server in the cluster, I believe you must have to set it up in somewhere else. otherwise, I don't think you are able to pipeline your requests.
once you have machines that are able to work for your pipeline task, I think there are two ways to complete your task.
the first one:
create another independent db server (let's call the db server you are using A and this isolated server B) that stores the info of requests, and write a function (let's call the function C) that keeps querying B in your api server (we call this action as 'polling'). of course, you can set query interval in C to improve the performance. the length of interval should depend on your situation.
Once the api node gets the request from B, it will query A to get relative info and do relative task. If the task is completed as expectation, C will ask B to delete the request. On the other hand, if A is not available, C will do nothing. Therefore, this request will not be deleted from B, and C will get the same request again when it queries B next time.
the second one:
set up a message queue server. all the requests from client-side will be sent to this mq server.
the duty of this mq server is keeping asking api server to work. if api works without any problem, then mq server should pop the request. otherwise, mq server should keep the request and ask api server to work again after an interval.
RabbitMQ with STOMP may be a good choice.
it's obvious that the first way is inefficient. you have to write, read and delete every request. all of these actions is time-consuming and may be the bottleneck if your service has to handle a lot of requests. However, if you don't want to use new framework, this may be a suitable option. Actually, I am currently working on a cloud computing system, and this is the way we use.
the second way is much more efficient, but you have to install new framework, which may be a maintaining issue in the future.
