In our applications we traditionally log events locally into the logging files. As our applications are distributed on multiple server instances, searching for particular events are complicated and time consuming. So we decide to introduce GrayLog as centralized logging solution.
All logging events from an applications are now sent to the GrayLog by UDP protocol, which works fine if the network and the GrayLog server are working fine. If something is wrong, log events sent to the GrayLog server are lost and normally you don't even notice. It is still possible to grep logs on servers but on GrayLog you are never sure that you can find all log events.
My question is: How to ensure that every log event will be delivered to the GrayLog server?