I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows:
- Visitor does something.
- Custom analytics server gathers data, makes an event out of it and puts it into Elasticsearch.
- One a day or so, custom batch processor aggregates a large set of events from Elasticsearch, transforms them to smaller chunks of data and puts back to Elasticsearch.
I know Logstash is made to gather and process events. Why should I consider it over my own custom solution?
I have no log files in the process. Events travel from one component to another via HTTP. I don't care transforming the data to events on my own either.
I'm especially concerned about not having the flexibility of dealing with the events and Elasticsearch directly from my own choice of programming language.