My service has a large ongoing number of user events, and we would like to do things like "count occurrence of event type T since date D."
We are trying to make two basic decisions:
What to store? Storing every event vs. only storing aggregates
- (Event log style) log every event and count them later, vs.
- (Time-series style) store a single aggregated "count of event E for date D" for every day
Where to store the data
- In a relational database (particularly MySQL)
- In a non-relational (NoSQL) database
- In flat log files (collected centrally over the network via
syslog-ng
)
What is standard practice / where can I read more about comparing the different types of systems?
Additional details:
- The total event stream is large, potentially hundreds of thousands of entries per day
- But our current need is only to count certain types of events within it
- We don't necessarily need real-time access to the raw data or aggregation results
IMHO, "log all events to files, crawl them at a later time to filter and aggregate the stream" is a pretty standard UNIX Way, but my Rails-y compatriots seem to think that nothing is real unless it's in MySQL.
SELECT...GROUP BY
, can easily store the results ofSELECT
s), 2) using Graphite for simple large-scale aggregation and visualization, and 3) logging full events for reference, and for watching details of the data flow in in real time. Each has actually been valuable in different ways.