I using Spark Streaming to roll up the analytic events and push to Cassandra 2.1+ database where data model resembles:

    metric_1 text,
    metric_1_value counter

The issue that concerns me is that pre Cassandra 2.1 there were some issues surrounding the accuracy of the counters

Losing some data because of a corrupt sstable or replaying the commit log can lead to invalid counter shards.

Now for Cassandra 2.1 + the datastax team state

Apache Cassandra 2.1 will have a safer, simpler, and often faster distributed counters implementation.

However it is fairly important for my use case to have accurate metrics. Perhaps +/- 1 for every 5000-10000 increments. Therefore I am left with two design choices,

  1. Perform the addition of metric counts using Cassandra database, and assume it will be mostly accurate
  2. First read the rolled up metrics using apache spark and perform addition using Apache Spark instead of Cassandra.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.