How should calculations be handled in a document database

Question

Ok, so I have a program that basically logs errors into a nosql database. Right now there is just a single model for an error and its stored as a document in the nosql database.

Basically I want to summarize across different errors and produce a summary of the "types" of errors that occured.

Traditionally in a SQL database the this normalization would work with groupings, sums and averages but in a NoSQL database I assume I need to use mapreduce.

My current model seems unfit for the task, how should I change the way I store "models" in order to make statistical analysis easy? Would a NoSQL database even be the right tool for this type of problem?

I'm storing things in Google AppEngine's BigTable, so there are some limitations to think of as well.

Answer 1

In short, AppEngine's Datastore may not be the right tool for this job, but I'm going to try to model this anyway.

I would add a few more Models to what you already have. I assume you already have one like:

class LoggedError(db.Model):
    full_stacktrace = db.StringProperty() 
    error_type_name = db.StringProperty()
    code_line = db.IntegerProperty()

I would then make another model just for storing the occurrences of the different types of errors. There will be one instance of this for each type of error

class ErrorOccurrences(db.Model):
    error_type_name = ...
    error_type_count = db.IntegerProperty()
    list_of_occurrences = db.ListProperty(int) # the line numbers the error happens on

The three choices I see for populating ErrorOccurrences would be

1. Add to ErrorOccurrences.list_of_occurrences as you log the errors
2. Setup a task or cron job to manually scan LoggedError.all()
3. Run MapReduce once you've finished adding your errors.

If you are continuously logging errors, #1 will be your best option. If you take that approach, having the counter attached to the ErrorOccurrences may not be the best method. You could make a separate counter model using the Sharded Counter method to more efficiently create your summary, if you don't have to provide the line numbers right away. That would also mean you are making at least 3 writes to the datastore for every error vs. just one.

I hope this answer made sense, and good luck.

Answer 2

Try to look at Hadoop wikipedia

Dunno if it works with you DB.

Maybe this post will be helpful:Mail archive

Answer 3

Why not simply add a classification column to the table? Something like "errorType:ReallyBadError".

Then you 'map' to the errorType, and 'reduce' with, for example, a 'Sum' function.