Ok, so I have a program that basically logs errors into a nosql database. Right now there is just a single model for an error and its stored as a document in the nosql database.

Basically I want to summarize across different errors and produce a summary of the "types" of errors that occured.

Traditionally in a SQL database the this normalization would work with groupings, sums and averages but in a NoSQL database I assume I need to use mapreduce.

My current model seems unfit for the task, how should I change the way I store "models" in order to make statistical analysis easy? Would a NoSQL database even be the right tool for this type of problem?

I'm storing things in Google AppEngine's BigTable, so there are some limitations to think of as well.

  • Just curious, but shouldn't this type of question be asked on StackOverflow.com? Dec 23, 2010 at 17:49
  • To be honest, I'm not sure, its more of a design question. Since as far as I've seen the problem is specific for Document style databases. the storage is modeled differently. Basiclly its something that would be dicussed around a whiteboard, however, you might be right.
    – Morten
    Dec 27, 2010 at 9:48

3 Answers 3


In short, AppEngine's Datastore may not be the right tool for this job, but I'm going to try to model this anyway.

I would add a few more Models to what you already have. I assume you already have one like:

class LoggedError(db.Model):
    full_stacktrace = db.StringProperty() 
    error_type_name = db.StringProperty()
    code_line = db.IntegerProperty()

I would then make another model just for storing the occurrences of the different types of errors. There will be one instance of this for each type of error

class ErrorOccurrences(db.Model):
    error_type_name = ...
    error_type_count = db.IntegerProperty()
    list_of_occurrences = db.ListProperty(int) # the line numbers the error happens on

The three choices I see for populating ErrorOccurrences would be

  1. Add to ErrorOccurrences.list_of_occurrences as you log the errors
  2. Setup a task or cron job to manually scan LoggedError.all()
  3. Run MapReduce once you've finished adding your errors.

If you are continuously logging errors, #1 will be your best option. If you take that approach, having the counter attached to the ErrorOccurrences may not be the best method. You could make a separate counter model using the Sharded Counter method to more efficiently create your summary, if you don't have to provide the line numbers right away. That would also mean you are making at least 3 writes to the datastore for every error vs. just one.

I hope this answer made sense, and good luck.

  • Great answer! It's more or less exactly the kind of answer I was looking for. It's good that you pointed out the amount of writes, since I hadn't thought in those terms at all.
    – Morten
    Jan 5, 2011 at 7:37

Try to look at Hadoop wikipedia

Dunno if it works with you DB.

Maybe this post will be helpful:Mail archive

  • 1
    No Hadoop wouldnt run on AppEngine.
    – Morten
    Dec 19, 2010 at 18:06
  • 1
    Maybe my edit will be helpful Dec 19, 2010 at 21:29

Why not simply add a classification column to the table? Something like "errorType:ReallyBadError".

Then you 'map' to the errorType, and 'reduce' with, for example, a 'Sum' function.

  • An error is aggregated over several columns, an example would be that the stacktrace shows that error occur at different locations in the code. So there is no way to precategories the errors. but remember that this is not a relational database. It stores complete documents, and its an (quite complex) aggregate over those documents that I am after.
    – Morten
    Dec 23, 2010 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.