Ok, so I have a program that basically logs errors into a nosql database. Right now there is just a single model for an error and its stored as a document in the nosql database.

Basically I want to summarize across different errors and produce a summary of the "types" of errors that occured.

Traditionally in a SQL database the this normalization would work with groupings, sums and averages but in a NoSQL database I assume I need to use mapreduce.

My current model seems unfit for the task, how should I change the way I store "models" in order to make statistical analysis easy? Would a NoSQL database even be the right tool for this type of problem?

I'm storing things in Google AppEngine's BigTable, so there are some limitations to think of as well.

  • Just curious, but shouldn't this type of question be asked on StackOverflow.com? – funkymushroom Dec 23 '10 at 17:49
  • To be honest, I'm not sure, its more of a design question. Since as far as I've seen the problem is specific for Document style databases. the storage is modeled differently. Basiclly its something that would be dicussed around a whiteboard, however, you might be right. – Morten Dec 27 '10 at 9:48

In short, AppEngine's Datastore may not be the right tool for this job, but I'm going to try to model this anyway.

I would add a few more Models to what you already have. I assume you already have one like:

class LoggedError(db.Model):
    full_stacktrace = db.StringProperty() 
    error_type_name = db.StringProperty()
    code_line = db.IntegerProperty()

I would then make another model just for storing the occurrences of the different types of errors. There will be one instance of this for each type of error

class ErrorOccurrences(db.Model):
    error_type_name = ...
    error_type_count = db.IntegerProperty()
    list_of_occurrences = db.ListProperty(int) # the line numbers the error happens on

The three choices I see for populating ErrorOccurrences would be

  1. Add to ErrorOccurrences.list_of_occurrences as you log the errors
  2. Setup a task or cron job to manually scan LoggedError.all()
  3. Run MapReduce once you've finished adding your errors.

If you are continuously logging errors, #1 will be your best option. If you take that approach, having the counter attached to the ErrorOccurrences may not be the best method. You could make a separate counter model using the Sharded Counter method to more efficiently create your summary, if you don't have to provide the line numbers right away. That would also mean you are making at least 3 writes to the datastore for every error vs. just one.

I hope this answer made sense, and good luck.

  • Great answer! It's more or less exactly the kind of answer I was looking for. It's good that you pointed out the amount of writes, since I hadn't thought in those terms at all. – Morten Jan 5 '11 at 7:37

Try to look at Hadoop wikipedia

Dunno if it works with you DB.

Maybe this post will be helpful:Mail archive

  • 1
    No Hadoop wouldnt run on AppEngine. – Morten Dec 19 '10 at 18:06
  • 1
    Maybe my edit will be helpful – Alexey Anufriyev Dec 19 '10 at 21:29

Why not simply add a classification column to the table? Something like "errorType:ReallyBadError".

Then you 'map' to the errorType, and 'reduce' with, for example, a 'Sum' function.

  • An error is aggregated over several columns, an example would be that the stacktrace shows that error occur at different locations in the code. So there is no way to precategories the errors. but remember that this is not a relational database. It stores complete documents, and its an (quite complex) aggregate over those documents that I am after. – Morten Dec 23 '10 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.