How would you go about designing a structured log pattern for systems?

Question

My web app generate log entries on hooks such as model.save. Right now, I have a class Entry (representing a log entry) with a string field content to contain the text of the log entry.

Examples of such entries:

1. Entry(content='Account 123-456-7890 spending changed to $500/day')
2. Entry(content='Account 123-456-7890 paused spending')

The main problem with this approach is that it is difficult to derive semantic value from these Entry objects if content is just a string field.

For example, if I want to find out how often the spending of an account is modified on average across my team, I will have to write some custom logic to find all the entries with the text 'spending changed to' and analyze them. This is error prone because text can contain unpredictable elements such as capitalization or white spaces.

Since most of the log entries are generated programmatically, it occurred to me that I can use an enum to represent the content of the Entry class.

This means the example above can be transformed into the following:

1. Entry(content_id=1, content_args="{ account_id: 123-456-7890, value: '500' }")
2. Entry(content_id=2, content_args="{ account_id: 123-456-7890 }")

And the Content Enum:

class Content(Enum):  
  1: 'Account {{ account_id }} spending changed to ${{ value }}/day' 
  2: 'Account {{ account_id }} paused spending'

With structured entries, performing custom analysis on this data becomes very easy.

I am still at an extremely early stage at designing this refactor. I would not be surprised if there are existing implemention of this pattern somewhere.

Rather than re-inventing the wheel, I would like to see how others do it. Are there existing plugins/tools out there that are already implementing this pattern?

Answer 1

At this moment memory is relatively cheap.
Instead of having one record in database and building log system around it - create new record every time application make change to the entity

You can retrieve current entity from lastly added record and you will have all nicely structured full logging data.

Another approach could be to use same table schema for log table, so you will have separate log table with same schema for every entity table you want to log.

Answer 2

Two things come to my mind with your question.

For many programming languages there are elaborated logging frameworks (e.g. the Log4J family for Java, C#, C++ and others). I'd recommend to use one of them (if you find that logging is the solution your task).

I'm not sure that your problem falls into the typical "logging" category. I typically produce logs for rather short-living information, maybe a week or so. My logs target system administrators or developers, by recording important actions that the system took or extraordinary situations (errors / warnings).

Recording user actions over a long enough time to support statistics is something I wouldn't do with a log file, but with a database table transaction having columns like type, source_account, target_account, amount, and the like. That makes creating statistics easy.