What design would you propose to log reporting data so that the capture of, say, an additional form field can implemented easier?

Question

Why is there an emphasis to log reporting information through an application to a separate database?

I have seen designs where a logging manager is available to every business layer class whether it needs it or not. This in itself can create overhead?

I was thinking that reporting information logging should be done as a separate service if it needs (at all) to consumed within another application. Nonetheless, much of the data (especially where data is stored as schemaless xml) just wastes database space.

Why are there so many bad designs such that it becomes difficult to respond to reporting requirements without changing, say three, different software layers?

What design would you proposed to log reporting data so that a change in reporting requirements can be done with ease? Typically, a change in report requiremnets would involve a request to the capture of an addition field on a form.

Answer 1

A lot of this does depend on the tech and solution, but I have used several tools that definitely do not think that way.

If you are doing web using something with an event model, some concept of a request/response lifecycle you can very easily do all your logging by hooking the exception event and hooking the response completed event then just maintaining a collection of loggable things through the lifecycle.

If you are doing a desktop app things get a little more varied. There usually are still some common events to hook, but other times you need to probably call out to the logger explicitly because saving everything to the end of the lifecycle would get messy quickly.

In .Net you can solve a lot of this by doing some implementing a custom trace listener, that way you can use the default trace reference, but still get data that doesn't look terrible when reviewing.

As for logging in a separate db or not, it depends on a lot of things like if the log has to be a valid audit source, and who has access to the data, and if you are trying to log issues connecting to databases :-)

If I need to search the log efficiently I do not store the messages as big blobs, but sometimes I do create the messages as a blob and split them up into the final format in a separate process. What you do to get the message in the first place will depend entirely on what you are doing. I have some apps where the overhead of serializing the objects is simply too much for ongoing logging not to have a perf impact, let alone the fact I would run out of storage in a few hours. Most of those I just log id strings. Other apps have objects I would like to log as serialized objects, but the object isn't serializable, for those you can use a custom formatter, or an intermediate. Other times I have the luxury of low throughput and serializable objects so I just use the default serializers.

One bit of advice, if you are really going to make the logging endlessly extensible mace certain you have a way to limit/throttle it

Answer 2

That depends on the technology in use.

I'd probably use a Dependency Injection container and a clearly defined interface for the logger to manage this cross cutting concern. You can then inject the logging service to where you need it.

If the logger needs to capture a new field in a class, I'd probably do it by using annotations/attributes, so you have a single point of change when you need to extend the datastructure. The logger just looks out for these attributes/annotations when you log a data object and react accordingly. You can also encapsulate complex logging-logic in those attributes/annotations. This works very well when you're databinding to objects.

Example in C#:

public class Customer
{
    [LogData]
    public string Name
    {
        get;
        set;
    }

    [LogData]
    public string FirstName
    {
        get;
        set;
    }
}

or

[LogData] //dumps all fields of the class to the log
public class Customer
{
    /*.../*
}

However, when you are binding your fields to a dataset or something similar, you'll need a different approach. I'd then create a logger that just dumps all the columns of the dataset by default. Since such datasets mostly consist of auto-generated code, it's hard to annoate them, but you could tie the type of the dataset to a configuration in which the fields to log are specified if you need to be more precised about the data being dumped, resulting in a single point of change as well when logging needs to be extended.

I'd store the logging information in a single content column (along with some more information), probably as XML. Space is of no concern here, as it's very cheap these days and the overhead of XML processing is negligible as well. If you need to have direct database-access to all logged data fields, you'll not get around creating a separate field for each business logical field in your application, which means you will have to extend the database. You could probably just always clone the original database and then log to this logging database in a whole new database schema. An ORM could thus just switch the schema and persist the logged objects quite easily, if you are using an ORM, that is.

Answer 3

Implement something to the nature of ActiveRecord. This has been done in a number of languages. Similarly, design the same 'active' concept into any part of your work that will need the feature (for instance something like "ActiveForms")

Basically the minimum of what you want is to

A) Detect changes in a data set (form fields)

• Keep a store of the current known version's structure
• Compare CurrentStructure to LastKnownStructure. If different then,..

B) Trigger dynamic updates

• Let an object, database or other method store a list (also dynamically defined) of reports/report sections/etc.. which relate to the data.
• Loop through the list and batch update or have each item in list check the data set's object for an update flag.
• Based on context, update or continue code

Just a note, you could just trigger full update at report generation (as opposed to on field changes) If there are many changes happening all the time, you may save yourself a good chunk of overhead. If you do this, there may need to be some sort of hierarchical topology..a meaningful, context aware method to traverse the data sets and check for changes.