Should I be logging every request in a web application?

Question

I know that logging is supposed to tell a story about what your users are doing. Eg:

User 1 created a thing
User 2 deleted a thing
User 1 tried to access a thing and encountered an error

This is very useful, but what is even more useful is having detailed information about every single HTTP request:

HTTP GET /Index, UserAgent="...", Username="...", HttpResponseStatus 200
HTTP POST /Index/123, UserAgent="...", Username="...", HttpResponseStatus 201
HTTP DELETE /Index/123, UserAgent="...", Username="...", HttpResponseStatus 200
HTTP GET /Index/123, UserAgent="...", Username="...", HttpResponseStatus 500

Having information on every request the user made up to the error is very useful when debugging, although this sort of information crosses over into analytics territory a bit and pollutes your logs with a lot of inane requests (HTTP GET /Index x 1000).

Many years ago I was taught that you shouldn't log everything. Your logs should "tell a story" as above and nothing more.

Somewhat recently in the C#/MVC/ASP.NET world, this sort of logging seems encouraged by Serilog which has built in "Enrichers" which will log request level properties by default:

var logger = new LoggerConfiguration()
            .Destructure.UsingAttributes()
            .Enrich.With(new HttpRequestIdEnricher())
            .Enrich.With(new UserNameEnricher())
            .Enrich.With(new HttpRequestRawUrlEnricher())
            .Enrich.With(new HttpRequestUserAgentEnricher())
            .Enrich.With(new HttpRequestTypeEnricher())
            .Enrich.WithProperty("ApplicationRole", "Web")
            .Enrich.FromLogContext()
            .WriteTo.Seq(ConfigurationManager.AppSettings["SeqServer"])
            .CreateLogger();

In conjunction with Seq, these HTTP requests can be queried and matched up with your traditional log statements to tell a more verbose story of your user's interaction with your web application.

Is this kind of logging considered a good practice now, or is there a better way to collate HTTP level logs?

Answer 1

I would say both yes and no, and here's why.

Yes, all requests should be logged.

No, you should not be logging them.

If that still doesn't make sense, let me expand. :) It's got to be done, but it's not your application's job to do so. If you're using ASP.NET, I presume that you're hosting it on an IIS server? IIS should already be configured to log all requests. If it's not, make sure that changes. Your application should log everything that's in it's own scope of responsibility. Things that the raw request won't necessarily know (e.g. that this user did this thing).

You can also (should also, for a production app) ensure that all logs (request, auth, app, etc) are shipped to some log aggregator. This both makes it easier to tell the 'full story' as you say as well as guards you against hackers that manage to get in and then want to hide their activity by mucking with your logs.

Answer 2

I think nowadays with the cheapness of storage and bandwidth and the availability of centralized logging, logging everything is easier and you can filter logs in real time. You can be as verbose as you want and not affect the ability to read the log "story".

Even if you're not using centralized logging, most logging frameworks support segregating logs into different files based on some criteria, so you could send your raw HTTP logs to one file and send your log "story" to a different file.