0

I'm designing an API and have reached the topic of logging. I'm going to store my logs in Elasticsearch.

I'm certainly going to do some logging at the time the HTTP response is sent back to the client, with info such as processing time, response code, user id, URL.

Is it best practice to also send a record to the logging system right when the HTTP request enters the API server?

What I have in mind here are situations when the response never materializes, e.g. because the server dies, or takes forever processing the request (e.g. due to bad business logic). If this occurred, I'd have no record at all of the client making a request.


Original wording below (to understand some of the answers). I tried creating a new question with the new wording and then attempted to delete this one. However, SO wouldn't let me delete this, and the new one started getting marked as a duplicate.

API logging: request, response, or both?

I'm designing a REST API and have reached the topic of logging. I'm going to store my logs in Elasticsearch.

Is it best practice to log both HTTP request and response, with some correlation id to match them in the logs? What are the advantages and challenges of doing it this way, as opposed to only logging requests or responses?

(I have some thoughts on this of my own: suspect it is best practice and see some advantages & challenges, but feel there's a lack of an expert treatment of this subject online. Hoping this question will result in one.)

Edit:

I'm NOT asking about whether to store in the logs the contents of every request and response. I'm asking whether to store some basic record for each request and response (e.g. timestamp, URL, IP, response code, some form of user id), or maybe just for requests, or maybe just for responses.

1

What is the goal of your logging? Logging request, response, and user info is perfectly valid if your goal is to build a profile of your users. Google certainly does.

If all you want to do is debug your service it's a bit much. Your focus should be on errors not recording how things went when it worked.

My 11th grade English teacher has the best advice for logging: Know your audience. Understand the needs of those that will use the log. If those people will have more than one goal you may want to create more than one log.

0

In my experience this is too much logging.

You should log errors and performance, which might be the path, response code, response time and count of http requests in the case of a REST api, but not query string parameters, request/response bodies etc.

Storing all that data has a cost and if your application is working you should never have occasion to look at it.

Additionally, storing the content of every message indiscriminately will fall foul of data protection laws.

  • Thanks for that! I clarified the question - I never intended to store the entire body of the request / response. Sorry for not being clear enough. – Jan Żankowski Jul 30 at 9:38
  • hmm request and response are usually a pair, so its unclear what you mean now. – Ewan Jul 30 at 9:40
  • What about requests failing to produce a response, e.g. when the server goes down while processing? This risk seems to ask for logging requests separately, not only after responses are produced. – Jan Żankowski Jul 30 at 9:46
  • sure, but what do those logs tell you that you dont already know? – Ewan Jul 30 at 9:49
  • I guess I can see it was under attack? Or which requests caused it to overload due to some bad business logic (e.g. some computation hogging the CPU)? (Btw, this kind of discussion is precisely what I was hoping for.) – Jan Żankowski Jul 30 at 9:57
0

Best practice is to log whatever you need. You should know your application well enough to know what information is required to troubleshoot issues. If you don't, then spend more time thinking about it.

0

When creating and debugging a webservice, it certainly helps to have proper logging to see if there are any issues. If there is an issue, you can easily find out the parameters passed to reproduce it.

For a production environment, you can get swarmed by log so logging just the errors/exceptions there makes sense.

A good logging framework allows you to filter the logging. Only log the 'error's for example.

We're using Dynatrace on production, it's a tool that logs all the webrequests that happen so no need for manual logging. Even better, you can see the call stack for each method. It's useful to find errors or figure out performance issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.