6

I'm working on a web application where the server side makes calls to a foreign server's web service API. The data passed to this API could contain sensitive data (like a user's password) as part of the header and/or in the body of the request. I want to be able to log what's in the request (URI, header, body, response) for troubleshooting purposes but I don't want to expose any sensitive information. I have a single method that makes this API request which other methods call and that's where I've been putting the log statements. This of course reveals everything, including sensitive data. Is there a common way to do this?

1
  • 1
    Is the API encrypted? Sep 14, 2016 at 21:06

2 Answers 2

13

I'm pretty dismayed at these other answers. Best practice is only to log what you need to log, so not the full request and response.

You must then consider data protection law. Obviously this varies by jurisdiction but generally it follows the same principles:

  • You must document what you are doing, why and the precautions you take to secure the data.

You will find it difficult to justify a blanket log everything policy. 'because its easy' doesn't count as a good reason.

  • You will probably have had to ask your customers for permission to use their data for the purpose you describe.

Although a blanket 'tick here to agree to all TnCs' with '... use your data for maintaining the system..' might seem to cover you this has been challenged in court a few times now. The reality is you wont know until you are sued.

  • Then you probably have some classes of data which have to be encrypted.

  • Then you have to have a policy around how long you keep the data and how you dispose of it.

Data protection law can be weirdly strict on some things and lax on others. For example I remember one case where in flight meal options were all encrypted because you had options for halal, kosher etc so they would potentially reveal the travellers religion, which counts as sensitive personal data.

Things that wont cut it.

Sanitising the logs. If even one thing slips though you are screwed. Not because of the data. Because of all that documentation you wrote which said you didn't store that data. And stuff will slip through.

Logging everything but to an encrypted store. It doesn't matter how secure it is because the law covers more than just keeping data safe. Any scenario where you are keeping 'everything' is going to be tripped up because you have to state the purpose of keeping it.

******** edit : suggested ways of solving bugs without logging ************

So for api calls with malformed requests I find its best to return a nice error message. It shouldn't contain any data, but it should indicate clearly and uniquely what the error was.

ie

BAD : error - user bob cant eat pork

GOOD : error - meal code not allowed for this user

Ensure you push the error all the way back to the user if the code cant handle it. Bob knows he loves pork and will phone up to report the error. Plus because there is no data, you can log it without worry and check for patterns.

rule 2 : testing testing testing. Write those unit tests and run em all the time. When Bob phones and says he cant order pork because of a bug, write a failing test before you fix it so you know if it ever happens again.

rule 3 : health checks, if you have an api or service, pop a health check method on it. Have you monitoring system hit that health check every few minutes to check everything is ok with the service, it can still talk to the DB is still answering requests promptly etc

4
  • 2
    With the exception of paswords surely anything hitting the API is going in the db anyway, I can say for one that problems in production are a nightmare to diagnose and if they were the problems you were expecting you'd have fixed them already. Further with API logs the thing I've found most in them is "you called the API wrong"; I don't see how you'd be able to find that without a "log (almost) everything" approach Sep 14, 2016 at 21:37
  • 2
    in the DB you can say 'look we are storing this data on 1000 people and you can see it is encrypted with blah blah and we delete it after 6 months'. then the auditor says.. 'what about that plain text log file there?'
    – Ewan
    Sep 14, 2016 at 21:41
  • 4
    I like this answer better than any log-then-sanitize or log "securely" strategy. If the data is sensitive to being logged you can't log it period. Pissing on that requirement because it's hard is wrong. It can impact people's lives, send people to jail, and in certain circles get people killed. There are cost benefits in any thing but this is the kind of thing you don't screw with. Log no more than should be logged. Sep 14, 2016 at 22:29
  • I like this answer over sanitizing. If you control the code where the logs originate, and you log failed authentication requests, then I think it's fair to log that it happened but nothing else. No request body or headers. The URL is a grey area. I could argue as long as your remove everything after the hostname it is OK. Aug 27, 2019 at 14:15
-2

Sanitize the log. The routine that logs API requests should search for and be able to recognize sensitive parameters such as passwords and replace them with *s or similar, and then pass the censored output on to the normal logger.

2
  • 1
    Would the downvoter care to comment? Sep 14, 2016 at 21:23
  • 3
    I've added my own answer. I wouldn't normally down vote, but this is plain crazy. one bug and you've exposed a million passwords before you notice
    – Ewan
    Sep 14, 2016 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.