62

I have an enterprise application running that uses both MySQL and MongoDB datastores. My development team all have SSH access to the machine in order to perform application releases, maintenance, etc.

I recently raised a risk in the business when users started storing highly sensitive data on the application that the developers have indirect access to this data which caused a bit of a storm, so I have now been mandated with securing the data so that it is not accessible.

To me this does not seem possible because if the application has access to the database then a developer with access to the machine and application source will always be able to access the data.

Improve this question
19
  • 30
    Users should be storing sensitive data only in encrypted form. There shouldn't be a huge problem if developers can access the encrypted form, as long as the matching keys are properly shielded from them.
    – MSalters
    Dec 4, 2014 at 13:13
  • 3
    @Clinton Do you have separate admin and developer teams? The server admin can always read the data and encryption doesn't help since they can easily get the key.
    – CodesInChaos
    Dec 4, 2014 at 15:09
  • 14
    To be completely honest, this is a complicated matter and doing it right requires a lot of expertise in data security. Even if you knew exactly what to do, you will face business opposition, political and technical roadblocks. I highly suggest you bring in a data security consultant. They not only know what to do here, upper management typically gives more credence to when a third party is telling them to change. Upper management generally doesn't put as much stock in what their internal experts are telling them.
    – maple_shaft ♦
    Dec 4, 2014 at 15:11
  • 3
    Might be worth asking on Information Security Stack Exchange. There's some related info on this question
    – paj28
    Dec 4, 2014 at 15:41
  • 23
    Why are humans touching the server and deploying code?
    – Wyatt Barnett
    Dec 4, 2014 at 15:58

8 Answers 8

Reset to default
89

Security is not a magic wand you can wave at the end of a project, it needs to be considered and built in from day 1. It is not a bolt-on, it is the consistent application of a range of solutions applied iteratively and reviewed regularly as part of a whole system, which is only as strong as the weakest link.

As it stands you have flagged a security concern which is a good first step. Now as a minimum you have to define:-

  • What data you are trying to protect?
  • Who are you trying to protect that data from?
  • Who actually needs access to what (and when)?
  • What is the legal/financial/business impact of that data being compromised?
  • What is the legal/financial/business need for a person/group having access to the data?
  • What budget is the business willing to assign to a "get secure, stay secure" strategy when it was not a business requirement previously?
  • What access does the system need to the data?
  • What does this process and systems this application rely on?
  • What is done to secure those environments?
  • Who is going to be responsible for implementing it and reviewing the whole process?

Until you know all those in detail you really don't have anything to work with. That information will define what mitigations to those threats you can (and cannot) apply and why.

It may be that the best thing to do is recognise that you don't have the necessary experience and that it would be best to bring in someone new with that experience. I quite often hear the response that there's no budget - if it is considered genuinely important then the budget will be found.

Improve this answer
2
  • 33
    Whoa...that makes security sound...non-trivial. (Sorry for the sarcasm; I've seen a lot of people surprised by this.)
    – Paul Draper
    Dec 5, 2014 at 5:46
  • 4
    I do believe that a number of people think there is a magic make-application-secure command they just need to run.
    – TMH
    Dec 8, 2014 at 11:13
27

You are right. If an application is capable of accessing content stored on corporate machines without the user passing extra information every time, then the programmers, maintainers, sysadmins etc. of the service provider can access that content. This is fundamentally unavoidable, and a major source of insecurity (Edward Snowden was a sysadmin, and had special privileges above "Top Secret" because there simply isn't a way of not granting them.)

The only way of avoiding this is to require the user to provide information that never makes it to the corporate machines. This is tedious and error-prone, since no one can possibly remember enough information to form a secure access barrier, and therefore people will immediately begin to store their credentials electronically in some place, which will then become the new attack target (and probably a weaker target than the one you are maintaining). But if you want to truthfully assert "Our employees are not physically capable of accessing our users' content", it's the only way.

(Note also that such a business model seems to be becoming politically untenable. Businesses have been forced out of operation by security services for trying to do exactly this. I understand that giving privacy guarantees has business value, but it can't possibly have more business value than the fundamental goal of staying in business.)

Improve this answer
5
  • 6
    It is possible to design hardware in such a way that it is physically impossible to access certain data without creating a permanent record of such access which could not be destroyed without the collaboration of multiple independent people, and which even with such collaboration could not be destroyed without leaving clear evidence of deliberate destruction. Updating such systems to handle changing requirements, however, is apt to be very expensive compared with updating software-based security systems.
    – supercat
    Dec 4, 2014 at 17:29
  • 5
    You're right, I completely forgot to mention auditability as a possible alternative to zero-knowledge hosting. It's somewhat easier to achieve and often enough for the business case.
    – Kilian Foth
    Dec 4, 2014 at 18:26
  • Your last paragraph. Are you referring to LavaBit-type stories? I'm confused.
    – jpmc26
    Dec 6, 2014 at 4:13
  • 1
    @supercat You also have to trust that the creators of the hardware made it do what they said it does.
    – user253751
    Dec 8, 2014 at 2:37
  • 2
    @immibis: True, but I would the design and manufacture of secure hardware could be audited by multiple independent people. Further, in a conventional system it would be possible for a "sneaky" piece of code to do something and then delete itself without a trace, but if a piece of secure hardware isn't supposed to have a writable control store, such a thing would be impossible. Either sneaky code would have to be permanently in the control store, or the control store would have to have a permanently-wired means of modification, either of which would be detectable after-the-fact.
    – supercat
    Dec 8, 2014 at 14:48
15

You're quite right; some developers will always need access to the Live data, if only to diagnose production problems. The best you can do is to limit the potential damage by reducing the number of people involved. 

With great power comes great ... opportunity to really, *really* foul things up.

Many developers won't want that responsibility and others, just won't be "ready" to hold it, and so shouldn't have.

Question: Why is your Development team performing Live releases?
I would suggest you need a Release Management "team" (even it that's just a subset of your team, plus Business representation to make any on-the-day "decisions", like "Go/No-Go")? This would remove much of the "need" for developers to touch anything Live.

Do you have any sort of non-disclosure/ confidentiality agreement between developers and company? It's heavy-handed, but it might have some merit.

Improve this answer
5
  • Which is still won't stop determined wrongdoer from hiding backdoor in the application, but it does reduce the opportunity that makes the thief.
    – Jan Hudec
    Dec 4, 2014 at 12:55
  • Yes, it is not the entire development team but rather a subset/release management team. We certainly have a clause in the employment contract about snooping around data you shouldnt be, it is a dismissable offence.
    – Clinton Bosch
    Dec 4, 2014 at 13:06
  • @JanHudec Especially since adding code the the application leaves traces in version control.
    – CodesInChaos
    Dec 4, 2014 at 15:13
  • @CodesInChaos: Good programmer can make backdoor look like an honest mistake. You'll suspect them, but you'll never make a case against them. But yes, it's another line of defence.
    – Jan Hudec
    Dec 4, 2014 at 15:32
  • @Jan: Which is why all code changes should be reviewed and signed off before being allowed into the release branch.
    – SilverlightFox
    Dec 6, 2014 at 11:51
9

The problem is your developers having access to the systems. If they need production data for debugging then give them a database dump where all that sensitive information is removed. Then the developers can work with that dump.

Deploying a new version should not involve any developer - that a pure admin task, or even better - a fully automated task. Also be aware of releasing and deploying being two very different tasks. If your process isn't aware of that then change it accordingly.

Improve this answer
3
  • 1
    We do not need production data from debugging, we have a sanitised data dump for that, but sometimes a deployment requires various data migrations etc. which are run by some developers in the release management team (but they are still developers)
    – Clinton Bosch
    Dec 4, 2014 at 13:24
  • 2
    @ClintonBosch Then you haven't clearly separated the roles of admins and developers. Then also one more question you should ask yourself is: how do we make sure that the software that is released also gets actually deployed? You would need to sign on release and only allow deployment of signed packages on production. Also again automation is your friend. Migrations shouldn't require any manual steps.
    – SpaceTrucker
    Dec 4, 2014 at 13:47
  • 4
    @ClintonBosch Identify what data fields are highly confidential and encrypt them. Make sure that you put production OS security so that you can access which user ids are reading the key file to make sure nobody but the application user is doing this. Don't give the developers the app user password. Make them sudo to get rights on production and log what they are doing. This is probably the safest way to make sure that you are babysitting the few people that would have access and so that they can't casually or accidentally see data they aren't supposed to.
    – maple_shaft ♦
    Dec 4, 2014 at 15:17
6

Rule #1 of security: If someone has access to information, they have access to that information

That tautology is annoying, but it is true. If you give access to an individual, they have access to the data. For users, this usually means access control, but for developers... well... they're the ones that have to write the access control.

If this is a major issue for you (and it sounds like it is), consider building security into your software. A common pattern is to design secure software in layers. At the lowest layer, a trusted development team designs software which manages the most naked of access control. That software is validated and verified by as many people as possible. Anyone designing that code has access to everything, so trust is essential.

After that, developers can build more flexible access control on top of that core layer. This code still has to be V&Vd, but it isn't quite as stringent because you can always rely on the core layer to cover the essentials.

The pattern extends outwards.

The hard part, indeed the art of designing these systems, is how to build each layer so that developers can continue to develop and debug while still providing your company with the security you expect. In particular, you will need to accept that debugging demands more privileges than you think it should, and attempting to lock that down will result in some very angry developers.

As a side solution, consider making "safe" databases for testing purposes where developers can rip out all of the safety mechanisms and do serious debugging.

In the end, both you and your developers need to understand a key tenet of security: All security is a balance between security and usability. You must strike your own balance as a company. The system will not be perfectly secure, and it will not be perfectly usable. That balance will probably even move as your company grows and/or demands on developers change. If you are open to this reality, you can address it.

Improve this answer
3

Set up two deployments of the application which also use separated database deployments. One is the production deployment and one is the test deployment.

The test deployment should only have test data. This can either be fantasy data which got created for that purpose or a copy of the production data which was anonymized to prevent the developers from finding out the real people and entities behind the data.

Improve this answer
1
  • Yes, this is exactly the scenario that we have. But at some point somebody needs to work on the production environment to facilitate a deployment/data migration
    – Clinton Bosch
    Dec 6, 2014 at 16:32
3

In two financial firms, developers did not have access to production machines. All requests to modify production machines had to go through an approval process, with a script, and was approved by managers. The dev-ops team completed the actual deployments. I assume this team was employees only, and passed background checks. They also did not have developer knowledge so probably couldn't snoop if they wanted to. In addition to this, you would encrypt all database entries using a secret key stored in the environment variables. Even if the databases leaked publicly no one could read it. This key can be further password protected (PBKDF) so only an executive can unlock it. Your system could require the executive password upon boot (or more likely delegated to dev-ops or a dev-ops manager). Basically the strategy is to disperse the knowledge so a critical mass of required knowledge does not exist in one person and there are checks-and-balances. This is how Coca-Cola protects its formula. Honestly, some of these answers are cop-outs.

Improve this answer
0
-1

MongoDB has limited security controls and depends on a secure environment. Binding to a specific ip and port (and ssl since 2.2), and a crude authentication, that's what it offers. MYSQL adds GRANT o ON db.t TO... Data at rest is not encrypted, and ssl is not used by default. Create a fence. Readonly access for developers to application related log files should be enough to debug. Automate the application lifecycle.

Ansible helped us automate standard operations (deploy, upgrade, restore) over many single-tennant environments while using distinct encrypted vaults to store sensitive environment variables such as hosts, ports, and credentials. If each vault can only be decrypted by different roles, and only on a bastion host, for logged operations, then auditability provides acceptable security. If you grant SSH, then please use selinux to avoid key tampering, use a bastion host with ldap/kerberos authentication for administration, and use sudo wisely.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.