This question is a result of an internal debate involving R&D, DevOps and Automation departments of the company I work for. Here's the nutshell of the debate:
automation: We need SSH access to all and every container during and after tests to automate (at least partially) RCA, if we don't have this, a lot of test failures will remain unexplained, and automation department will have to do a lot of manual labor to analyze the tests. The time better spent writing new tests / providing better test coverage.
R&D: We would be happy to have SSH access because we want to be able to casually debug application after it is deployed, or, reconfigure it for the purpose of testing using configuration parameters that need not be generally available / documented.
DevOps: Providing SSH access is a security risk (what if we forget, and leave it in production?). Parts of application deployed in the cloud, so, even though there aren't any actual data in that deployment, the computing resources can be compromised. A lot of containers used in actual deployment are third-party containers which come without SSH daemon, sometimes without any OS at all, so deal with it: you won't have complete access to every container you want.
What do you say, is a requirement of SSH justified? Should DevOps make an effort, or should automation / R&D think about workarounds?
If you have an opinion, can you back it up with a reputable source?
The actual deployment involves a dozen of Amazon EC2 instances running Kubernetes cluster with mostly Scala microservices combined with Amazon RDB, ELK etc. services and some non-containerized Amazon EC2 boxes dedicated to things like Kafka / MQ etc.