0

This question is a result of an internal debate involving R&D, DevOps and Automation departments of the company I work for. Here's the nutshell of the debate:

automation: We need SSH access to all and every container during and after tests to automate (at least partially) RCA, if we don't have this, a lot of test failures will remain unexplained, and automation department will have to do a lot of manual labor to analyze the tests. The time better spent writing new tests / providing better test coverage.

R&D: We would be happy to have SSH access because we want to be able to casually debug application after it is deployed, or, reconfigure it for the purpose of testing using configuration parameters that need not be generally available / documented.

DevOps: Providing SSH access is a security risk (what if we forget, and leave it in production?). Parts of application deployed in the cloud, so, even though there aren't any actual data in that deployment, the computing resources can be compromised. A lot of containers used in actual deployment are third-party containers which come without SSH daemon, sometimes without any OS at all, so deal with it: you won't have complete access to every container you want.


What do you say, is a requirement of SSH justified? Should DevOps make an effort, or should automation / R&D think about workarounds?

If you have an opinion, can you back it up with a reputable source?

The actual deployment involves a dozen of Amazon EC2 instances running Kubernetes cluster with mostly Scala microservices combined with Amazon RDB, ELK etc. services and some non-containerized Amazon EC2 boxes dedicated to things like Kafka / MQ etc.

2

You should maximize amount of valiable data and configuration options and minimize number of elements you need to access for this data and configuration.

How to achieve it?

First, analyze what you really need:
- if you need logs from the machine create mechanism to provide these logs without loging into the machine itself (there are many solutions, e.g. ELK you have already mentioned)
- if you need to change parameters of the application/server/db you should automate this process and extract these params as a part of automate deployment (I assume you use come CI/CD tool)
- if you need to change parameters frequently for test purpose then consider separate environment for that. In that case you can access machines without any risk.

Probably you have many other cases - the rule is to identify WHAT you need and HOW you can achive it without touching machines. It is not only a security concern but also maintainability issue if you need to log into dozen of machines. Centralization of logs, monitoring and configuration is really benefitial in this case and pays all cost of automation off.

  • "there are many solutions, e.g. ELK" -- they are all times worse from user perspective than plain SSH. Not to mention, expensive. "if you need to change parameters" -- we are talking about what happens during CI, most of the times, these parameters need to be introduced ad hoc, at the moment, not planned for weeks in advance. The speed at which you can do this is the most important factor. "consider separate environment" -- this doesn't help because most of the time you need to work on that specific environment, because the bug reproduces only there. – wvxvw Mar 14 '18 at 14:53
  • Regarding centralized logs - how does checking one page with logs (taged) can be worse then loging to dozen of machines separetly? It takes time and time is money. Imagine (and in the world of microservices it is rather a standard) having 100 microservices. Debugging in such a environment is a nightmare! – Tomasz Maciejewski Mar 14 '18 at 15:03
  • Regarding parameters - if you have contenerized environment it shouldn't be a problem to duplicate your prod infrastructure and configure there. You can have access to anything without any risk. But think twice if you really have infinite number of params which you cannot list upfront. – Tomasz Maciejewski Mar 14 '18 at 15:05
  • "how does checking one page with logs ...?" -- Murphy's law, the important logs are never there. ELK filters are too restrictive, they don't combine with any other tools, most people won't go as far as to try to learn that. grep is easier and combines well with other tools. With or without ELK debugging microservices is a nightmare. Presence of ELK is orthogonal to this problem. – wvxvw Mar 14 '18 at 15:11
  • "it shouldn't be a problem to duplicate your prod" -- it never works like that. Did you ever actually try doing that? It sounds nice in PowerPoints, but in practice, is nothing like that. Bugs that are easy to reproduce in any environment would've been probably caught earlier in the testing cycle. – wvxvw Mar 14 '18 at 15:14
1

DevOps, CI, The Cloud, Containers, VMs etc is all about abstracting your running code away from the metal box that runs it and just having CPU power/storage/network etc as a utility.

You can't SSH to a box when you don't have boxes.

You might not be living in this perfect world yet. But it should be your goal. That means resisting these kind of requests/requirements.

Lets look at the reasons for the requests

  1. automate (at least partially) RCA

Instead, expose whatever the thing is they are checking to a metrics or monitoring tool.

  1. debug application after it is deployed, or, reconfigure it for the purpose of testing

No no no. Make a new version and deploy it.

  • What makes you think it's the "perfect" world? To me it seems like the exact opposite. There isn't such thing as "no boxes". It's always "other people's computers". That's just how computers work: they have operating systems, which run user programs. If anyone tells you otherwise, they must be selling you a London Bridge. Other than that, what you say isn't even a falsifiable hypothesis, it's more of a religious credo, I cannot relate to that... – wvxvw Mar 15 '18 at 7:27
  • 'perfect' in the context of DevOps – Ewan Mar 15 '18 at 7:29
  • @wvxvw just reading through your other comments, its obvious that you disagree with the ideas behind 'devops' (for want of a better term) You are swimming against the tide I think. Maybe ask another question which clearly asks about the benefits (if any) of the approach – Ewan Mar 15 '18 at 7:33
0

You need a way to instrument and summarize your system as a whole. Most of the successful stories that discovered the root cause in a microservices infrastructure had the following:

  • Instrumented microservices
  • Traceable transactions across microservices
  • A team devoted to analyzing the information

Netflix and Spring have support for microservices that fit in to this need fairly well. A summary of technologies to look at include:

  • ELK stack--consolidate all logs in one place with robust ad-hoc searching
  • Zipkin--trace a user's request all the way through each microservice--including asyncronous ones
  • Netflix Eureka and Hystrix--service discovery and health monitoring

There's a few articles that talk about product suites to handle this sort of thing such as:

You mentioned that you are dealing with Scala services, and I did see support for Zipkin that you can add.

The items listed above are what my company uses, but the articles mention products we haven't explored yet. Hopefully this gives insight into how to determine root causes that are more systemic in nature. The AirBnB video had some good practical application dealing with a time that one of their services was going out of control regularly, and how that affected the microservices system.

  • You mention Netflix, but Netflix is a failure... they spend tons of money on their infrastructure. I know, they blog a lot about it, but if you let the marketing fairy dust settle, you see that they simply lose a lot of money doing a lot of busy work... I don't want to be like Netflix: it's a bunch of hollow promises, smoke and mirrors with no end goal in sight. When you advise for using ELK, did you ever actually bothered to measure the efficacy of grepping the logs vs working with it? Because I did... it suck to use ELK. – wvxvw Mar 15 '18 at 7:32
  • 1
    @wvxvw, I mentioned technologies created by Netflix which my company uses and finds value in. Your opinion on Netflix as a whole doesn't change that they have made some useful open source technologies. ELK is a tool, and every tool has a threshold where using it is more efficient than not. You probably haven't hit that threshold yet. If grepping logs works for you at the scale you are working, then continue doing so. – Berin Loritsch Mar 15 '18 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.