DevOps Team With No Security

A friend is going through an interview process at a well-known IT security company. This is one of the interview questions:

"Imagine a situation where you come to a DevOps team with no idea on security. What would be your first step to implement security into their processes and why?"

To me, as a DevOps Engineer, this seems like a very wide question, perhaps even deserving of a blog post. I wouldn’t want to be in my friend’s position to have to answer it, but I’ll do my best to give my 2 cents.

The theory

Nowadays, the term DevSecOps is becoming popular. While traditional DevOps supposedly considers security an afterthought, DevSecOps on the other hand considers security as one of the initial corner stones.

Currently, there is an abundance of both DevOps and DevSecOps positions on the job market. In my experience, the two terms are commonly used quite interchangeably. Nevertheless, it offers a glimpse into how seriously is security meant to be taken in this line of work. Indeed, when it comes to keeping critical IT systems running, it’s usually the DevOps engineers who are in the best position to either keep everything afloat or sink it. For that reason, security is a critically important aspect to be kept in mind for DevOps engineers.

The reality

In my experience, vast majority of DevOps engineers are security-conscious. Some slightly more, some slightly less. Most are aware of how delicate the work they do is, and that if they produce an insecure solution that leads to a security incident, there will be serious consequences.

Even the most mediocre DevOps Engineer is likely to know basic principles of how to keep their solutions reasonably secure. Many DevOps engineers work within teams, and their solutions are typically reviewed with scrutiny. Thus, I’d say a well-functioning DevOps team typically produces sufficiently secure solutions.

Within bigger, enterprise-grade companies, cooperation with various security teams is common. For example, a security team may audit, review, or set security requirements for the solutions created by the DevOps team.

In small companies with barely any DevOps engineers and perhaps no dedicated IT security personnel, the security may be a bit more lax. But even then it tends to be good enough to not be too vulnerable to most types of security incidents.

When an urgent incident occurs, security-related or otherwise, it’s typically the DevOps engineer who’s paged in the middle of the night. This ensures they’re fairly motivated to deliver robust and secure solutions.

DevOps/Kubernetes security-focused consultancy companies exist, such as Control Plane.

Non-exhaustive list

• Managing Public Cloud infrastructure (e.g. AWS)

  • all sensitive credentials must be securely stored and securely provided to the correct users/machines
  • IAM (Identity Access Management) must be securely configured to only provide minimum access to minimum amount of people
  • most infrastructure should be on private networks, inaccessible from the outside
  • networking must be strictly configured at the point of exposure to the outside (firewall rules, DDOS protection, etc.)

• Managing CI/CD

  • should be fully secure from an outside attack
  • should be reasonably secure from an insider attack/mistake

• Virtual Machines (on-premises or cloud)

  • SSH hardening
  • Regular updates to avoid vulnerabilities

• Kubernetes

  • RBAC (Role-Based Access Control) must be securely configured to only provide minimum access to minimum amount of people
  • Kubernetes security hardening

• Secrets Management

  • e.g. Hashicorp Vault, CyberArk, etc.
  • secrets should be rotated, only short-lived tokens should be handed out, it should be integrated with some sort of user management (e.g. AD), etc.

• TLS

  • insecure protocols such as HTTP should be avoided
  • Let’s Encrypt
  • Where to do TLS termination and where insecure protocols are sufficient

• VPN

  • Many companies put almost everything behind a VPN for extra security

• Database security

  • Users/passwords security

• Managing git servers, e.g. GitLab

  • particular care required when exposed to the outside Internet
  • Users/passwords security
  • users shouldn’t commit sensitive data, such as credentials

• Container vulnerability scanning

  • e.g. Trivy or Snyk

• Storage encryption

  • e.g. AWS S3

• Intrusion detection

  • e.g. Falco

• Enforcing MFA wherever possible

• Generally trying to follow the least privilege principle

• Generally trying to follow, avoid, and patch CVEs

• Incident prevention and response (angry ex-employee, DDOS attack, aggressive web crawlers, critical CVE patching…)

Real-life examples of security done poorly in DevOps

A DevOps engineer committed a Kubernetes admin kubeconfig in plaintext format into a developer app git repo. He then tried to use it for authentication as part of his CI/CD solution. This is bad, because we wanted to keep the admin kubeconfig away from the developers. Also, he shouldn’t commit sensitive plaintext credentials into a git repo anyway. The rest of the DevOps team had a much better CI/CD solution in place, so all he had to do was ask and cooperate.

The team used a publicly accessible Jfrog artifactory where they uploaded private Helm charts. But then someone accidentally set the folder permissions to allow the Helm charts to be downloadable without authentication. The Helm charts contained some sensitive credentials, and were now downloadable by anyone from the Internet. I noticed it, escalated, and ensured remediation. Then the folder was made private again. I think the artifactory shouldn’t be accessible to the Internet, because that just removes one layer of protection right off the bat.

Conclusion & My answer to the question

The scenario in the interview question is quite a nightmare, and extremely unusual. I would look at it from the leadership/management point of view.

• The DevOps engineers need to feel responsibility and ownership over delivering secure and robust solutions – that’s a key part of their job.
  • If something breaks, it needs to be them who’ll rush to fix it.
  • This should give them the necessary motivation.
  • In order to help them accomplish this, they need to be given the resources and freedom to get things done the way they feel is best.
• The DevOps engineers need to be lead by example.
  • They need to have a skilled senior who’s going to show them the better way, and work with them until their work culture is sufficiently improved.
  • Even as a regular team member, you can review the work of others, and raise identified issues, as long as you can successfully prove your correctness.

Bad team culture takes a lot of time to fix. Rome wasn’t built in a day. With the above measures, it might still take 6-12 months for the team to become sufficiently security-conscious.