We've talked about how Continuous Integration and Continuous Delivery (CI/CD) tools can be a source of secrets sprawl. While it's not as insecure as leaving them laying around in a publicly accessible file, CI/CD pipelines can be exploited in a number of ways and I'm going to share a few with you.
This article is not exhaustive. GitHub's Security Hardening Guide for GitHub Actions alone is 16 pages long if you try to print it. OWASP's Top 10 CI/CD Security Risks is 38 pages long. Protecting your CI/CD systems is not a tiny task, but it's an important one. To get you started, here's a quick read on five ways attackers can leverage your CI/CD to gain access to additional systems.
#1: When someone can find your credentials
In an NCC Group report on 10 exploits they ran, their first exploit starts with an improperly permissioned S3 bucket. The bucket had a script with a hardcoded Git credential. From there, it was a matter of steps to gain access to the CI/CD environment and get more credentials the CI/CD processes had access to… Pwned!
GitGuardian has found millions of hardcoded secrets in Git repositories, but secrets can be anywhere: S3 buckets, screencaps, you name it. Secrets hygiene should be followed everywhere. And given how many exploits have started in misconfigured S3 buckets, if you're going to use S3, make sure you're using it correctly.
#2: When GitHub has your AWS credentials
While environment variables are considered somewhat of a minimum standard for keeping secrets out of your code, if the code that uses them is compromised or is poorly configured, it can start revealing them through your logfiles.
GitHub and AWS actually have an ingenious workaround for this, using OpenID Connect (OIDC). In this case, AWS uses GitHub as an identity provider (IDP) for an Identity and Access Management (IAM) role you create. That IAM role becomes accessible only by your repository, and you can even scope it to a specific branch. GitHub and AWS negotiate temporary tokens between themselves and no password is stored.
When no password is stored by the system, no password can be revealed by the system.
#3: When access is poorly managed
The number two item in OWASP's Top 10 CI/CD Security Risks deals with IAM as well. It goes into a number of ways you can improperly manage this. To quote them: "Ensuring each human and application identity has been granted only the permissions required and only against the actual repositories it needs to access is not trivial."
This isn't limited to the service AWS calls "IAM," either. Identity and Access Management relates to everything from a keycard to get in the door of the building to an access policy assigned to a specific branch of a Git repository. Ensuring your CI/CD processes can only touch what they're supposed to touch and see what they're supposed to see is crucial.
Additionally, the number of people with access to your CI/CD logging and how they're able to access it matters. If your CI/CD process is leaking credentials into the logs and a contractor's login is shoulder-surfed in a Starbucks, the clock to a complete takeover starts ticking.
Some of the worst hacks don't start with someone finding the keys to the castle laying around. They start with someone who found the keys to the stable laying around. In the stable, they found the keys to the scullery in a saddlebag. And so on and so on.
#4: When you obfuscate your credentials
Many CI/CD logging systems have some method of identifying and redacting credentials in the logs. But if you make it so the redaction filters can't spot the credential, but an experienced human can, you might just clever yourself into getting hacked.
In many demonstrations of poisoning pipelines (getting a bit of bad code into your CI/CD process), the hackers have the bad code output your passwords in base 64. For example, PASSWORD='LetMeIn' in base 64 is UEFTU1dPUkQ9J0xldE1lSW4n. While redaction filters may get better and better at spotting these kinds of things, some systems' redaction filters are just a handful of regular expression patterns. Converting secrets into another format can help sneak them past the sniffers that are there to protect you.
In GitHub's security hardening advice, one of the primary points of guidance for secrets is "never use structured data as a secret." Packing up your secrets in XML, JSON, or YAML may help put the helpful secret-sniffing robots off the scent and let those secrets escape into your logs.
#5: When every file has the same write permissions
GitHub offers a feature called CODEOWNERS. We don't mean to bang the GitHub drum, but this offers a practical application that can be translated to other systems.
With CODEOWNERS, you can set the permissions on your .github/workflows directory such that whenever someone tries to change anything in it (creating a "poisoned pipeline"), an admin specified by CODEOWNERS is not only alerted, but has to approve the change.
Depending on your systems, it's possible to lock down important files or directories within a development environment and ensure only the people who need to be able to change build scripts or configurations can. Whether on a local build or the primary server, this helps ensure that you're not accidentally or intentionally sabotaged by an insider.
What else can you do?
After telling you all the ways you can be compromised, even though knowing them helps avoid it, we wanted to leave you with a couple of things you can proactively do (and if you're already doing them, you can finish this feeling good that you are).
Shift Left: This can relate to multiple things, but here, we mean don't wait until the build to test your code. Run more tests before code gets to the CI/CD pipeline. Tests need access and generate logs, and if those are being run in development environments, they're not providing an additional attack surface at the CI/CD level. Additionally, scan, scan, and scan again. There are multiple points, from a pre-commit hook in a developer environment to scanning the log files and artifacts from a CI/CD run, where you can integrate a secrets scanner like GitGuardian.
Use Vaults: Remember how AWS and GitHub can communicate without you giving GitHub an AWS secret? That doesn't just need to be limited to pushing builds to an EC2 instance or S3. It can also let you pull secrets for other resources from AWS Secrets Manager, so they're never stored in GitHub. You may not use AWS or GitHub, but you can explore how your favorite CI/CD tools and your favorite vault can integrate so you're not storing secrets in your build scripts, build tools, or the artifacts they produce.
We hope you're leaving this with some better understanding of how CI/CD tools can get exploited, how you can protect yourself better, and some ideas for exploring the topic further. To quote the excellent 1980s police drama, "Hill Street Blues": Hey, let's be careful out there.