If you’re looking to improve the security posture of your Kubernetes applications, you can get a lot of bang for your buck with vulnerability scanning. In this blog, I’ll talk about the fundamentals of scanning container images: how to pick a vulnerability scanner, when to use it in the application pipeline, and why you should automate this process.
Why is vulnerability scanning important?
Most application software relies heavily on open source packages, libraries, and other third party components – a vulnerability in any one of these dependencies means you’re likely affected too. So, the chances are high that at least some of the applications that you’re running today include vulnerabilities.
A common way for attackers to target deployments is to exploit known vulnerabilities in common dependency code, so you need tools to spot these vulnerable dependencies. Automating scanning into the lifecycle of a container image that’s being deployed into Kubernetes can help reduce the potential attack surface and stop attackers from stealing data or tampering with your deployment.
What to look for in an image scanner
The job of a container image scanner is to look at all the software in an image and report on any vulnerabilities it discovers. Images typically are based on a Linux distribution such as Ubuntu, Amazon Linux, or Alpine. Across your organization you might have different images based on different distributions, so your scanner will need to support all the distributions your images are using.
As each Linux distribution has its own approach to handling security patches, the vulnerability scanner you choose should know about vulnerability information not just from the National Vulnerability Database (NVD), but also from any distribution-specific security advisories.
Since software typically is installed into the image at build time using a package manager, your scanner also needs to support the package manager so that it can find the version information. And because vulnerabilities can also exist in the language-specific dependencies and libraries that your application uses, language support is another important feature to look for in a scanner.
Automating vulnerability scanning
While you can run a scan manually, in practice you’ll want to automate scanning so that you know it’s being done regularly on all images, and so that it doesn’t slow down shipping new functionality for your business. There are several stages during the lifecycle of your application images where you need to do this.
- In the build
- In the registry
- Before deployment
- In running Kubernetes workloads
Scanning in the build
To start, you should run vulnerability scanning as a build step, just like you might run a test as part of the build pipeline. If a vulnerability is discovered, the build step would fail. The container image wouldn’t get stored in the registry, and there would be no chance of deploying the vulnerable image.
Below I show an example of Trivy, Aqua’s open source image scanner, built into AWS CodeBuild – it’s set up to fail if the scan reveals any high or critical severity vulnerabilities. In this example it did find one, so the build step failed.
High-severity vulnerability fails the build
Scanning in the registry
Security researchers find new vulnerabilities in existing software all the time, so it’s possible that a new vulnerability might get discovered in packages that are included in your image after it has been built. For this reason, many image registries support regularly scanning the images that they store. The major public cloud providers all provide some built-in scanning in their registries, and CNCF’s Harbor registry integrates Trivy as the default scanner. With these built-in scanners it’s as simple as flicking a switch for the image repository and viewing the report in the console. Commercial container security solutions like Aqua integrate into all the major registries out of the box so that they’re practically as easy to enable, and give you a choice if the built-in scanner isn’t suitable for your needs. We’ll come to the reasons why you might want to pick a specific scanner later in this article.
Scanning before deployment
The next place where you need to automate a check for image vulnerabilities is at the point where you deploy the image. Kubernetes provides a nice opportunity to do this by implementing an admission webhook that gets called whenever a new workload is deployed. You could use a solution like Open Policy Agent, or implement a validating webhook as a Lambda function that determines whether a pod is OK to deploy or not, according to some set of policies that you define. This is also another case where commercial solutions like Aqua can take care of these policy checks for you.
Scanning in running workloads
Let’s suppose your images passed these checks and were deployed successfully into the cluster. What if you want to check for known vulnerabilities in your running workloads?
Our open source Starboard project integrates vulnerability scanning (and other security checks) into the Kubernetes experience, making security information accessible over the same Kubernetes interface. We’ve defined some custom resource definitions (CRDs) so you can store security information in Kubernetes and access it over the same API.
Starboard enables you to trigger security scan jobs for vulnerabilities, and for other types of security reports, through the kubectl command line. The security reports are associated with the resources they apply to — so, for example, if you do vulnerability scanning for an unmanaged pod, the vulnerability report for that pod is owned by that pod. We have also added Starboard extensions to some Kubernetes dashboards like Lens and Octant, so that you can easily view the vulnerability information alongside the workload it is associated with.
We’ve also added a Starboard Operator that watches for new pods and automatically triggers vulnerability scans for them, so that as new workloads get deployed, you can easily see the vulnerability information associated with them, using kubectl or one of the Kubernetes dashboard tools.
Choosing a vulnerability scanner
There are several different vulnerability scanners available, including open source, free and commercial offerings. So how do you choose the scanner that will best suit your needs?
The most important thing to ensure is that the scanner you choose supports the Linux distributions that you use in your images, and the programming languages that you employ. If you don’t have support for the right Linux distro, you’ll likely see lots of false positives because the scanner won’t know about the right patches.
Your scanner should update its vulnerability database frequently — at least daily — so that you can pick up right away on newly discovered vulnerabilities. And, of course, you’ll get more value from your scanner if it’s easy to use and easy to integrate into your existing pipeline and tools.
Our open source tool Trivy has comprehensive vulnerability tracking across both OS packages and language-specific dependencies for a wide range of Linux distributions. The vulnerability database is updated twice daily, and it is designed to be easy to use and efficient to run in your CI pipeline If you’re looking to start scanning your container images, why not try Trivy?
