In community we trust: Open source software and supply chain security

In the United States and around the globe, businesses and organizations have experienced a number of high-profile and costly security attacks over the past few years. And the sobering truth is, the attacks are not going to stop. 

According to Forrester’s report — “The State of Application Security 2021” —30% of external breaches were caused by software vulnerabilities. But as SolarWinds showed, not only are your internal operations disrupted by a breach, but your customers’ lives can be severely disrupted as a result. Even entire supply chains. 

Which is why our collective work on security is so important right now.

It takes a village

In response to these high-profile security breaches and the ongoing need to enhance software security, President Joe Biden issued an executive order in May 2021. The order seeks to improve the nation’s cybersecurity, and has significant implications for U.S. companies.

Though its scope is limited to the United States, it does reflect similar efforts in other regions around the world. The difference being that the U.S. order is far more specific and has tight deadlines. And those will be felt across the open source community.

For a company like Red Hat, we are very interested in Section 4 of the executive order: “Enhancing Software Supply Chain Security.” Of the numerous standards and requirements to be met, here are three that stood out to me. (I’m paraphrasing the main points.)

1. We must use automated tools or processes to maintain the integrity of source code, and to check for known and potential vulnerabilities — and remediate them.

2. Maintain data about audits, origin of software and components, and utilize secure software development practices.

3. Ensure and attest to the integrity and provenance of open source software used within any portion of a product.

The Administration knows it can't do this on its own. As the order states: “The private sector must partner with the federal government to foster a secure cyberspace,” and “in the end, the trust we place in our digital infrastructure should be proportional to how trustworthy and transparent that infrastructure is.”

Transparency. Partnership. Trust. 

Those important words should be familiar to everyone who’s involved in open source communities and software development.

A fragile software supply chain

“The software supply chain is fragile.” That’s according to Forrester’s report: “The Top Security Technology Trends to Watch, 2021.”

This challenge is not breaking news for open source advocates. It’s been said before by my fellow Red Hatters, and it’s worth mentioning again: How do you secure a supply chain for a product that has no physical form, no box to lock, and is created in an environment where anyone can contribute to it?

The open source community includes any number of developers from all over the globe who prefer to contribute anonymously or pseudonymously. So you don’t necessarily have to know by name the other person who created the code, but you need to trust them. 

This is the practical reality of open source that’s at odds with knowing exactly where all your code is coming from, which adds to the potential scope of the challenge facing all of us. 

The relationships that exist in open source communities are unique, if you stop and consider it. People in open source communities are fundamentally dependent on each other to get anything done. And as we all know, the software created by these communities comes with an astounding number of software dependencies. We pull each other’s code all the time, which demonstrates that high level of implicit trust.

That trust creates the collaborative spirit--the bonds that form over time--that powers open source communities. Linux, for example, has evolved from a single developer to a distributed community that maintains software that literally powers the Internet. A more recent example is Kubernetes, which has rapidly become a core piece of infrastructure for many businesses.

What Kubernetes and the Linux kernel have in common, along with many other projects, is dependency on many other pieces of open source software. The Linux kernel relies on a host of programs to compile and run. Kubernetes, likewise, pulls in a number of dependencies and its dependencies have dependencies of their own.

Note that this is not unique to open source. When you run proprietary software it, too, likely has a supply chain behind it of dependencies--some of which may be open source, some of which may not. The difference is that we can trace the dependencies of open source software that we depend on, whereas the bill of materials for a proprietary application are more opaque.

The good news and bad news

So the good news is: Open source software usage is only going to grow. Did you know that almost 99% of audited codebases contain some amount of open source code? That’s according to “The State of Application Security 2021” Forrester report.

And that’s also the bad news: That open source software usage is only going to grow. What I mean is that these increases are going to attract more bad actors and more new threats. In 2021, there was a whopping 650% year-over-year increase in software supply chain attacks aimed at exploiting weaknesses in upstream, open source ecosystems, according to this year’s “State of the Software Supply Chain” report.

The thing is, this critical security work can — and will — be done.

How do I know this? Look at what happened with Red Hat Enterprise Linux. Despite the backdrop of open source supply chain security concerns, 95% of Fortune 500 companies today rely on Red Hat Enterprise Linux. It’s an enterprise Linux operating system that provides a consistent foundation for workloads across environments, certified on hundreds of clouds and with thousands of vendors. It has achieved this in part as a result of mature change controls, and is trusted not just in the U.S. but all over the world. 

So coming back to the three keywords I mentioned earlier: transparency, partnership, trust. That’s how we’ve gotten it done in the past, and that’s how we’ll continue to do it in the future.

Trust and verify your software sources

Because open source means everyone can see what you’re doing, they can also verify and validate your software and security protocols.

One idea to help secure software supply chains lies in digitally signing the artifacts that make up applications, including: software bill of materials, component manifests, dependency trees, and the like. 

As Luke Hinds, security engineering lead in Red Hat's office of the CTO, writes, “Digital signatures effectively ‘freeze’ an object in time. Which indicates that in its current state, the object is verified to be what it says it is, and that it hasn’t been altered in any way. This is great for developers who are pulling lots of code from many sources into complicated workloads.”

Many of today’s digital signing tools are fine for their intended purpose. But they’re often laborious, cumbersome or expensive. And they also don’t fit the model of open source’s innovation engine and automated development processes, such as DevOps and CI/CD.

But there’s also the matter of who holds and protects the cryptographic keys. These keys are critical. For example, in open communities with countless contributors, is everyone supposed to hold a private key? Just the community leads (if they exist)? Or the sponsor? What happens if an individual is compromised, and how do we know IT IS the individual keyholder and not an imposter?

This is the massive challenge facing the software supply chain, especially in open source. 

We’re excited about an open source project originally prototyped at Red Hat and now under the auspices of the Linux Foundation with backing from Red Hat, Google, and others. 

Sigstore offers a method to enhance security for software supply chains in an open, transparent, and accessible manner. It’s intended to make cryptographic signing easier and available to all. And most importantly, it’s available at no cost.

So far, the community counts more than 465 members and 20 organizations, and we’re excited about its future.

Giving more confidence to data scientists

Transparency, partnership and trust for the sources you use in your applications and underlying software stacks are not the whole “trust story.”

Even an application composed completely from trusted sources can introduce new vulnerabilities and needs to be actively checked for new defects and security issues--both in the application itself and its underlying code dependencies. DevSecOps practices and their automated tools have evolved to make this manageable. But we want to do even better.

The goal of Red Hat’s AI DevSecOps initiative is to ease this burden using AI-enabled automation of best practices for software development and life cycle management. Integrations and bots that plug into the CI/CD system help developers define software stacks in a predictable manner. They also automate the task of ongoing maintenance--stack optimization, updates to new versions, tests, and security fixes.

These automation tools are backed by a guidance system that learns from actual developer use, community interactions and automated testing that improves its behavior over time. Our initial implementation of this concept is an open source service called Project Thoth. 

Thoth uses AI to analyze and recommend software stacks for Python applications with an initial focus on data science use cases. Data scientists can use Thoth through a JupyterLab plugin that integrates the guidance directly into their preferred tool as well as automation integrated in the build pipelines. 

This allows data scientists to focus on the actual job — data science — without being burdened by the complexities and security challenges of the underlying software stacks.

The value of an open approach

An “open approach” is important to project security and vulnerability management. Because everything we do at Red Hat is open, you’ll be unsurprised to know that we adhere to an open methodology to vulnerability management.

In fact, we just released a white paper that explains our approach to managing vulnerabilities discovered within components of our software. The paper describes the frameworks, standards, techniques and tools Red Hat employs for our customers.

As part of our open approach, we partner with the community and our customers. And we hope, in the end, the work we do on open-source security creates the necessary trust for everyone we are fortunate to collaborate with.

Transparency. Partnership. Trust.

It’s what’s helped the open source community get to where we are in 2021 and it’s what’s going to help us to continue to move forward — together.