Category Archives: Security

Security

Reactive Product Security at Red Hat

The goal of Product Security at Red Hat is “to help protect customers from meaningful security concerns when using Red Hat products and services.” What does that really mean and how do we go about it? In this blog, we take a look at how Red Hat handles security vulnerabilities and what we do to reduce risk to our customers.

In 2001, we founded a dedicated security team within Red Hat to handle product security. Back then, we really had just one product line, the Red Hat® Linux® distribution. Now, 14 years later, we support well over 100 different products and versions from Red Hat Enterprise Linux, to OpenStack®, to Docker. In addition to handling the reaction to vulnerabilities found in our products, we also proactively work on improving security for the future. An upcoming blog post will highlight some of those activities.

All software, no matter what the license, provenance, or supply-chain involved, have bugs–mistakes in the code which introduce errors. Some of those errors may cause a program to behave differently than what is expected, others may cause a program to crash. Of these errors, a small proportion are classified as vulnerabilities if they pose a security risk where an attacker can deliberately cause a program to fail.

Our products are generally made up of many different open source components; for example, Red Hat Enterprise Linux 7 is composed of several thousand different packages and each one can be a separate open source project. Red Hat Product Security is accountable for knowing every component used in every product so we can keep track of the security issues. This has become an area of expertise for us and is recognized by the industry as handling vulnerabilities in third party software is not a trivial task.

It all starts with a team which monitors a number of sources to find out about security issues in such third party components. In a previous blog post, we gave some metrics for a years worth of vulnerabilities and showed that in nearly half of the vulnerabilities we fixed, we were aware in advance of the issue being made public. The biggest source of information regarding non-public issues was through the two-way relationships we have with upstream open source projects and our peer vendors. Additionally, 17% of all the issues we fixed were found internally by Red Hat through security audits by our Quality Engineering team or by the product engineering teams themselves.

The next step is to assess whether these issues affect any of our products and determine the severity of each one. We do this based on a technical assessment from our team of skilled researchers. In addition to the nature of the vulnerability itself and the types of exploits likely to operate against it, other considerations include which specific pieces of code are impacted, the sensitivity of the applications they support, and their potential degree of exposure. For any given a vulnerability in an Open Source component, different products across different vendors could be affected in different ways depending on the versions being used, what patches are included, and even how the package is compiled.

In order to manage this workload, the Product Security team makes use of a number of tools and workflow processes all built around the principles of GTD.

Depending on the severity of the issue and the life-cycle for the product, patches get created and updates prepared. For many of our products, our policy is to back-port fixes an approach that significantly reduces the potential for compatibility issues and the introduction of additional vulnerabilities while making it easier for customers to consume updates. These updates, together with our advisory text explaining the issue, make their way to customers as security errata.

We actively monitor the time it takes for vulnerabilities to pass through this entire process. For example, Red Hat Enterprise Linux 5, since its release in 2007, has had 98% of all critical flaw fixes available to customers either the same day or next calendar day, once the issue was known to the public. We make all of our data on this available so customers can determine metrics for their particular environment.

In practice, what this means is that a Red Hat subscription provides customers with guidance, stability, and security that can only come from Red Hat. For a given product, there is a single mechanism to get updates for security issues across all components and technologies included, no matter which open source project they came from. Products are supported with long life cycles, and we maintain security updates for open source components included even beyond their upstream end of life.

We’ve briefly shown that we have well-established processes to effectively manage vulnerabilities in open source software, and that we are effective in getting fixes for these issues to customers, but there’s more that we do on the reactive side of handling security events.

2014 will be remembered for a number of high profile vulnerabilities, including several in widely used open source components: Heartbleed, ShellShock, and Poodle. Where these affected Red Hat products, we provided fast updates to correct them. However, getting fast fixes out was only part of the value.

In September last year, serious issues were found in the UNIX-like shell, Bash, called ShellShock. During this incident, Red Hat customers also received:

  • Timely advice. By the time the issue went public, we had specific knowledge base articles on the Customer Portal explaining how products were affected, how to get and install the fixes, and how to determine if you were vulnerable to the issue. Our article, linked above, was the definitive source of information about the vulnerability–being cited by most news articles, Wikipedia, and even US-CERT. The knowledge base and blog were continually updated with the latest knowledge and best practices.
  • Industry-leading security expertise. After the original flaw in Bash was identified and fixed, a second issue was discovered in public. It was a Red Hat Product Security engineer who designed and wrote the comprehensive patch used by most vendors in fully addressing this issue.
  • Immediate support. The Red Hat Customer Portal had an alert on every page, with notifications, and our support staff had access to the technical information. We were ready to provide immediate support to customers.
  • Proactive notifications. For customers with products affected by the issue, we sent email notifications within the first few hours. This email provided a call to action and linked to our specific knowledge and fixes for this issue. Posts on our Red Hat Support social media channels also directed customers to our knowledge base articles and fixes.
  • A self-detection tool: We also released a self-detection tool via Red Hat Access Labs to allow customers to easily identify whether their environment was vulnerable.

We’d like customers to hear about these major security issues from us first and then be able to install the fix for the issue. When a significant security event occurs, customers can can come to Red Hat first, safe in the knowledge that we’ll be on top of the situation and be able to give specific, timely, calm, and technically-accurate advice on how the issues affect all of our products and services.

Frequently Asked Questions about the Shellshock Bash flaws

The recent few days have been hectic for everyone who works in the Linux/Unix world. Bash security flaws have rocked the globe leaving people confused, worried, or just frustrated. Now that the storm is over and patches are available for most operating systems, here are the answers to some of the common questions we’ve been asked:

Why are there four CVE assignments?

The original flaw in Bash was assigned CVE-2014-6271. Shortly after that issue went public a researcher found a similar flaw that wasn’t blocked by the first fix and this was assigned CVE-2014-7169. Later, Red Hat Product Security researcher Florian Weimer found additional problems and they were assigned CVE-2014-7186 and CVE-2014-7187. It’s possible that other issues will be found in the future and assigned a CVE designator even if they are blocked by the existing patches.

Is CVE-2014-7169 the same severity as the original flaw?

Our research, and that of others, shows that it would not have been possible to exploit the CVE-2014-7169 flaw remotely in the same way that it was for the previous flaw. So, even though there were security consequences of the CVE-2014-7169 flaw, it was certainly not as severe as the original flaw.

Why did Red Hat delay in providing a patch for CVE-2014-7169?

When a second issue with Bash was found a few minutes after the first one went public, we knew there was something wrong. We could have followed a duct-tape approach and issued patches to our customers quickly or we could have done this correctly. Applying multiple security updates is extremely difficult!

When CVE-2014-7169 went public, there was a lot of visible confusion around how to address this issue. This was fueled by the media and by the fact that exploits were immediately available on the Internet.

Red Hat carefully analyzed the root cause of the issue and wrote and tested patches. We posted these patches to the community for review and allowing everyone to freely use them if they wanted to. Doing things correctly takes time!

Why is Red Hat using a different patch then others?

Our patch addresses the CVE-2014-7169 issue in a much better way than the upstream patch, we wanted to make sure the issue was properly dealt with.

I have deployed web application filters to block CVE-2014-6271. Are these filters also effective against the subsequent flaws?

If configured properly and applied to all relevant places, the “() {” signature will work against these additional flaws.

Does SELinux help protect against this flaw?

SELinux can help reduce the impact of some of the exploits for this issue. SELinux guru Dan Walsh has written about this in depth in his blog.

Are you aware of any new ways to exploit this issue?

Within a few hours of the first issue being public (CVE-2014-6271), various exploits were seen live, they attacked the services we identified at risk in our first post: from dhclient, CGI serving web servers, sshd+ForceCommand configuration, git repositories. We did not see any exploits which were targeted at servers which had the first issue fixed, but were affected by the second issue. We are currently not aware of any exploits which target bash packages which have both CVE patches applied.

Why wasn’t this flaw noticed sooner?

The flaws in Bash were in a quite obscure feature that was rarely used; it is not surprising that this code had not been given much attention. When the first flaw was discovered it was reported responsibly to vendors who worked over a period of under 2 weeks to address the issue.

Did you have an outage?

Our security blog article was widely regarded as the definitive source of information about the flaw, being referenced in news articles, on Wikipedia, and from organizations such as US-CERT. This caused more demand than we expected so we did have some periods on Thursday where the blog was unavailable. Our customer portal also had some problems keeping up with demand at times. Many of the issues we saw have already been corrected.

Is my lightbulb really affected by these flaws?

Only if your lightbulb runs Bash! Lots of press have latched onto the fact that this flaw could affect the Internet Of Things, allowing attackers to take control of your systems via home appliances. In reality, embedded devices rarely use Bash, going for more lightweight solutions such as BusyBox, which includes the ash shell that was not vulnerable to these issues. So while it’s certainly plausible that some devices may be affected by this flaw, it won’t be very common.