This report takes a look at the state of security risk for Red Hat products for calendar year 2015. We look at key metrics, specific vulnerabilities, and the most common ways users of Red Hat products were affected by security issues.

Our methodology is to look at how many vulnerabilities we addressed and their severity, then look at which issues were of meaningful risk, and which were exploited. All of the data used to create this report is available from public data maintained by Red Hat Product Security.

Red Hat Product Security assigns a Common Vulnerabilities and Exposures (CVE) name to every security issue we fix. If we fix a bug that later turns out to have had a security implication we’ll go back and assign a CVE name to that issue retrospectively. Every CVE fixed has an entry in our public CVE database in the Red Hat Customer Portal as well as a public bug that has more technical detail of the issue. Therefore, for the purposes of this report we will equate vulnerabilities to CVEs.

Note: Vulnerability counts can be used for comparing Red Hat issues within particular products or dates because we apply a consistent methodology on how we allocate names and how we score their severity. You should not use vulnerability count data (such as the number of CVEs addressed) to compare with any other product from another company, because the methodology used to assign and report on vulnerabilities varies. Even products from different vendors that are affected by the same CVE can have variance in the severity of the CVE given the unique way the product is built or integrated.

Vulnerabilities

Across all Red Hat products, and for all issue severities, we fixed more than 1300 vulnerabilities by releasing more than 600 security advisories in 2015. At first that may seem like a lot of vulnerabilities, but for a given user only a subset of those issues will be applicable for the products and versions of the products in use. Even then, within a product such as Red Hat Enterprise Linux, not every package is installed in a default or even likely installation.

Red Hat rates vulnerabilities using a 4 point scale designed to be an at-a-glance guide to the amount of concern Red Hat has for each security issue. This scale is designed to align as closely as possible with similar scales from other open source groups and enterprise vendors, such as Microsoft. The severity levels are designed to help users determine which advisories mattered the most. Providing a prioritised risk assessment helps customers understand and better schedule upgrades to their systems, being able to make a more informed decision about the risk that each issue places on their unique environment.

Since 2009, we also publish Common Vulnerability Scoring System (CVSS) scores for every vulnerability addressed to aid customers who use CVSS scoring for their internal processes. However, CVSS scores have some limitations and we do not use CVSS as a way to prioritise vulnerabilities.

The 4 point scale rates vulnerabilities as Low, Moderate, Important, or Critical.

Vulnerabilities rated Critical in severity can pose the most risk to an organisation. By definition, a Critical vulnerability is one that could potentially be exploited remotely and automatically by a worm. However we, like other vendors, also stretch the definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. These flaws actually account for the majority of the Critical issues fixed as we will show in this report. If you’re using a Red Hat product that does not have a desktop, for example, you’ll be affected by a lot less Critical issues.

The table below gives some examples for advisory and vulnerability counts for a subset of products and product families. A given Red Hat advisory may fix multiple vulnerabilities across multiple versions of a product. Therefore, a count of vulnerabilities can be used as an estimate of the amount of effort in understanding the issues and fixes. A count of advisories can be used as an estimate of the amount of effort to understand and deploy updates.

One product broken out in the table is Red Hat Enterprise Linux 6. During Red Hat Enterprise Linux 6 installation, the user gets a choice of installing either the default selection of packages, or making a custom selection. If the user installs a “default” “server” and does not add any additional packages or layered products, then in 2015 there were just 6 Critical and 19 Important security advisories applicable to that system (and 29 advisories that also addressed moderate/low issues).

Where there are more advisories shown than vulnerabilities (such as for OpenStack), this is because the same vulnerability may affect multiple currently supported versions of the product, each version got it’s own security advisory.

In 2015, for every Red Hat product there were 112 Critical Red Hat security advisories released addressing 373 Critical vulnerabilities. 82% of the Critical issues had updates available to address them the same or next day after the issue was public. 99% of Critical vulnerabilities were addressed within a week of the issue being public.

Looking at just the subset of issues affecting base Red Hat Enterprise Linux releases, there were 46 Critical Red Hat security advisories released addressing 61 Critical vulnerabilities. 96% of the Critical issues had updates available to address them the same or next day after the issue was public.

For Red Hat Enterprise Linux, server installations will generally be affected by far fewer Critical vulnerabilities, just because most Critical vulnerabilities occur in browsers or browser components. A great way to reduce risk when using our modular products is to make sure you install the right variant, and review the package set to remove packages you don’t need.

Vulnerability trending

The number of vulnerabilities addressed by Red Hat year on year is increasing as a function of new products and versions of products being continually added. However, for any given version of a product we find that the number of vulnerabilities being fixed actually decreases over time. This is influenced by Red Hat backporting security fixes.

We use the term backporting to describe the action of taking a fix for a security flaw out of the most recent version of an upstream software package and applying that fix to an older version of the package we distribute. Backporting is common among vendors like Red Hat and is essential to ensuring we can deploy automated updates to customers with minimal risk.

The trends can be investigated using our public data, and from time to time we do Risk Reports that delve into a given product and version. For example see our Red Hat Enterprise Linux 6.5 to 6.6 Risk Report.

What issues really mattered in 2015

In 2014, the OpenSSL Heartbleed vulnerability started a trend of branding vulnerabilities changing the way security vulnerabilities affecting open source software were being reported and perceived. Vulnerabilities are found and fixed all of the time, and just because a vulnerability gets a catchy name, fancy logo, or media attention doesn’t mean it’s of real risk to users.

So let’s take a chronological tour through 2015 to see which issues got branded or media attention, but more importantly which issues actually mattered for Red Hat customers.

“Ghost” (January 2015) CVE-2015-0235

A bug was found affecting certain function calls in the glibc library. A remote attacker that was able to make an application call to an affected function could execute arbitrary code. While a proof of concept exploit is available, as is a Metasploit module targeting Exim, not many applications were found to be vulnerable in a way that would have allowed remote exploitation.

Red Hat Enterprise Linux versions were affected. This was given Critical impact, and updates were available the same day the issue was public. This issue was given enhanced coverage in the Red Hat Customer Portal, with a banner on all pages and a customer outreach email campaign.

“Freak” (March 2015) CVE-2015-0204

A flaw was found in the cryptography library OpenSSL where clients accepted EXPORT-grade (insecure) keys even when the client had not initially asked for them. This could have been exploited using a man-in-the-middle attack, downgrading to a weak key, factorizing it, then decrypting communication between the client and the server. Like the branded OpenSSL issues from 2014 such as Poodle and CCS Injection, this issue is hard to exploit as it requires a man-in-the-middle attack. We’re therefore not aware of active exploitation of this issue.

Red Hat Enterprise Linux versions were affected. This was given Moderate impact, and updates were available within a few weeks of the issue being public.

ABRT (April 2015) CVE-2015-3315

ABRT (Automatic Bug Reporting Tool) is a tool to help users detect defects in applications and create a bug report. ABRT was vulnerable to multiple race condition and symbolic link flaws. A local attacker could have used these flaws to potentially escalate their privileges on an affected system to root.

This issue affected Red Hat Enterprise Linux 7. This was given Important impact, and updates were made available. Other products and versions of Red Hat Enterprise Linux were either not affected, or not vulnerable to privilege escalation. A working public exploit is available for this issue.

JBoss Operations Network open APIs (April 2015) CVE-2015-0297

Red Hat JBoss Operations Network is a middleware management solution that provides a single point of control to deploy, manage, and monitor JBoss Enterprise Middleware, applications, and services. The JBoss Operations Network server did not correctly restrict access to certain remote APIs which could have allowed a remote, unauthenticated attacker to execute arbitrary Java methods. We’re not aware of active exploitation of this issue.

This issue affected versions of JBoss Operations Network. It was given Critical impact, and updates were made available within a week of the issue being public.

“Venom” (May 2015) CVE-2015-3456

Venom was a branded flaw which affected QEMU. A privileged user of a guest virtual machine could use this flaw to crash the guest or, potentially, execute arbitrary code on the host with the privileges of the host’s QEMU process corresponding to the guest.

A number of Red Hat products were affected and updates were released the same day as the issue was public. Red Hat products by default would block arbitrary code execution as SELinux sVirt protection confines each QEMU process. We therefore are not aware of any exploitation of this issue.

This issue was given enhanced coverage in the Red Hat Customer Portal, with a banner on all pages and a customer outreach email campaign.

“LogJam” (May 2015) CVE-2015-4000

TLS connections using the Diffie-Hellman key exchange protocol were found to be vulnerable to an attack in which a man-in-the-middle attacker could downgrade vulnerable TLS connections to weak cryptography which could then be broken to decrypt the connection.

This issue affected various cryptographic libraries across several Red Hat products. It was rated Moderate impact and updates were made available.

Like Poodle and Freak, this issue is hard to exploit as it requires a man-in-the-middle attack. We’re not aware of active exploitation of this issue.

libuser privilege escalation (July 2015) CVE-2015-3246

The libuser library implements an interface for manipulating and administering user and group accounts. Flaws in libuser could allow authenticated local users with shell access to escalate privileges to root.

Red Hat Enterprise Linux 6 and 7 were affected. This issue was rated Important impact, and updates were made available the same day as issue was made public. Red Hat Enterprise Linux 5 was affected and a mitigation was published. A public exploit exists for this issue.

BIND DoS (July 2015) CVE-2015-5477

A flaw in the Berkeley Internet Name Domain (BIND) allowed a remote attacker to cause named (functioning as an authoritative DNS server or a DNS resolver) to exit, causing a denial of service against BIND.

This issue affected the versions of BIND shipped with all versions of Red Hat Enterprise Linux. This issue was rated Important impact, and updates were available the same day as the issue was made public. A public exploit and a Metasploit module exist for this issue.

Several other similar flaws in BIND leading to denial of service were found and addressed through the year, such as CVE-2015-8704, CVE-2015-8000, and CVE-2015-5722. Public exploits exist for some of these issues.

Firefox local file stealing via PDF reader (August 2015) CVE-2015-4495

A flaw in Mozilla Firefox could allow an attacker to access local files with the permissions of the user running Firefox. Public exploits exist for this issue, including part of Metasploit, and specifically targeting Linux systems.

This issue affected Firefox that was shipped with versions of Red Hat Enterprise Linux. It was rated Important impact, and updates were available the following day after the issue was public.

Firefox add-on permission warning (August 2015) CVE-2015-4498

Mozilla Firefox normally warns a user when trying to install an add-on if initiated by a web page. A flaw allowed this dialog to be bypassed. We’re not aware that this issue has been exploited.

This issue affected Firefox shipped with Red Hat Enterprise Linux versions. It was rated Important impact, and updates were available the same day as the issue was public.

Java Deserialization (November 2015) CVE-2015-7501

An issue was found in Java Object Serialization affecting the JMXInvokerServlet interface. This could lead to arbitrary code execution when deserializing Java objects from untrusted sources with the Apache commons-collections library when containing certain risky classes on the classpath.

This issue impacted many products in the JBoss Middleware suite and updates were made available in November and the following months. Direct exploitation of this vulnerability requires some means of getting an application to accept an object containing one of the risky classes.

Grub2 password bypass (December 2015) CVE-2015-8370

A flaw was found in the way the grub2 handled backspace characters entered in username and password prompts. An attacker with access to the system console could use this flaw to bypass grub2 password protection.

This issue only affected Red Hat Enterprise Linux 7. It was rated Moderate severity, and updates were made available within a week. Steps on how to exploit this issue are public.

Various flaws in software in supplementary channels (various dates)

Red Hat provides some packages which are not open source software in supplementary channels for users of Red Hat Enterprise Linux. This channel contains software such as Adobe Flash Player, IBM Java, Oracle Java, and Chromium browser.

A large number of Critical flaws affected these packages. For example, for Adobe Flash Player in 2015, we issued 15 Critical advisories to address nearly 300 Critical vulnerabilities. Linux exploits exist for some of these critical vulnerabilities, 5 having Metasploit modules. As these projects release security updates, we ship appropriate updated packages to customers.

The issues examined in this section were included because they were meaningful. This includes the issues that are of high severity and likely to be exploited (or already have a public working exploit), as well as issues that were highly visible or branded (with a name or logo or enhanced media attention), regardless of their severity. See the Venn diagram below for our opinion on the intersection.

Lower risk issues with increased customer attention

Another way we gauge the level of customer concern around an issue is to measure web traffic, specifically how many page views each of the vulnerability (CVE) pages gets in the Red Hat Customer Portal.

The graph above gives an indication of customer interest in given vulnerabilities. Many of the top issues were highlighted earlier in this report. Of the rest, the top viewed issues were ones predominantly affecting Red Hat Enterprise Linux:

• A flaw in Samba, CVE-2015-0240, where a remote attacker could potentially execute arbitrary code as root. Samba servers are likely to be internal and not exposed to the internet, limiting the attack surface. No exploits that lead to code execution are known to exist, and some analyses have shown that creation of such a working exploit is unlikely.
• Various flaws in OpenSSL, After high profile issues such as Heartbleed and Poodle in previous years, OpenSSL issues tend to always get increased customer interest independent of the actual severity or risk:
  • Two issues, CVE-2015-1793 and CVE-2015-0291 did not affect Red Hat products. A Metasploit module exists for CVE-2015-1793.
  • Three issues: CVE-2015-0286, CVE-2015-3195, and CVE-2015-3194 all had Moderate or Low impact for Red Hat products.
• Two flaws in OpenSSH, of which one, CVE-2015-5600 did not affect Red Hat products in a default configuration and was rated Low impact; CVE-2015-5352 which affected some versions of Red Hat Enterprise Linux but at Moderate impact.
• Two flaws in the Red Hat Enterprise Linux kernel, both rated Important impact; CVE-2015-1805 that could allow a local attacker to escalate their privileges to root; CVE-2015-5364 where a remote attacker who is able to send UDP packages to a listening server could cause it to crash. We are not aware of public exploits for either issue.
• A Moderate rated flaw in the Apache web server CVE-2015-3183 which could lead to proxy smuggling attacks. We are not aware of a public exploit for this issue.

The open source supply chain

Red Hat products are based on open source software. Some Red Hat products contain several thousand individual packages, each of which is based on separate, third-party, software from upstream. While Red Hat engineers play a part in many upstream components, handling and managing vulnerabilities across thousands of third-party components is non-trivial.

Red Hat has a dedicated Product Security team who monitor issues affecting Red Hat products and work closely in relationships with upstream projects. In 2015, more than 2000 vulnerabilities were investigated that potentially affected parts of our products, leading to fixing 1363 vulnerabilities.

Every one of those 2000+ vulnerabilities is tracked in the Red Hat Bugzilla tool and is publicly accessible. Each vulnerability has a master bug including the CVE name as an alias and a “whiteboard” field which contains a comma separated list of metadata. The metadata we publish includes the dates we found out about the issue, the severity, and the source. We also summarise this in a file containing all of the information gathered for every CVE, as well as a readable entry in the CVE database in the Red Hat Customer Portal.

For example, for CVE-2015-0297 mentioned above:

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2015-0297

Whiteboard: impact=critical,public=20150414,reported=20150220,
source=customer,cvss2=7.5/AV:N/AC:L/Au:N/C:P/I:P/A:P,
cwe=CWE-306,jon-3/Security=affected

This example shows us the issue was reported to Red Hat Product Security by a customer on February 20, 2015, the issue became known to the public on April 14, 2015, and it affected the JBoss Operations Network 3 product. An automated comment in the Bugzilla shows an errata was released to address this on April 21, 2015 as RHSA-2015:0862.

Issues that are not yet public still get an entry in Bugzilla, but they are initially private to Red Hat. Once an issue becomes public, the associated Bugzilla is updated and made public.

We make use of this data to create metrics and spot trends. One interesting metric is to look at how vulnerabilities are reported to us. We can do this by looking at the whiteboard “source” data to see how we found out about all the issues we fixed in 2015. This is shown on the chart below.

Key:

• Internet: for issues not disclosed in advance, we monitor a number of mailing lists and security web pages of upstream projects.
• Relationship: issues reported to us by upstream projects, generally in advance of public disclosure.
• Red Hat: issues found by Red Hat employees.
• Individual: issues reported to Red Hat Product Security directly by a customer or researcher.
• Peer vendors: issues reported to us by other open source distributions, through relationships or a shared private forum.
• CVE: if we haven’t found out about an issue any other way, we can catch it from the list of public assigned CVE names from Mitre.
• CERT: issues reported to us from a national Computer Emergency Response Team like CERT/CC or CPNI.

We can make some observations from this data. First, Red Hat employees find a lot of the vulnerabilities we fix. We don’t take a passive role and wait for others to find flaws for us to fix. We actively look for issues ourselves and these are found by engineering, quality assurance, as well as our security teams. 12% of the issues we fixed in the year were found by Red Hat employees. The issues we find are shared back upstream and if they are risky, under embargo to other peer vendors (generally via the ‘distros’ shared private forum). In addition to those 167 issues, Red Hat also finds and reports flaws in software that isn’t part of a current shipped product or affects other vendors’ software.

Next, relationships matter. When you are fixing vulnerabilities in third-party software, having a relationship with the upstream community makes a big difference. Red Hat Product Security are often asked how to get notified of issues in open source software in advance, but there is no single place you can go to get notifications. If an upstream is willing to give information about flaws in advance, then you should also be willing to give value back to that notification, making it a two-way street. At Red Hat we do this by sanity checking draft advisories, checking patches, and feeding back the results from our quality testing when there is enough time. A good example of this is the OpenSSL CCS Injection flaw in 2014. Our relationship with OpenSSL gave us advance notice of the issue. We found a mistake in the advisory as well as a mistake in the patch, which otherwise would have caused OpenSSL to have to do a secondary fix after release. Only two of the dozens of companies pre-notified about those OpenSSL vulnerabilities noticed issues and fed back information to upstream.

Finally, it’s non-trivial to replicate this yourself. If you are an organization that uses open source software that you manage yourself, then you need to ensure you are able to find out about vulnerabilities that affect those components so you can analyse and remediate. Vendors without a sizable dedicated security team have to watch what other vendors do, or rely on other vulnerability feeds such as the list of assigned CVE names from Mitre. Red Hat chooses to invest in a dedicated team handling vulnerability notifications to ensure we find out about issues that affect our products and build upstream relationships.

Embargo and release timings

Vulnerabilities known to Red Hat in advance of being public are known as being “under embargo”, mirroring the way journalists use the term for stories under a press embargo which are not to be made public until an agreed date and time.

The component parts that make up Red Hat products are open source, and this means we’re in most cases not the only vendor shipping each particular part. Unlike companies shipping proprietary software, Red Hat therefore is not in sole control of the date each flaw is made public. This is actually a good thing and leads to much shorter response times between flaws being first reported to being made public. It also keeps us honest; Red Hat can’t play games to artificially reduce our “days of risk” statistics by using tactics such as holding off public disclosure of meaningful flaws for a long period, or until some regularly scheduled patch day.

Shorter embargo periods also make flaws much less valuable to attackers; they know a flaw in open source is likely to get fixed quickly, shortening their window of opportunity to exploit it.

For the issues found by Red Hat, we choose to only embargo the issues that really matter and even then we use embargoes sparingly. Bringing in additional security experts, who would not normally be aware due to the embargo, rather than just the original researcher and the upstream project, increases the chances of the issue being properly understood and patched the first time around. For the majority of lower severity issues, attackers have little to no interest in them. By definition, these are issues that lead to minimal consequences even if they are exploitable, so the cost of embargoes is not justified. If we do choose to embargo an issue due to the severity, we share the details with the relevant upstream developers as well as other peer vendors, working together to address the issues. We talk about this more in our blog post “The hidden costs of embargos“.

For 2015, we knew about 438 (32%) of the vulnerabilities we addressed in advance of them being public. Across all products and vulnerabilities of all severities known to us in advance, the median embargo was 13 days.

There are many positives to releasing fixes for issues that matter quickly, but the drawback to not having a regular patch day is that you need to respond to more issues as they happen. We do help suggest embargo dates that avoid weekends and major holidays, so let’s look how well that works in practice.

The chart above shows a heat-map for 2015 with the days and times we push most issues for Critical and Important advisories for all Red Hat products. The more advisories pushed for a given date and hour, the darker that section of the heat-map.

The most popular times we pushed advisories can be seen as Tuesdays 11 a.m. to 2 p.m. EST and Thursdays 9 a.m. to 3 p.m. EST. Fridays are pretty light for pushes. There were no Saturday pushes. The only Sunday pushes were ones arranged to arrive first thing Monday morning (these are usually pushed during Monday in India or Europe time zones).

Conclusion

This report looked at the security risk to users of Red Hat products in 2015 by giving metrics around vulnerabilities, highlighting those that were the most severe, looking at threat, those that were exploited, and showing which were branded or gained media attention.

There are other types of security risks, such as malware or ransomware, that we haven’t covered in this report. They rely on an attacker having access to a system through an intrusion or by exploiting a vulnerability.

For the last year of vulnerabilities affecting Red Hat products the issues that matter and the issues that got branded do have an overlap, but they certainly don’t closely match. Just because an issue gets given a name, a logo, or press attention does not mean it’s of increased risk. We’ve also shown there were some vulnerabilities of increased risk that did not get branded or media attention at all.

At Red Hat, our dedicated Product Security team analyses threats and vulnerabilities against all of our products every day, and provide relevant advice and updates through the Red Hat Customer Portal. Customers can call on this expertise to ensure that they respond quickly to address the issues that matter, while avoiding being caught up in a media whirlwind for those that don’t.

Appendix: Common security abbreviations and terms

Acronyms are used extensively in security standards, so here are some of the more common terms and abbreviations you’ll see used by Red Hat relating to vulnerability handling and errata. You can find more in this blog post.

CVE:

The Common Vulnerabilities and Exposures (CVE) project is a list of standardized names for vulnerabilities and security exposures.

Since November 2001 Red Hat has used CVE names in security advisories to describe all vulnerabilities affecting Red Hat products. Red Hat has CVE Editorial Board membership and is a Candidate Naming Authority. We have a public CVE compatibility page and provide a CVE database in the Red Hat Customer Portal.

CVRF:

The goal of the Common Vulnerability Reporting Framework (CVRF) is to provide a way to share information about security updates in an XML machine-readable format.

Since 2012, Red Hat has provided CVRF representations of Red Hat Security Advisories, and details can be found in this page.

OVAL:

The Open Vulnerability and Assessment Language (OVAL) project promotes open and publicly available security content, and seeks to standardize the transfer of this information across the entire spectrum of security tools and services.

Since 2006, Red Hat has been providing machine-readable XML versions of our Red Hat Enterprise Linux security advisories as OVAL definitions. Our OVAL definitions are designed for use by automated test tools to determine the patch state of a machine.

Red Hat provides OVAL patch definitions for security updates to Red Hat Enterprise Linux 4, 5, 6, and 7. The first OVAL-compatible version was Red Hat Enterprise Linux 3, for which OVAL patch definitions continue to be available for download. For more information read this page.

RHSA:

Since 1999, all Red Hat security updates are accompanied by a security advisory (RHSA). The advisories are publicly available via the Red Hat Customer Portal as well as other notification methods such as email. These are sometimes also referred to as security errata. The other advisory types are Red Hat Bugfix Advisory (RHBA) and Red Hat Enhancement Advisory (RHEA).

CVSS:

Common Vulnerability Scoring System (CVSS) base scores give a detailed severity rating by scoring the constant aspects of a vulnerability: Access Vector, Access Complexity, Authentication, Confidentiality, Integrity, and Availability.

Since 2009, Red Hat provides CVSS version 2 base metrics for all vulnerabilities affecting Red Hat products. These scores are found on the CVE pages (linked to from the References section of each Red Hat Security Advisory) and also from our Security Measurements page.

CVSS scores are not used by Red Hat to determine the priority with which flaws are fixed. It is used as a guideline to identify key metrics of a flaw, but the priority for which flaws are fixed is determined by the overall impact of the flaw using the aforementioned 4 point scale.

CWE:

Common Weakness Enumeration (CWE) is a dictionary or formal list of common software weaknesses. It is a common language or taxonomy for describing vulnerabilities and weaknesses; a standard measurement for software assurance tools and services’ capabilities; and a base for software vulnerability and weakness identification, mitigation, and prevention.
The Red Hat Customer Portal is officially CWE Compatible.

CPE:

CPE is a structured naming scheme for information technology systems, software, and packages. For reference, we provide a dictionary mapping the CPE names we use, to Red Hat product descriptions. Some of these CPE names will be for new products that are not in the official CPE dictionary, and should therefore be treated as temporary CPE names.