Security vulnerabilities, like most things, go through a life cycle from discovery to installation of a fix on an affected system. Red Hat devotes many hours a day to combing through code, researching vulnerabilities, working with the community, and testing fixes–often before customers even know a problem exists.
Discovery
When a vulnerability is discovered, Red Hat engineers go to work verifying the vulnerability and rating it to determine it’s overall impact to a system. This is a most important step as mis-identifying the risk could lead to a partial fix and leave systems vulnerable to a variation of the original problem. This also allows prioritization of fixes so that those issues with the greatest risk to customers are handled first and issues of low or minimal risk are not passed on to customers who also need to invest time in validating new packages for their environment.
Research
Many times a vulnerability is discovered outside of Red Hat’s domain. This means that the vulnerability must be researched and reproduced in-house to fully understand the risk involved. Sometimes reproducing a vulnerability leads to discovering other vulnerabilities which need fixes or re-engineering.
Notification
When a vulnerability has been discovered, Red Hat works with upstream developers to develop and ship a patch that fixes the problem. A CVE assignment will be made that records the vulnerability and links the problem with the fix among all applicable implementations. Sometimes the vulnerability is embedded in other software and that host software would acquire the CVE. This CVE is also used by other vendors or projects that ship the same package with the same code—CVEs assigned to software Red Hat ships are not necessarily Red Hat specific.
Patch development
One of the most difficult parts of the process is the development of the fix. This fix must remedy the vulnerability completely while not introducing any other problems along the way. Red Hat reviews all patches to verify it fixes the underlying vulnerability while also checking for regressions. Sometimes Red Hat must come up with our own patches to fix a vulnerability. When this happens, we fix not only our shipped software, but also provide this fix back upstream for possible inclusion into the master software repository. In other cases, the upstream patch is not applicable because the version of the software we ship is older, and in these cases Red Hat has to backport the patch to the version we do ship. This allows us to minimize any changes exclusively to those required to fix the flaw without introducing and possible regressions or API/ABI changes that could have an impact on our customers.
Quality assurance
As important as patch development, Red Hat’s QE teams validate the vulnerability fix and also check for regressions. This step can take a significant amount of time and effort depending on the package, but any potential delays introduced due to the quality assurance effort is worth it as it significantly reduces any possible risk that the security fix may be incomplete or introduces other regressions or incompatibilities. Red Hat takes the delivery of security fixes seriously and we want to ensure that we get it right the first time as the overhead of re-delivering a fix, not to mention the additional effort by customers to re-validate a secondary fix, can be costly.
Documentation
To make understanding flaws easier, Red Hat spends time to document what the flaw is and what it can do. This documentation is used to describe flaws in the errata that is released and in our public CVE pages. Having descriptions of issues that are easier to understand than developer comments in patches is important to customers who want to know what the flaw is and what it can do. This allows customers to properly assess the impact of the issue to their own environment. A single flaw may have much different exposure and impact to different customers and different environments, and properly-described issues allow customers to make appropriate decisions on how, when, and if the fix will be deployed in their own environment.
Patch shipment
Once a fix has made it through the engineering and verification processes, it is time to send it to the customers. At the same time the fixes are made available in the repositories, a Red Hat Security Advisory (RHSA) is published and customers are notified using the rhsa-announce list. The RHSA will provide information on the vulnerability and will point to errata that more thoroughly explain the fix.
Customers will begin to see updates available on their system almost immediately.
Follow-on support
Sometimes questions arise when security vulnerabilities are made public. Red Hat customers have access to our technical support team that help support all Red Hat products. Not only can they answer questions, but they can also help customers apply fixes.
Conclusion
Handling security issues properly is a complex process that involves a number of people and steps. Ensuring these steps are dealt with correctly and all issues are properly prioritized is one of the things Red Hat provides with each subscription. The level of expertise required to properly handle security issues can be quite high. Red Hat has a team of talented individuals who worry about these things so you don’t have to.