Cisco’s Process for Fixed Software Release and Vulnerability Disclosure

This blog was collaboratively written by Lou Ronnau, Scott Bradley, and Dan Maunz on the Cisco Customer Assurance Security Programs (CASP) team.

One of Cisco’s guiding principles is to protect the security of our customers’ networks, and our policies related to vulnerabilities in our products and services are a reflection of those principles. Cisco is committed to avoiding security issues in our products and to handling issues professionally when they arise. To minimize risk associated with vulnerabilities, Cisco employs a well-established and trusted process to disclose vulnerabilities as soon as possible, while taking every effort to minimize the overall impact to customers’ network operations.

Cisco’s Security Vulnerability Policy and disclosure practices align with the Responsible Disclosure model. This is a model in which a vulnerability or software defect that has security implications is disclosed only after the period of time that is necessary for the issue to be patched or mitigated across all affected releases/versions. We believe that it is important to provide customers with a patch or mitigation strategy to secure their networks as soon as there is public disclosure of the issue. This is in contrast to the full disclosure model, which discloses vulnerabilities immediately.

Read more about Cisco’s Responsible Disclosure & Security Vulnerability Policies

Cisco’s Product Security Incident Response Team (PSIRT) frequently receives questions from customers about why fixed code has been available for several months before the vulnerability itself is publicly disclosed.

Before Cisco can responsibly disclose a vulnerability with a “High” or “Critical” severity rating, fixed code for all affected versions must be available. This ensures that fixes are delivered to all our customers at the same time, even if they are running different releases of an affected version. However, the fixed versions themselves are released as soon as they are ready and are not held for simultaneous publication. Often, that can mean customers have already updated their systems and are running fixed versions before there is public awareness of the specific issue.

There have been some questions as to why creating fixes and releasing updates can take several weeks, or sometimes even months, before an advisory is published. In some cases, there is a large number of supported software versions to be updated. The number of affected versions that will be updated can range from single digits to nearly 50 or more. We are committed to issuing fixes for every one of those supported versions.

A key example would be as follows: if a vulnerability is discovered in IOS-XE 16.6 for the ISR series of routers, a fix must be developed for 16.6 in addition to any supported versions released prior to 16.6, as well as all supported versions released after 16.6, such as 16.7, 16.8, and 16.8.1. Furthermore, the fix must be integrated into code trains developed for different hardware platforms.

A Cisco code train is a vehicle for delivering releases with a common code base. A train is analogous to a version. A Cisco software release is a copy of the code base of a train (version) at a specific moment in time.

If we disclosed the vulnerability after only fixing one release, we would unnecessarily expose all customers running other releases to potential exploitation once details about the attack itself became public.

Many Cisco products such as Cisco IOS, Cisco NX-OS, Cisco IOS-XE, and Cisco IOS-XR Software run on a wide variety of platforms, including multiple types of purpose-built hardware dedicated to routing or switching functionality, in addition to open computing virtual machine (VM) platforms that run virtual editions of Cisco software. Other Cisco products may run on a few platforms but support multiple trains of software to support customers with different required feature sets.

Whenever possible before disclosing any vulnerability, Cisco will ensure that fixed code is available for all platforms affected by that particular vulnerability in order to best protect our customers. However, implementing the fix into each supported code train for each supported platform will often affect the speed at which vulnerabilities can be remediated for any given platform.

Exceptions to this policy may occur when we find there is 1) public awareness of the issue, 2) public disclosure/instructions for exploit, or 3) if we have evidence that the vulnerability has been exploited. In cases such as these, public notification takes precedence over waiting for fixed code to become available, allowing customers to assess and take necessary steps to monitor and protect their network while fixes are being developed. Information on detection and mitigation of an exploit will also be published anytime it is available. Protection of customers, the industry, and the Internet as a whole is paramount. And Cisco will continue to take active measures to safeguard the security and reliability of our equipment, delivering on our commitment to security and transparency.

Key Takeaways:

To ensure the greatest level of protection for customer networks, Cisco will fix all supported software versions/trains prior to disclosure, unless a security flaw is publicly disclosed or an active exploit is found.

There will always be some delay between fixing the first affected software train and the last, though disclosure will immediately follow the last fixed software publication in accordance with our standing disclosure timeline/policy. We are committed to minimizing this delay.

Cisco’s responsible disclosure process ensures that vulnerabilities are not disclosed prior to fixed software becoming available for all supported release trains.