Responding to Third Party Vulnerabilities

We are now more than one year on from the release of HeartBleed, the first major vulnerability disclosed in widely used third-party code. This is an excellent point in time to look back at what Cisco and our customers have achieved since, including how the Cisco Product Security Incident Response Team (PSIRT) has evolved to meet this new type of threat. It’s also a key time for us to confirm and clarify our commitment to transparency in the vulnerability disclosure process.

Since HeartBleed, the security industry has seen several other highly visible vulnerabilities in third-party software (such as ShellShock, GHOST, Poodle, NTPd, and additional OpenSSL issues.) These have impacted every major networking vendor, including Cisco. Given the large number of products affected by third-party software vulnerabilities and the potential value these offer attackers, we expect the number of these type of disclosures to grow.

In order to provide the right support and protection for our customers, the Cisco PSIRT has revisited how we manage these types of vulnerability disclosures. In our traditional Security Advisories, PSIRT coordinates with our internal product development groups to:

1. Rapidly analyze an identified threat,
2. Design and test a fix, and
3. Protect our customers by publicly disclosing the vulnerability and fixed code versions.

The most significant change in this new world is that third-party software issues start our stopwatch at the moment of public disclosure. This means that attackers and customers are both aware of the threat at the same time, before a fix can be put in place. There is now a much shorter runway for our vulnerability analysis, determining the level of criticality, analyzing each product, developing a fix, testing it, and communicating this information to our customers.

It’s also important to understand that not all third-party software vulnerabilities are critical vulnerabilities. Given that the changing landscape, including the number of issues being disclosed, the breadth of affected products, and the pervasiveness of media coverage, it would be easy to think every new disclosure is a major threat. Therefore, our emphasis on the empirical severity of these vulnerabilities becomes even more important.

To ensure that vulnerabilities are consistently evaluated, PSIRT scores all vulnerabilities with the Common Vulnerability Scoring System (CVSS), and we manage all disclosures according to our published disclosure policy. We have previously enjoyed a longer time window for the assessment of our products and release of a fix for third party announcements. Over the course of the last year we have adjusted our program, and continuous strive to improve our response.

What We’ve Done So Far:

Today, our target is to complete all our assessments within seven days of the disclosure, and to provide fixes for critical issues in our core products as quickly as possible. People, process, and technology – the three sides of the triangle – have all been improved as part of our evolution.

• First, we grew the team to help manage the increased logistical and technical effort, including adding a team dedicated to the analysis and management of third-party software incidents.
• Second, because these disclosures have been driven by events beyond our control, we’ve adjusted how we manage vulnerability disclosure. We have streamlined our internal notification and bug management process, which now alerts the necessary teams to begin investigating potential vulnerabilities in parallel.
• And finally, we have developed internal tooling to manage the increased workload.

These changes allow us to get the right information to the appropriate engineering teams more efficiently, and therefore provide faster information to Cisco customers.

Our Commitment to Customers:

• We will continue adapting PSIRT’s infrastructure and staff to enable our customers to quickly assess and mitigate any risks in their networks. Our mission is to do the right thing quickly, and to keep our customers protected.
• We want to be equally transparent with third party software vulnerabilities affecting Cisco’s products. The Cisco PSIRT will aim to provide an accurate assessment of a potential security threat within seven days, so that our engineering teams can begin their fix process.
• We will continue to work with the engineering organizations to provide a vulnerability fix quickly. With internal Cisco code, we work with our engineering teams to insert the fix in an existing release train, but the need for a rapid fix sometimes forces special releases. Short-turnaround fixes disrupt the regular release flow and require more effort for our team and customers, but they are the right thing to do, so we are doing it.
• To hold ourselves accountable, we will begin publishing metrics demonstrating our progress against these commitments. These metrics will enable customers to feel confident that Cisco is taking this problem, and the safety of our customers’ assets, very seriously.

For Cisco PSIRT, our “True North” has always been transparency. We remain committed to this ideal, as it best serves our customers and protects the trust they have placed in us.