Trustworthy Systems: A Peek Behind the Curtain

In a recent post, “Evolution of attacks on Cisco IOS devices”, we discussed how threats against network devices have evolved. There was no evidence that a remote attack vector or vulnerability in Cisco IOS was related to these attacks. This reinforces the value of creating more hardened and resilient systems.

The strategy for creating more secure technology has an unending goal, yet it’s a journey worth sharing.

Much has been written about and shared on our secure development lifecycle and our efforts to ensure security in the supply chain. However, there are two lesser-known initiatives that have had significant impact on Cisco product security: 1) the use of Common Security Modules and 2) sophisticated attack-focused penetration testing.

Initial Threat and Response

In late 2005, an independent security researcher was able to create and demonstrate an IOS rootkit for the first time.

Cisco built on this research by conducting our own security analysis of IOS, and determined the scope of the threat was more significant than the initial research suggested.

Our immediate response was to make sweeping changes that, in the course of nine months, resulted in significantly improved security posture. Simultaneously, we focused on the implementation of improved input sanitization and memory protection measures. The result: more than 400 hardened Cisco IOS images were made available to customers.

In the years that followed, the effort grew to address secure development more broadly, and in 2008, Cisco formally launched the Cisco Secure Development Lifecycle (CSDL). Today, it continues to serve as our foundation for developing and delivering secure and trustworthy products and solutions.

Accelerating CSDL with CSM

There are more than 200 Product Security Baseline requirements (PSB) that development teams review, determine its applicability to the product, and then implement.

To accelerate and ensure secure implementation of these requirements, Cisco began producing Common Security Modules (CSM) that are integrated into a product’s code base. These modules provide critical security functionality, as well as implementation of those technologies. For example, CiscoSSL creates a wrapper around OpenSSL for use in Cisco products and embodies the goals of CSMs:

1. Securely implement commonly used code
2. Deliver timely and controlled updates as security issues and fixes are identified
3. Lower the cost in security compliance

Today, within 24 hours of receiving an OpenSSL update, it can be integrated into our CSM, tested, and made available for updates in all Cisco products.

Taking that one step further, we are currently developing a “Cisco Product Security Cookbook.” This will include recipes designed to further assist developers’ understanding about how to implement PSB requirements more quickly and effectively. These recipes will address common risks to Cisco products and mitigate attack vectors in Cisco products, as well as those documented by SANS and OWASP.

The Art of the Possible

Developing security training for our team members to help them understand what is possible from an attacker’s perspective is also an important element of CSDL. Supporting this goal, the Security Ninja program provides a novel, inspiring, and competitive way to educate every Cisco employee and contract worker across the company. Today, nearly every Cisco engineer is a Security Ninja White belt. And the continuing growth in our team members who achieve Green, Blue, Brown, and Black belts gives testimony to the motivating and educational power of this program.

Another significant event took place in 2008 with the creation of a dedicated corps of sophisticated penetration testers called the Advanced Security Initiatives Group (ASIG). It is comprised of over 65 software and hardware engineers with one mission: use advanced penetration testing tools, techniques, and practices to identify exploitable architectural weaknesses in critical Cisco products and solutions.

The team relies on design and code analysis, creates custom or off-the-shelf attack tools, and other adversarial techniques to find security issues in our products. The findings are then provided to development teams as security defects to be resolved, along with mitigation and architectural change recommendations. Where applicable, they are disclosed as PSIRT security vulnerabilities to our customers.

What ASIG also provides to software and hardware developers is an appreciation for the art of the possible, and a new perspective on how to secure a product. Seeing how a theoretical flaw in BIOS update process logic could be used to remotely inject firmware changes in the real world creates a vivid impression and immediate change in design thinking. ASIG demonstrates to developers not just the concept of chaining defects, but how it works in their code, affords an opportunity to improve architectures and mitigates risks in trust boundaries. A more visceral understanding of coding securely and thinking defensively comes alive. A more visceral understanding of coding securely and thinking defensively comes alive when ASIG demonstrates how a cross-site scripting attack on an admin web interface, combined with other low-level security defects could enable an attacker to create and hide admin privileges and totally own the system due to logic or implementation errors.

Our team proactively discovers and mitigates potential defects in this way each year. While the products are developed to be secure, threats evolve, so we’re constantly testing to make them better given the new information. These discoveries also present us with an opportunity to reinforce and close gaps in PSB requirements selection, threat modeling, static analysis, fuzzing, CSMs, recipes, and product security compliance. It’s also creating new requirements, technologies, and tools that will expand our products’ ability to detect and recover from future attacks.

The result of these broad efforts and investments in our journey has been to provide trustworthy products and solutions to our customers. We acknowledge there are many challenges ahead. The threat landscape is continuously changing, new development and delivery models are emerging, new technologies are born; new companies appear; and IoT and cloud solutions all come together in a crazy fabric that will shape the security future. The goal of these broad efforts and investments in our journey is always to provide products and solutions to our customers that are even more trustworthy. It’s a great journey ahead of us. And we intend to make it a secure one.