Intuitive Test Automation for an Intuitive Network

The rise of intent-based, programmable networks has created new and exciting opportunities: ones that require new tools, new developer skills, and sometimes, new ways of thinking. So what do you do when presented with the opportunity to shape Cisco’s next 20 years of test automation?

Our DevX test engineering team at Cisco had an answer to that question, and about 3½ years ago we embarked on a quest to create the next-generation test automation framework for Cisco engineering. One that would be crafted using a modern programming language; and tailor fitted to operate in rapidly iterating, agile development environments.

An intuitive test automation framework for an intuitive network.

After multiple major internal releases, amassing thousands of test developers and millions of lines of test scripts and libraries, today, pyATS (Python Automation Test System) is now available to the world through Cisco DevNet.

It’s not about the destination; it’s about the journey.
Test frameworks are intrinsically simple: define test cases, run test cases, report results. Anyone can go and create their own tool to do just that. But what differentiates a tool from a successful ecosystem is the culture that it brings about: enabling people to share and collaborate; proliferating positive designs, ideas and methodologies; and ultimately moving the testing community forward.

We did not just aim for today. We designed for tomorrow.
The last 20 years of Cisco test automation has been deeply rooted in Tcl, a popular language used for automation in the early days of telecommunication. Shifting this [procedural based] momentum into a new, object-oriented modern era presented a unique challenge. We had to rethink the impossible, consider the improbable, and reinvent test automation from the bottom up: architecting for what would be the new ways of doing things for the next decades, and bridging the ways of yesterday into the ways of tomorrow.

At the helm of a new era of test automation
pyATS is a test framework featuring all the necessary bits required for network test automation, and leverages all of the perks of the Python-3 programming language:

The ability to define reusable, inheritable, data-driven test cases and suites, with distinct sections for common, suite-level configuration and teardown

User-controlled, on-demand, selective & asynchronous execution

YAML representation of network topology and devices

Reuse of any existing Tcl-based libraries and packages (eg, HLTAPIs)

Built-in debugging capability, such as automatically starting interactive shell on point of failure

Customization/extension via plugins, configurations, and hooks

Platform agnostic, with new platform (including third-party devices) support added through polymorphic plugin interfaces

“pyATS code is beautiful to read” – Raymond Hettinger, veteran Python Core Developer.

With pyATS, network devices and interconnects are defined through YAML files. These YAML [testbed] files then loads into corresponding Python objects, using references & relationships to depict network topology; properties & methods to represent network functionality, device control, configuration and status.

From there on, developers can start with simple, linear tests, such as configuring interface and checking pings; and easily migrate towards larger, complex, data-driven and reusable tests, such as scaling MPLS with various routing protocols and number of traffic streams.

The framework is suitable for a wide variety of testing methodologies. In Cisco today, pyATS is deployed within multiple business groups, executing test cases in sanity, regression and solution labs; covering products ranging from Wi-Fi access points to core routing platforms, from IOS CLI to DNA-C web UI.

Now available through Cisco DevNet
One of the key principles to true organizational agility lies with development transparency. As pyATS gathered momentum within Cisco, teams began inquiring whether it’s possible to share their automation with customers as part of collaborative development and release. To do so requires providing customers with access to the test framework. Could it be done? Yes it could!

With pyATS available through Cisco DevNet, we bring customers one step closer. Streamlining development-to-deployment and unifying the end-to-end toolchain, pyATS is finding its way into customer facing teams such as Advanced Services, where soon our customers will be serviced using the very same tools, packages and libraries used and developed by Cisco Engineering.

And one more thing …
Great test automation is founded on two pillars: the framework, and the libraries. One major design feature of pyATS is to bridge developers: start with reusing existing Tcl-based packages; transition to pure Python over time, without having to start from absolute zero. Meanwhile, we began working on a new, Pythonic library system.

It is called Genie, a package and library system that supplements pyATS, featuring:

Object-oriented, feature-centric base objects modeling device configuration and operation

OS/Platform agnostic class designs inspired from common YANG models

Event-based testing through triggers & verifications

Plug & play with existing pyATS test suites

Genie will be made available through Cisco DevNet in the upcoming months.

Calling all engineers…
The future of networking is intent-based technologies that consistently learn, adapt, and protect. The future of network test automation lies with repeatable, reusable and re-scalable tests and libraries that grow alongside these technologies, where we develop, build and share as one community.

This is a call to all developers – to spend a few minutes on our newly launched DevNet website; watch the short introductory video by Raymond Hettinger; try the sample code and scripts; see the vision; and join the circle of engineers that devote their time to pyATS: an all-out effort to build a better network test automation ecosystem.