Look Ma! No Hands!

Over the last couple of months, I got involved in a project aimed at minimizing the time taken to bring a new device into the network, using capabilities offered by our Prime Infrastructure application. This project is called Zero-Touch Deployment, or ZTD. Now it’s time to define what ZTD is, and how Cisco IT made it a reality.

Let’s Build a House!

As with every company, every device deployed in our infrastructure must have a configuration compliant to the standards set by our design teams.

Each device that we deploy is of a specific model, is assigned to a specific network location (data center, core, etc.), has a specific role (gateway, etc.). We use a set of rules, or blocks, to configure each of these features (a “cookbook”), and we use these specific rules to build the complete device configuration. So if the device to deploy is a 4510, and is going to be in a “data center” environment, assigned to this or that specific role, then its complete configuration will have to contain block A, block B, block C, and so on.

But this is a manual process, and has a complexity exponential to the number of devices managed; and even with our experienced group of engineers, a mistake is possible. Building this config is not just a couple of copy/paste exercises. If the device has 100 interfaces, then we need to tailor the config for each of these interfaces. And once the configuration is fine-tuned, then we still need to put it on the device.

So, is there a way to automatically build and distribute this configuration? The answer is yes, with our new Zero-Touch Deployment capability.

Like a house built with plastic bricks from a famous toy company, a complete configuration is made of individual building blocks, each one tailored to suit the exact device location/role/model/you-name-it. So, we imported those building blocks into Cisco Prime Infrastructure (PI calls them “templates”). These blocks have some “holes” that must be filled with the correct data when preparing the device config: its management IP interface, its name, etc.

We then put all these blocks together, and we build our “brick house”, which PI knows as a “composite template”. And this “brick house” can be re-used for hundreds of devices, by just filling in the necessary data. Or slightly modified to accommodate a new device type and deploy hundreds of them in no time. Or easily updated to the latest technologies we bring to our network. Or … OK, you get the idea.

To Make a Tasty ZTD for 25000 Devices, You Will Need 

Ok, so now we have this “brick house” built with PI. Then what?

Well, the next action item is to pre-provision the device in the Plug and Play section of Prime Infrastructure. The installation engineer must link a device to an existing configuration (our “brick house”), and then fill in all the missing data (hostname, management IP address, etc.). PI then creates and stores the device config, and generates a unique identifier for this device/config pair.

All that is left now is to physically install and connect the device (the infamous “rack ‘n’ stack” stage). The installation engineer must make sure that Layer2 connectivity exists between this device management interface, and its uplink switch.

So device pre-provisioned in PI? Check. Rack ‘n’ stack? Check. What is left? Getting that special device the configuration it has been dreaming of.

On the 3rd Day of Christmas, ZTD Gave to Me

We’ve broken up the different components of a deployment into logical phases, that we call ‘days’.

• Day 0: Configure the device for basic IP connectivity (using a bootstrap configuration)
• Day 1: Using Cisco Networking Services (CNS), connect to and gets managed by PI
• Day 2: Finalize the whole configuration

Day 0 is an easy one: get your smartphone (or PC) and its dedicated cable, plug it into the device serial port, start the Cisco app, and fill in a few fields: device unique identifier, username/password, … press the upload button, then sit back and relax, it’s all completely automated from here.

The smartphone app will contact the PnP gateway server, download from this server the Day 0 part of the “brick house” config and push it on the device. The device now has IP and CNS connectivity. This is enough to automatically start the second phase of ZTD: Day 1.

Day 1 is where the device gets all necessary config to be properly integrated in our network management application: it receives its routing protocols details, its SNMP config, security commands to hook it up properly to a AAA server, etc. This stage is also where the device will establish a secured connection to its Prime Infrastructure server, that will trigger an inventory of the device in PI, and will initiate an image upgrade if necessary. At the end of Day 1, the device is now in PI, secured, and reachable over our network.

But why do we need a Day 2 config then? Can’t we just put everything into Day 1 if the device is reachable and in PI after this Day 1 stage?

In fact, no, we can’t cram the complete config into Day 1. And the reason is quite simple: when the Day 1 stage starts, PI has no idea what the device will be made of: each one of them can have a different HW configuration, various module, single or dual supervisor, etc. So, it is only after pushing this Day 1 config, that PI precisely knows how many interfaces (and of what type) a device is made of. And this is why a final step is necessary.

Day 2 is where every part of the configuration that depends on the outcome of the inventory will be pushed onto the device: if we connect an AP to the last port of each blade, is it port 24 or 48? Is this a 40 Gigabit Ethernet or a Ten Gigabit Ethernet port? All these questions can now be answered by PI, and we can easily push the final part of the configuration to the device. To make our life really easy, we took advantage of a programming language integrated in PI configuration tools (Apache Velocity). With this language, we can navigate through the list of all interfaces, and automatically map the proper software configuration to the hardware one.

“Z” Stands for “Zero”

Cool. But! Where’s that “Zero” part if we still need someone to bootstrap the whole process? Shouldn’t there be zero human interaction after the “rack ‘n’ stack” stage?

True, true. In this early stage of the project, we still need a little human interaction to bootstrap the ZTD process. But this is limited to plugging a smartphone or laptop into the serial port of the device, then passing the device PIN to the PnP application. And then moving on to the next device. So, it’s already looking good.

And wait. There’s more. SUDI, Smart Install, DHCP extensions, … all possible ways that we will put in place to extend the ZTD capability to ensure a true “Zero”-Touch experience, and speed up the deployment of new devices.

Saving time – and thus reducing deployment costs – is not the only benefit brought in by the ZTD capability. The “brick house” we built to automatically deploy devices will become the official configuration, or “golden config”, for this device/location/role. It can then be compared to a device running configuration, to report any changes that have been done. Or it can make the RMA of a broken device a very simple exercise.

So, ZTD is All Sunshine and Happiness Then?

Unfortunately, not yet. Although our Prime Infrastructure application has matured over the last releases and slowly replaces all the auto-configuration tools we have, we will probably still discover several challenges that need to be addressed, as with any other application, and there are always user experience improvements to add. But we are our own customer, and as such, Cisco IT is actively engaged with our business units to help improving these tools to suit complex infrastructures.

And ZTD is already there. We upgraded our PI servers to the latest release to take advantage of some new features, and a couple of deployments already happened in November and December as part of the pilot phase (several 4510s are now up and running thanks to ZTD). And this is just a beginning! 2014 has already new deployments planned, with more locations and more devices.

There were definitely challenges along the way. However, with these lessons learned, Cisco IT successfully deployed devices into the Production network. It makes for a very nice looking brick house!