The average IT refresh cycle for access layer switches is typically anywhere between 5 to 8 years. This means the typical engineer configures a switch once, and then, other than minor changes, doesn’t have to reconfigure that switch again for a long time. The rollout process also tends to be staggered over time and location, with the typical rollout going building by building or floor by floor.
If you are with me so far, you can imagine my shock when I first took on the role of leading the NOC team at Cisco Live and learned that not only do I need to configure 500+ switches but I need to do so in about 3 months with the bulk of the requirements coming in about 3 weeks out. To make matters worse most of these were not brand new out-of-the-box switches. I was faced with the uphill task of wiping 500 switches, loading code on them and configuring them to be consistent and stable.
Fortunately, the range of products available to me made this enormous task relatively straightforward.
STAGING SETUP
We started off by building racks that we used to stage 3560cx switches. Each rack can hold 42 switches. We broke up each rack logically as bins and slots to make it easier to manage. As you can see below each rack has 3 bins and each bin has 14 slots.
Each slot has power, Ethernet and a console connection. Power is connected to a network controlled PDU so each slot can be turned off and on individually. Console cable connects to a terminal server that has a menu created that lists which line connects to which slot. Ethernet all runs back to a 3850 to provide network connectivity (and POE if APs are being staged). The 3850 uplinks to the network.
Now that we have all the logistics and setup are taken care of we can start the staging process.
PRE-STAGING CONFIG
As we load up the switches in a slot we scan the serial and the mac into a spreadsheet and use some formula magic to convert the base mac address to the address that will be used for the VLAN SVI of our switch mgmt. VLAN. The mac address is then used to create a reservation in Cisco Prime Network Registrar for DHCP.
This way, when the switch boots up and gets a DHCP address, it will be the same address that we will end up using for the show via static assignment. This helps us in two ways, one the address doesn’t change during the staging process so there’s no disruption to the process and two we see the same IP-SERIAL-MAC combo everywhere in the network, start to finish.
The scope on the DHCP server serving the switches had option 43 configured to point to APIC-EM
Switch staging
We used APIC-EM and its PNP features to stage all our switches this year. APIC-EM supports template variables which is what we used to configure a unique hostname and IP address for each switch. You remember all those serials we scanned before? We used that csv to bulk upload the switch info to APIC-EM and into a project. This time around we did not get an opportunity to use APIC-EMs rich API capabilities to program a project but, are definitely looking to do so in the future.
Once the switch info is in APIC-EM, we are ready to begin staging. On the uplink switch we added the PNP startup VLAN command to ensure the switch gets an IP in the right VLAN and can talk to APIC-EM. At this point, all we had to do is power on the switch. The switch would then boot up, grab a DHCP IP, connect to APIC-EM, download code, reboot and then download config. That’s it! We’re done.
Using APIC-EM we staged 3560CX switches, 3850-24XU switches and 3850-24XU stack switches. We had almost a 100% success rate. The few failures we had was because of code dependencies on the switch which is easily fixed by manually adding the PNP commands via console. A lot of these switches were provisioned over lunch, on the weekend at home and yes, even over a few beers while watching TV. That’s the beauty of this setup, it beats the alternative and ensures consistency on all the switches. Honestly, I don’t think I could ever go back to the old way of copy paste or USB.
Great use case for APIC-EM/PnP,
Key take-away:
Apart from the usual marketing terminologies like Reducing OPEX etc., , here is the most important bulletpoint!
***APIC-EM allows you to have more beers while watching TV 🙂
Love APIC-EM and your article! Keep coming….
It’s amazing how IT transforms the way we run business nowadays
Great example of how Cisco uses its own technology to get the necessary done. I really like the comment on not going back to “old way of copy paste or USB”.
Great use case for PnP with APIC-EM. Well written!!
APIC-EM with PnP. Outstanding.
A huge project made easier with the Automation and Orchestration tools of Cisco Network Intuitive.
Congratulations for leading NOC@CLUS LV 2017 successfully, Ryan! Succinct and insightful blog covering APIC-EM/PnP use case for staging.