October 1, 2019 4 Comments
Avatar
Developer

Tune in: “Demystifying Cisco Orchestration for Infrastructure as Code”

On October 24th 2019 we will discuss “Demystifying Cisco Orchestration for Infrastructure as Code” on BrightTalk. Register Here.  The following blog is a primer for this upcoming webinar.

Automating the software development life-cycle

DevOps teams are becoming more agile, reducing costs, and delivering a superb customer experience by automating the software development life-cycle. Cisco Orchestration solutions extend the benefits of automation to the entire stack. Each layer of the underlying infrastructure is delivered as Code (IaC). Orchestrators reduce the complexity of programmability, operational state, and visibility. In this session, we decode the differences between domain-specific workflow automation versus cross-domain orchestration. Achieving the goal of ‘Automate everything’ requires the right tool for the right use case. With use cases in mind, we will cover several Cisco orchestration solutions in their respective domains and cross domain capabilities. A brief demo will showcase Open Source and Cisco Orchestration tools working together hand-in-hand.

Level Set

What is Infrastructure as Code (IaC)?

IaC means writing imperative or declaration code to automate programmable infrastructure deployments and manage configurations. Imperative is how you do something step-by-step, as opposed to declarative which is ‘what to do’ by abstracting the configuration and state. DevOps best practices such as source control, verification, and visibility are building blocks to support infrastructure types (compute, network, storage, etc) as code.

Why do we need IaC?

With the advent of Continuous Integration and Continuous Delivery (CI/CD), we are able to build pipelines to automate the entire software development life-cycle (SDLC). Continuous integration (CI) is a set of tools to develop applications. Continuous delivery (CD) is the process of delivering updated software releases to infrastructure environments such as test, stage, and production. Using IaC for these platforms and environments is paramount to enabling software agility and rapid time to value. One could say, IaC is the easy button to building infrastructure to deliver software or other IT services.

Orchestration

What is orchestration? Wikipedia defines orchestration as an automated arrangement, coordination, and management that defines the policies and service levels through automated workflows, provisioning, and change management. In this same vein, a coffee grinder is automation where a brewing machine is orchestration.

Why do we need orchestration in addition to scripting? Production grade IaC at scale requires orchestration versus scripting to deliver advanced features such as intent, policy, governance, and Service Level Agreements. By building IaC to include configuration management, CICD, and other advanced orchestration features, similar benefits to application development are now possible in large scale technology domains (mulit-cloud, containers, campus, WAN, Data Center).

With network automation in mind, we see many pit falls with multi-threaded tasks running exclusively from scripts. Step (1) gather facts, Step (2) set conditions, Step (3) loop through items in jinja2 templates and parse to TextFSM and save the data to YAML files. Step (4) Push changes to devices and validate.

This level of scripted multi-threaded workflow is difficult to manage at scale. The main concerns are slow changes, configuration drift, lack of operation state, out of band config overwrites, and disruptive rollbacks. In spite of the current gaps, the Ansible engine is one of my favorite tools for pushing network configurations. One could argue that Tower provides a workflow for the playbooks to manage the order of these tasks but no configuration state is possible. In order to remediate some of these gaps, Ansible engine is adding a new ‘facts’ resource module in a future 2.9 release.

Caution

Sometimes we automate ourselves into a corner with too many scripts. What happens when major platform changes are made to the scripting tools? We’ve experienced this before with python 2.x to 3.x.and the impact to many of our product SDK libraries. We are seeing it again with the major change to the Ansible engine coming in 2.9 to introduce the ‘facts’ resource module. This change requires users to rewrite their playbooks from scratch to use these features. As a caution, consider limiting scripting to single threaded (CRUD ) actions while shifting the complexities of operational state and rollback to the domain specific orchestration engine.

Domain Orchestration

What is a domain orchestration? A domain orchestration engine focuses on delivering automation targeted to a single technology domain. For instance, the Cisco Network Services Orchestator (NSO) is focused on model driven “network” automation with Netconf and YANG. NSO converts CLI to YANG with network element drivers (NEDS) to supports a multitude of uses cases ranging from stand alone network devices, network services, and multiple controller domains (Meraki, Viptela, and ACI).

In a nut shell, NSO can deploy greenfield or snyc-from brownfield devices to build a transaction based configuration database state. Tools like Ansible engine have modules to integrate with NSO’s northbound JSON API to harness these differentiated capabilities for operations. In the following example, we are using Ansible playbooks with the Ansible NSO/Json module to make CRUD changes to NSO’s configuration database as a means to configure and operate tenants running on a N9K EVPN/VXLAN Data Center network fabrics versus CLI to the stand alone NXOS. The Ansible playbooks are then version controlled as YAML files in a git repository.

Top-level Orchestration

What is Top-level orchestration? A top-level orchestration engine is used to stitch together collaboration, notifications, governance, and source control for other lower level scripting tools and device APIs. Top-level orchestration supports use cases ranging from CICD pipelines for application development to automated infrastructure build and testing. The Cisco Action Orchestrator (AO) is a powerful Top-level orchestrator that enables automated workflows across technology domains and ITSM (ie., ServiceNow). Integrations to ITSM are key for customers who need low or no code catalogs and templates to simplify the delivery of IT services.

Internally, Cisco relies on ITSM and AO to automate the rapid delivery of CiscoLive and Devnet Sandboxes during our customer events. In the below example, the open source tools such as Gitlab work hand-in-hand with AO to create a workflow pipeline to automate the build and test for a tenant configuration across EVPN/VXLAN fabric and SDWAN network domains.

Confusion

Are you confused by CICD pipelines and their relationship with IaC? My ‘Ah hah’ moment was a realization that many of these DevOps methodologies are not mutually exclusive but highly complementary between AppDev and IaC. As operators, we can support the automated development life-cycle with CICD pipelines. This same knowledge and tools are adaptable to automated infrastructures. Operations can adopt the tools (open source or vendor) that make sense for updating, configuring and management of IaC in many domains.

If you look at AppDev the CICD pipeline for software development must CODE, BUILD, TEST, and DEPLOY the software to an environment that includes infrastructure (compute, network, and storage). Do we build these infrastructure environments ahead of time manually or automated on demand?

If the developer is not willing to patiently wait several weeks for the infrastructure environment to test their CODE, then fully automated IaC is the only answer! A second CICD pipeline managing the configuration, versioning, and alignment of the software build to the environment (test, stage, prod) version allows us to move much quicker and rebuild the environment later if needed.

AppDev CI/CD pipeline to IaC CI/CD pipeline

In the following example, we are using Gitlab to manage an application development CICD pipeline. Upon completion the AppDev pipeline triggers Action Orchestrator to build a second pipeline with workflows to automate the test environment to ultimately test the application stack. The idea is to test the software release in a test environment prior to pushing the same software into production. The Action Orchestrator (AO) has many adapters to make IaC very easy to build and test infrastructure technology domains.

Router/switch software upgrades are another use case for a network specific CICD pipeline. With CICD we can automate the upgrade of specific IOS software versions to devices in a version controlled and tested environment prior to production.

Controllers

Are controllers and orchestration one in the same? NOPE…  Controllers are a single API touch point and management system for Software Defined systems (SD-Access, SD-Wan, and SD-Networks) to manage the configuration state of the underlay and overlay and underlying protocols. Controllers are similar to orchestration by providing access to configuration snapshots and rollbacks, but unable to compose top-level workflows with other tools. In most cases, Controllers are bound to their single technology domain (campus, data center, WAN, or cloud). Often times, IaC is configured adequately with only scripting and source control in a single controller domain. Suffice it to say, when expanding from a single domain to cross domain controllers (ie SDWAN, and SDN) this cross domain integration introduces a catalyst for orchestration.

The Automation Challenge

There is a broad set of technology domains, each with many use cases for IaC. In order to succeed with IaC, we first need to address our automation challenges. From there we can target each specific use case mapped to the appropriate technology domain.

Challenges:

To many touchpoints: Need to consolidate and coordinate tasks using common automation tools.

Complexity: Need to abstract automation as much as possible to make resources consumable for the end users.

Operational Instrumentation: Need to automate and operationalize the tools into workflows that include visual dashboards, role-based access control, and other security services.

Verification: Need to make changes and check changes. With automation, we can move really fast and break things. Hence, we need the proverbial looking over our shoulder versus traditional stare and compare configuration checks. Ideally, verification should start in a test environment through production. See PyATS…

Community and Collaboration:  Need to share finished code and avoid recreating the wheel with every workflow.

The key take away for automated solutions is to strive for a sharing culture, agility, simplicity, intent, security, and lower costs.

Technology Domains and Use Cases

The following table depicts the taxonomy of several Cisco orchestration options. Please refer to October 24th 2019 for a deeper discussion on each solution and use case. As depicted below, the Action Orchestrator is positioned as the glue to bind together the multiple technology domains into a unified workflow.

 

What’s Next?

Multi Domain Policy

As our customers continue to strive for end-to-end automation their orchestration workflows are now spanning across multiple technology domains. As these workflows evolve we need to consolidate and coordinate tasks using a common automation platform.

A major step in the “automate everywhere” strategy is to consolidate automation on a Multi-Domain Policy (MDP) platform. Conceptually this upcoming platform is targeted to unify the existing orchestration engines across domains with a consistent UI, catalog, united operations, common segmentation, consistent on-boarding, and delivered on-prem or cloud.

AI Ops

Logs, telemetry, and health monitoring are currently used to build reactive dashboards for visibility. With the advent of AI Ops, the trend is predictive and self healing operations. AI Ops platforms utilize big data, modern machine learning, and other advanced analytics technologies. This new technology, directly and indirectly, enhances IT operations functions with proactive, personal and dynamic insight. Cisco Intersight is a SaaS addition to the portfolio of domain orchestration engines, making actionable intelligence available from AI Ops in Hyperflex and server domains. AI Ops capabilities are road-mapped into many other orchestration engines as well.

Stay tuned for follow-on blogs discussing these emerging technologies. A replay of the BrightTalk “Demystifying Cisco Orchestration for Infrastructure as Code”  is available here.

Thoughts

What do you make of it all? Leave a question or comment below. Also feel free to hit me up on Twitter @tdubiel or LinkedIn @tdubiel.

Related resources

DevNet training resources

DevNet Sandbox

Product resources