Several customers have asked me how Cisco IT does project planning for a large enterprise deployment such as the Identity Services Engine, or ISE. What’s our approach? How do we manage operational costs? How do we measure performance? What personnel are involved throughout the process?
ISE Deployment Project Planning
I’ll start by laying out the major steps in our project planning for ISE. Cisco IT uses a well-defined solution delivery methodology or project lifecycle framework (PLC). The PLC defines several stages in the delivery of a solution or IT capability, with approval gates and specific deliverables at each stage. All projects begin with a Business Commit phase where the business case and benefits are clearly articulated. Subsequent phases include Concept Commit, where the solution architecture is the primary outcome, and Execute Commit, where detailed design work and planning starts. Architecture, Design, and Operational Readiness Reviews occur throughout the process, ensuring no solution is implemented without the appropriate resources, support, and communications plans in place.
Careful project management, along with governance from the Service Management Office and the Project Management Office, and regular syncing with the project stakeholders ensure that projects stay on time and on budget. Deviations are quickly identified, and appropriate remediation or business decisions can be made expediently to get back on track.
Our global ISE deployment began well over two years ago with a pilot in China. The initial team was composed of a project manager, analyst, design engineer, sponsor, and an architect. At Cisco we rely on architects to present the vision and goals for each project and program. Once that vision is written, it is vetted by our Architecture Review Board (a group of peer architects who review and approve the document). Then the project has an approved business requirement, and it’s ready to move to the next stage in the process: defining performance metrics.
In the last few years, a committed focus on service metrics has been applied to all Cisco IT projects. Each project must show clear, verifiable benefits, either through positive return on investment (ROI), or improvements in service metrics, such as enhanced user experience, reduced service-impacting incidents, greater reliability, higher security, etc. For security-related projects, quantifying costs savings in specific dollars can be difficult. Many organizations will try to express the savings in terms of cost avoidance (if we implement X, it will prevent the company from spending Y). However, these figures can be very hard to quantify and even harder to validate. For ISE the metrics were focused in three service areas: Leverage, Risk, and User Experience. In the Leverage category, we defined the metric as reducing the number of units (devices) covered by 802.1X authentication. In the Risk category, it was increasing the on-time closure of vulnerabilities (achieved via several ISE security capabilities). In the User Experience category, we deliver improved business value by using the ISE profiling. Knowing exactly what and who are on the network, from mobile devices to laptops, enables Cisco IT to improve operational excellence and deliver a better user experience.
Ultimately, the business driver for deploying ISE is our strategy of embedding security within the network fabric. Relying on outdated “hardened perimeter” based models does not provide the security essential in a borderless enterprise, and an environment where trends such as Bring Your Own Device (BYOD) and the Internet of Everything (IoE) result in many more devices that require secure connectivity.
Cisco IT’s global ISE deployment also delivers operational and architectural simplicity, which, in turn, does drive real cost savings. Previously we had to maintain 12 servers globally with long access control lists (ACLs) to provide guest access. With the deployment of ISE Guest, we’ve reduced our footprint to 2 virtual machines (VMs) globally and an ACL of fewer than 20 lines. Deriving the exact cost savings will depend on your total cost of ownership (TCO) calculator. Our operational, infrastructure, and upgrade costs have experienced actual TCO reductions for guest access service alone.
The next major step in the project planning process is creating the design guide. Building upon the vision and goals developed by the architect, the design document lays out how the implementation will be completed physically and logically. The document goes to a Design Review Board composed of design engineers who read, review, and provide comments. Then the author updates the document as needed. The ISE design guide has undergone numerous edits and reviews as business needs and requirements change over time and new capabilities are added to the solution plan.
The final pre-deployment steps for a project the size and scope of ISE consist of communications, implementation resourcing (more on this below), and operational support documentation. Communications vary depending on the capability being deployed. For example, communication to end users wasn’t deemed necessary when we migrated wireless authentication from the Cisco Access Control System (ACS) to ISE. While this migration delivered benefits to Cisco IT, the change was transparent to end users. On the other hand, deploying guest networking and wired authentication alters the end user experience, and so a more comprehensive communications plan was documented and implemented.
A global company such as Cisco needs to layer communications and use multiple channels to ensure the broadest audience and reach. To alert employees of upcoming changes, we leveraged Cisco Now. This system of large flat screens located in public and common spaces throughout Cisco campuses globally use our Digital Media Players to push out company-wide communications. Additionally, the ISE project team attended some of the regular meetings held by Cisco IT theater leads (senior management responsible for Cisco’s five global theaters). This gave the project team an opportunity to inform large groups of users simultaneously about impending changes and impacts on the user experience. Group and Cisco building-wide emails were used for communications as well.
Like communications, operational support documentation and training depends on the capability being deployed. At the lower levels of support (often called Tier 1 and Tier 2) for deployments like wireless authentication, there wasn’t a noticeable difference in troubleshooting and escalation between ACS and ISE. At the upper levels of support, Tier 3 and higher, training on troubleshooting authentication and product issues (from ACS to ISE) was required. The team that manages Tier 2 and Tier 3 support had a “Train the Trainer” approach, sending staff to Cisco’s course in ISE deployment and configuration provided by a third-party training vendor. Because wired authentication is a new capability to Cisco’s network, support training and documentation was required for all tiers of the organization.
Lessons learned from the metrics and communications steps in the project planning process:
- When trying to derive ROI or service metrics, many of the capabilities delivered by ISE will not have a direct cost savings; however, the service metrics are positively impacted as a direct result of the ISE deployment.
- Communicating to end users about changes that will alter the user experience is critical and need to be multichannel. Do not rely on email alone as many users get hundreds of emails daily. Signage, other alert mechanisms, and management support are ideal.
ISE Deployment Personnel
I can share how we have staffed the deployment and support of ISE. A team of implementation engineers plans, oversees, and executes the deployment of each capability. A project manager and engineers are assigned after the project team completes a resourcing process to determine the size and scope of each project.
For the ISE deployment, there is an implementation project manager, a lead implementation engineer, and three to ten engineers to perform the deployments. The number varies depending on the size and complexity of the capability being deployed. For example, the early stages of our guest networking and wireless authentication deployments involved the project manager, lead engineer, and only three engineers. All the work is done remotely, either at the command line or via Cisco Prime Infrastructure utilizing templates. Most often the bulk of the work is quality assurance, to verify that everything is going as intended. The changes involved in deployment are generally small configuration modifications of a few lines at most.
Operational support is split into two main categories: direct support and infrastructure / upper levels of support. Direct support personnel answer phone calls and perform troubleshooting and escalations. In terms of numbers, the direct support team for the ISE deployment has not changed in the last two years. In Cisco fiscal-year 2015 (starts in August and runs through the following July), we have not budgeted or forecast any changes in this level of support.
Infrastructure and upper levels of support are handled by the team that manages our ACS and ISE infrastructure. This team was augmented with two additional personnel to handle the new services offered by ISE. In the upcoming fiscal year, we’re planning an aggressive capability deployment that includes Mobile Device Management (MDM) and expansion of wired authentication globally. We have budgeted to continue with the same two additional Change-the-Business (CtB) personnel. As the capabilities are fully deployed in fiscal-year 2015, this CtB cost will become a Run-the-Business (RtB) expense. Providing this category of infrastructure support is a team that develops and deploys monitors for such services at PEAP, MS-CHAP, and EAP-TLS used by Operations. These personnel are strictly associated with CtB costs and are paid only during their engagement. We have budgeted for one person in this team to be used year over year during the life of the project.
Lessons learned from ISE deployment and support staffing:
- New ISE support staffing has been required only for the upper levels of support. However, this isn’t a function of ISE but a function of the capability deployment.
- Leveraging a global engineering staff allows Cisco IT to seek favorable staffing costs, when possible.
ISE Deployment Progress
Since my last blog, Cisco IT has made good progress with our ISE deployment. In mid-May 2014, we completed the last limited deployment (LD) of wireless authentication (from ACS to ISE). For this final LD (there were three total of increasing sizes) our target was to get total endpoint count to 50 percent of known endpoints (from ACS figures). The previous LDs focused on a wide range of global, large sites (by endpoints) to ensure that we would get to the 50 percent target. Once achieved, the remaining 50 percent of the globe will be migrated rapidly.
This latest deployment pushed our total profiled endpoint count to more than 230,000 globally, with 35,000 peak active endpoints at one time. As we look to complete our wireless authentication deployment by the end of June 2014, we anticipate the profiled endpoint count to exceed 300,000 and peak active endpoints to trend to 70,000. After the wireless authentication deployment is successfully completed, I plan on blogging about it in more detail
Cisco IT is finalizing our deployment plans for FY15. As mentioned, our focus will be on MDM, wired authentication, and EPS (also known as quarantining). MDM expands our reach into the mobile devices and will involve the Mobility team, a separate group within Cisco IT that manages the MDM service. Because of the breadth, speed, and focus on capability deployments in FY15, the team is organizing an internal “Cisco IT summit” to bring all the disparate groups (Network Services, Information Security, Mobility, ISE Infrastructure, Desktop) together for three days to discuss and plan the fiscal year in detail. Bringing these teams together under one roof, live and via telepresence, will greatly enhance our understanding of the deliverables, dependencies, and deadlines.
Stay tuned for more blogs about Cisco IT’s ISE deployment journey. Much of the ISE project background and deployment work before my blog can be found in this article.
CONNECT WITH CISCO
LET US HELP
Call us: 1.800.553.6387 - Ext 118
US/Can | 5am-5pm Pacific Other Countries