What is Cisco IT’s UC Global Cluster Architecture?

Over the last 15 years, we’ve progressively centralized our UC architecture. As WAN services have become bigger, better and more cost effective, we’ve been able to rely more heavily on the network to extend UC services out to the branch networks. Our clusters are distributed around the globe in our regional high class data center hosting facilities, including at major regional hubs–San Jose, Research Triangle Park, Amsterdam, Bangalore, Hong Kong, and Sydney.

So, what does a single cluster look like today? Due to the consolidation journey we’ve been on, most of our clusters are quite large and support several 1000s of users, and consists of Cisco Unified Communications Manager (CUCM) servers that have specialized roles. Each cluster has a single dedicated database Publisher node which masters the configuration data for the cluster. Failover pairs of subscriber nodes replicate copies of the clusters database and provide the core call processing function to the other UC clients and devices that are registered to it. These endpoints can be phones or software phone clients like Jabber, or high definition video TelePresence units.

These Subscribers are the critical core of the cluster. Cisco recommends having backup Subscriber servers that can take over the call processing function should a primary Subscriber fail. Cisco IT designs clusters with the failover subscriber pairs deployed into separate data center facilities whenever possible to avoid any possible voice outage due to an extremely rare full Data Center outage. In Europe, Cisco maintains a single cluster, with the servers spread over 200 miles in three locations (Amsterdam, London and Brussels; see Figure 1). Each cluster also has a separate pair of TFTP servers to handle the distribution of software images to each individual phone.

Figure 1. EMEAR Clustering

A small cluster would have a publisher, a pair of TFTP services and then one or two pairs of subscribers depending on how many devices need to be supported. For example, in Johannesburg, we have only two pairs of subscribers. A large cluster could have as many as 6 pairs of subscribers. Beyond that, our San Jose campus required a super-cluster design with 8 pairs of subscribers to support the cluster (see Figure 2).

Figure 2. Cluster Guidelines

Cisco Unity Connection provides voicemail services to the entire workforce. Every CUCM cluster is integrated into one of the 14 Unity Connection clusters globally. Each Unity Connection cluster consists of just two hosts and can currently support up to 20,000 voicemail boxes. In some regions, a single Unity Connection cluster will be integrated into multiple CUCM clusters.

In addition to the database and call processing nodes, each of the Cisco IT CUCM clusters has a server node allocated as a Trace File Server. This node serves a number of operational use cases. Its primary purpose is to provide a powerful compute resource, with lots of storage local to each CUCM cluster which enables the collection and analysis of trace and CDR files by operations without compromising the performance of the call processing servers. It is important that the trace file servers are local to each cluster as this enables a remote operations engineer to undertake trace analysis via remote desktop interface without needing to transfer large data files over the WAN. Over time, the operations teams have evolved this server to be the location for upgrade source files and other tools and scripts used to manage a cluster.

Each cluster is also integrated into centralized instances of Cisco Prime Collaboration for managing our voice and video environments.

There are also other types of clusters. Most of our clusters support groups of phone locations in regional WANs, like our Western US region or our European region. Two Cisco locations are so large, however, that in San Jose, California and Raleigh, North Carolina, we had to build separate campus clusters. We also have a special cluster to support TelePresence service around the world, and another special cluster that supports the voice conferencing related to our extensive use of WebEx conferencing, as part of our WebEx Cloud Connected Audio service.

Lastly, we have completed migration of our inter-cluster connections into a full SIP network, end-to-end, by interconnecting all these clusters together with a single SME cluster (see Figure 3).

Figure 3. Inter-Cluster SME Trunk Cluster

The SME is a CUCM cluster connected entirely by trunk connections to all the other clusters (this requires CUCM 8.5 or above; today the Cisco IT network is migrating to CUCM 9.1). That central SME cluster will connect to each other cluster via a regional pair of SME subscriber nodes, and route inter-cluster calls via SIP, for full SIP connectivity end-to-end.

Regardless of the type of cluster, or the various roles each CUCM server might play, all servers are part of our standard image: they are all virtual machines running on Cisco UCS, either B-series or C-series UCS servers.

If you’d like to find out how we manage our voice and video services that run on these clusters, check out my earlier blog, How Cisco IT Manages UC and Video Services.