Scaling the Internet with 6500 Switches Ternary Content Addressable Memory(TCAM) Customization

The Internet routing table size has continued to grow steadily. In 2008 we reached 256K routes and now the table has exceed 512K Routes. This is of significance for customers running some of the older PFC3 based Supervisor 720 engines on the 6500 and 7600 switches.

On the Catalyst 6500 and 7600 Series platforms, all of the routing information is stored in special high-speed memory called TCAM.

The PFC3BXL and PFC3CXL support one millon (1M) IPv4 routes and 512,000 (512k) IPv6 routes in TCAM. However, default outputs look different:

6500#show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :
---------
IPv4 + MPLS         - 512k (default)
IPv6 + IP Multicast - 256k (default)

This blog focuses on the FIB TCAM; however, the information in this blog can also be used in order to resolve these error messages:

%MLSCEF-SP-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry
usage is at 95% capacity for IPv4 unicast protocol

%MLSCEF-DFC4-7-FIB_EXCEPTION: FIB TCAM exception, Some
entries will be software switched

%MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception, Some
entries will be software switched

It is important to be aware that the aforementioned errors are reported only once when the TCAM limit is exceeded. Even if the total number of routes withdraws below the configured limit, the switch remains in the exception state until it is cleared:

7600#show mls cef exception status
Current IPv4 FIB exception state = TRUE
Current IPv6 FIB exception state = FALSE
Current MPLS FIB exception state = FALSE

As a result of this TCAM exception condition, connectivity is affected and might result in elevated CPU usage due to software switching.

Solving the Internet Routing Table exceeding 512K Problem

The default numbers for PFC3BXL/PFC3CXL are 512k IPv4 routes and 256k IPv6 routes. These numbers can be increased to 1M or 768K IPv4 routes if you enter mls cef maximum-routes ip|ipv6 [] and reload. But, you cannot achieve both 1M IPv4 AND 512k IPv6 routes at the same time. If you increase the IPv4 TCAM size above the default value, it automatically takes up the IPv6 space and vice versa.

6500(config)#mls cef maximum-routes ip 1007

6500#show mls cef maximum-routes
FIB TCAM maximum routes:
=======================
IPv4                 - 1007k
MPLS                - 1k (default)
IPv6 + IP Multicast  - 8k (default)

6500(config)#mls cef maximum-routes ipv6 503
6500#show mls cef maximum-routes
FIB TCAM maximum routes:
=======================
IPv4 + MPLS         - 16k (default)
IPv6                - 503k
IP Multicast        - 1k (default)

Note: The Sup2T supervisor engine and DFC4XL modules support 1,000,000 routes
that are dynamically shared between IPv4 and IPv6 by default and wont face
this issue.