Streaming Telemetry with Google Protocol Buffers

In a recent blog on “Why You Should Care About Model-Driven Telemetry”, I reviewed the need for streaming telemetry and the high-level architecture that makes it possible. Today I’m going to drill down on some specific design choices we made in order to quickly export large amounts of data.

Tell Me How You Really Feel About XML

When we first started working on streaming telemetry, one of the key questions was how we should encode (or “serialize”) the data on the wire. XML, that warhorse of NETCONF fame, seemed like an obvious first choice. XML is a well-defined, text-based markup language that encloses everything in tags. Now some people like XML and some people really do not.  But one thing is for sure: XML is not a compact encoding.  All those tags are strings and strings simply take up more space than, say, integers. Moreover, string operations are just plain slower in most programming languages.  Since high performance was one of our key goals for telemetry, XML was not really the best fit.

After some experiments and customer engagements, we settled on Google Protocol Buffers (aka “GPB” or “protobufs”) as a widely adopted, highly efficient way of encoding telemetry data (we also support JSON encoding, but that’s another story).   In our first release (XR 6.0.0), we went all-in on performance with a “compact” GPB format that represents the ultimate in efficiency and speed.  However, that compactness came at a cost of some operational complexity.  So in XR 6.0.1, we introduced an alternative format (“key-value” or “self-describing”) that is less efficient but simpler to use.

The main difference between the two GPB telemetry formats is how they represent keys. Any piece of data is only interesting if you know what it is.  For example, if a router sent you an integer like “449825”, it could represent anything — uptime, packets in, bytes out, etc.  In order to know what “449852” is, you need a key.  So let’s take a closer look at how these keys are represented in the different encodings.

Compact GPB

Compact GPB was the original format supported in XR 6.0.0.  In compact GPB, the “key” that the router includes in the telemetry packet is just an integer.  So, for the interface statistics, you’d get data that looked like this:

1: GigabitEthernet0/0/0/0
50: 449825
51: 41624083
52: 360333
53: 29699362
54: 91299
55: 25
56: 188801
<snipped for brevity>

You might guess that “1” stands for the interface name, but what about 50, 51, 52?  In order for you to decode this, you need a decoder ring.  The decoder ring for GPB is called a “.proto” file.  With compact GPB, you need to generate a “.proto” file on the router for every path that you want to stream and upload it to your collector.  For example, if you wanted to stream the interface statistics, you would need to generate a “.proto” (‘int_stats.proto’ below) as follows:

RP/0/RP0/CPU0:Sun601# telemetry generate gpb-encoding path 'RootOper.InfraStatistics.Interface(*).Latest.GenericCounters' file disk0:int_stats.proto syntax proto3

Here’s part of the int_stats.proto I just generated:

With this “.proto” file, your collector can determine that key (or “field number”) “50” means packets_received, “51” means bytes_received, and so on.

Now you might be wondering why anyone would bother with all this secret-squirrel-decoder-proto stuff.  The answer goes back to the name.  This encoding is compact.  It is far more efficient to send integers like “54” across the wire than strings like “MulticastPacketsReceived.”  And GPB is really good at sending integers on the wire:  it uses the concept of “varints” to serialize integers even more efficiently (i.e. a 64 bit integer doesn’t actually need to take up 64 bits on the wire most of the time).  As you can see from the sniffer trace, GPB allows us to get 35 counters for 5 interfaces in 833 bytes.  So if efficiency, speed, and performance are paramount for you, you might be willing to put up with the operational headache of managing the “.proto” files on your collector.

Note: Unfortunately for those of us who like to read sniffer traces, GPB’s heavy use of varints means that you can’t easily “eyeball” the trace and pick out field numbers and values.  You’ll have to take a deep dive into the mysteries of varints to understand why 0x0A16 indicates that field number “1” (InterfaceName) with a length-delimited value of 20 bytes will follow (“GigabitEthernet0/0/0/0”).

Key-Value / “Self-Describing” Format

In the key-value format, the key is sent as a string.  Strings are much less efficient on the wire than varints, but they are self-describing.  This means that you don’t need a “.proto” file decoder ring for every path.  You use a single “.proto” file for all paths, then read the keys to figure out what the values refer to.  The sniffer trace for key-value encoding “scans” more easily because of all the strings:

That’s much friendlier to look at, but note how much bigger the data is.  Instead of 833 bytes in a single UDP packet, you now have more than 4000 bytes across 4 TCP packets (plus the overhead of the TCP session itself).

Base Proto Comparison

The following table compares the “.proto” files for compact GPB vs. Key-Value.  Note that the “policy_path” in the Telemetry_Table of the compact GPB is used by the collector to select which path-specific “.proto” (like the int_stats.proto above) to decode the data in the repeated “rows” that follow.  In contrast, the key-value “.proto” is the only file the collector needs.

Transport

Because it’s so compact, the compact GPB encoding can fit even some of the longest “rows” of data in XR into a single packet (tables can be split across multiple packets but rows cannot).  Therefore, UDP can be used as a transport for compact GPB and it is the default.  If you really like TCP or encounter one of the few rows in IOS XR that can’t fit in a compact packet (even with IncludeFilters), then you can optionally enable TCP (which handles fragmentation for you).  For key-value, TCP is the only supported transport since the packets will be so much bigger.

Configuration

The following table summarizes the differences in configuration for compact GPB (over UDP and TCP) and Key-Value GPB in XR 6.0.1. As you can see, the key-value encoding is the default if you specify TCP transport for GPB.

The Bottom Line

The key-value GPB encoding has a lot of appeal from an operational perspective since you’re managing a single static “.proto” file on the collector. On the other hand, the compact GPB encoding is (on average) three times more compact and at least twice as fast in our early performance testing (and that’s not counting any improvements on the collector side that come from avoiding a lot of string manipulation).  Another relevant point is that the performance of key-value GPB encoding is in the same ballpark as compressed JSON.  So if protobufs are unfamiliar or intimidating, you can always use the JSON encoding and know that you’re getting a good balance of usability and performance. Ultimately, the one that’s right for you will depend on your use case and your toolchain.  Give them a try and let us know what you think!

Further Resources

Check out the compact telemetry.proto and the key-value telemetry.proto on github.com, along with sample collector scripts and open-source collector stacks.  For more information about streaming telemetry, including access to a virtual IOS XR image, go to our DevNet site for IOS-XR 6.0. And stay tuned for more blogs on telemetry, including some thoughts on the limits of SNMP and customer use cases.