Introducing the First Benchmark Standard for HCI

Industry standards enable fair competition and provide an objective “truth” to the customers. No one buys a car without first inquiring about its miles per gallon and vehicle safety ratings. Prior to the advent of standards, developed and administered by neutral parties, there was no way to compare different systems as each vendor would showcase their best performance in a contrived environment. Standards level the playing field, so to speak, bringing everyone to the same starting position and then based on their capabilities see who will finish first.

The Transaction Processing Performance Council (TPC) and the Standard Performance Evaluation Corporation (SPEC) have defined several standards for benchmarking computing systems. In recent years, the TPC has developed standards in the big data analytics and Internet of Things space: the TPCx-HS, TPCx-BB and TPC-DS V2 are standards for big data systems; while the TPCx-IoT is the industry’s first standard for IoT gateway systems.

These standards are widely used by customers in their key purchasing decisions; and they also influence technology enhancement with newer versions of products performing at higher levels with better energy efficiency at lower cost than previous versions. As pioneered by Jim Gray, et al. (in their early attempts at civilized competition) the TPC doesn’t just create the benchmarks, it also implements a process to review and monitor those benchmarks, and this lays the foundation for a fair competition.

Continuing with their charter, today, the TPC announced the industry’s first standard for benchmarking Hyper-Converged Infrastructure clusters under a database workload, TPC express benchmark HCI (TPCx-HCI). A Hyper-Converged Infrastructure is a software-defined IT architecture that virtualizes all of the compute, storage and networking resources. It enables you to make more effective use of the underlying hardware by providing flexibility and agility for workload placement.

Until now, the only available benchmarks for the hyper-converged infrastructure market have been the simple micro-benchmarks that measure IOPS. TPCx-HCI measures the performance of a complex application at the system level, and was crafted in direct response to demand from the user community.

The TPCx-HCI benchmark is designed to stress this hyper-converged platform under a demanding database workload. The standard has two unique characteristics:

• Workload Elasticity. The benchmark features an elastic workload that varies the load delivered to each of the virtual machines by as much as 16 times, while maintaining a constant load at the cluster level. Sustaining optimal throughput for this workload on a multi-node HCI cluster would typically benefit from frequent migrations of virtual machines to rebalance the load across nodes. This property measures the efficiency of virtual machine migration as well as the uniformity of access to data from all the nodes.
• Data Accessibility. In the Data Accessibility test, a node is powered down ungracefully, and tested for continues application uptime. The test sponsor is required to include a throughput graph for this test, demonstrating the impact on performance, as well as report the recovery time to establish the resilience of the infrastructure.

Cisco has worked closely with our industry peers, Reza Taheri, Chair (VMware), Eric Wendel (DataCore), Ncholas Wakou (Dell), Paul Cao (HPE), Da-Qi Ren (Huawei), Jamie Reding (Microsoft), Bhavik Desai & Gary Little (Nutanix), Mike Brey (Oracle), Andy Bond (Red Hat), Greg Kopczynski (VMware) in developing this benchmark. We believe that the TPCx-HCI standard will be a useful tool for buyers considering different Hyper-Converged Infrastructure offerings. The specification and kit are publicly available for download via the TPC website at: http://www.tpc.org/tpcx-hci/default.asp

Sincerely
Raghu Nambiar, Chair – TPC IoT/AI/PR

Share