Goals and Methods#

This application note defines a set of tests performed to evaluate mesh network performance, scalability, and reliability. The test conditions and infrastructure are described, in addition to the message latency and reliability. This testing is conducted over actual wireless devices in test networks and not in simulation.

This testing is done to provide a relative comparison between the different mesh technologies to better understand and recommend their usage. Different network and system designs have different requirements for devices and networks. As such, no one network fulfills all possible network requirements.

Normally, when analyzing data on network performance, we then consider what improvements can be made in the network to improve performance. Because of the limited data available publicly on mesh network performance in large networks today, it is difficult to have industry discussions on possible improvements or changes. For example, in commercial buildings there is concern over:

• Other network traffic, since there may be many subnets that interfere with each other.

• Wi-Fi interference from the normal building Wi-Fi infrastructure as these technologies are generally operating in the 2.4 GHz ISM band.

• Network throughput and latency as well as large network multicast latency and reliability, since multicasts are commonly used for lighting controls in dense office environments and users of the system expect responsiveness in lighting controls.

Note: This application note does not specifically address system interference or other such effects that have been addressed in other published results.

Review of Other Performance Testing#

There are no specific, defined methods for evaluating and reporting large network reliability, scalability, or latency. Silicon Labs has published papers in the past comparing network performances based on network testing. This testing is focused on device behavior and impact on battery life, and network throughput and latency. Large scale multicast testing also requires capturing accurate timing and reliability information from large distributed networks. All testing was conducted using the Silicon Labs’ Wireless Gecko SoC platform capable of running Zigbee, Thread, Bluetooth Mesh and Proprietary RF protocols to eliminate the device itself as a difference in the testing. Previously published results showed differences between transceivers, network co-processors and System-on-Chip designs. These tests all use System-on-Chip designs.

Other papers published on performance include a master’s thesis paper on Zigbee network performance, "Performance Evaluation of Zigbee Network for Embedded Electricity Meters”, by Kui Liu, which was published by the Department of Electrical Engineering at the Royal Institute of Technology in Sweden. For round trip time in open-air within a single hop, they tested at different ranges with the following packet loss results:

This test was for 50 bytes of payload unicast at a 100-millisecond interval with security off. The results show a 1 hop test consistently takes ~18 milliseconds round trip time. This testing was repeated indoors with various interference conditions, but the round trip time for 1 hop did not vary much from the 18 milliseconds. Multi-hop test results were not presented.

Note that the results in the following sections are based on a Zigbee 3.0 compliant stack, which--as required--has network security enabled, resulting in the expected longer round trip times.