Session 6: Inner Workings of Z-Ware#

The previous session examined the Z-Ware portal and how it is included in the SDK. The Z-Ware portal is a Z-Wave certified controller. It is used internally for software qualification of not only Z-Ware itself, but also of the Z/IP Gateway, a proof of concept illustrating that the Z-Wave controller is fully Z-Wave certifiable. However, Z-Ware can also serve as the foundation of a custom controller. To explore Z-ware in this capacity, this session covers a part of the C-API, some of the central APIs, and the general structure of Z-Ware. Additionally, it covers some of the similar APIs on the HTTPS REST API. Finally, it includes an exercise in enumerating a Z-Wave network and exercising control using the REST API instead of the Portal interface.

Network Initialization, zwnet_init, c-api#

The first requirement for a Z-Ware based controller is to initialize the network. With initialization, Z-Ware connects to the Z/IP Gateway and configures the required connections. The Z-Ware C-API populates an input structure and then passes it to the API. The network initialization, zwnet_init, has this input structure:

Some of the values are familiar because they are also exposed on the Web interface and used by other Z/IP clients. The host_port is the port which Z-Ware is listening to. The unsolicited destination, discussed previously, is where the Z/IP client binds to a port and listens for unsolicited frames, such as lifeline reporting. Then, a define indicates whether to use IPv4 or IPv6. The address of the Z/IP Gateway is the address that Z-Ware should connect to. Then, there are several callback functions, as follows:

• The notify function that notifies about network operations in progress.

• The node callback that informs about changes to the nodes in the network.

• The application TX which has information about the status of outgoing frame, so when transmitting a frame, the application can get information about whether the frame was delivered.

Some functions in bold relate to the portal mode, with Z-Ware typically running in the cloud and Z/IP Gateway connected through a TLS tunnel. These functions are mainly related to the portal connection and some SSL functions. A file descriptor contains information about the certificates required for the SSL link. When installing the Web portal, the installer prompts for this information and subsequently populates the structure. As a result, when using the portal, these are handled automatically.

Next, a pointer references a directory where Z-Ware saves information about the network. This structure is not quite up to date, as it is shown as a .dat, while the latest release has it changed to a json file that is human readable. Then, there is the device configuration which is covered later. Z-Ware has a device database, which can be used to manipulate how a device behaves in a Z-Wave network. It also has the option to disable command classes, which is typically not used. The next entries are not described.

Then, there is the S2 unsolicited callback, typically used Z-Wave networks with multiple controllers, where Z-Ware is the SIS. In this case, the other controllers in the network, the inclusion controllers, may include a node and hand over the security key exchange to Z-Ware. In this case, Z-Ware will relay some information about the key exchange to the application, which is promoted to the user to prompt input of required information.

Finally, some information is required when using Z-Ware running in local mode, as in the lab exercise, where it relies on the DTLS key.

Network Initialization, /zw_gw_set, REST API#

As outlined above, network initialization is a lot simpler when using the Z-Ware Web portal because a lot of this has been configured automatically by the Z-Ware daemon. When doing the same initialization on the Z-Ware Web portal, it looks a lot simpler, like this:

zip_gw_name={zip_gw_name}&unsol_rpt_port={unsol_rpt_port}&key={preshared_key}

Z-Ware is called, providing a gateway name, URL, or an IP address of Z/IP Gateway, and an unsolicited destination port and the DTLS key. All remaining entries are provided by the Z-Ware Web daemon.

Polling for Network Status on REST API, /zwnet_get_operation#

While the C-API has an option to configure required callbacks, the Web portal lacks this capability. To receive this information at runtime, the Z-Ware client needs to poll zwnet_get_operation at a relatively short interval. This function returns the same information as multiple C-API callback functions: the notify, node, application TX and S2 unsolicited callbacks. If taking a look at the top of the list, the options are the same as the callbacks. The Z-Ware Web daemon provides information about: adding nodes into the network, network operations, status of the nodes, and S2 bootstrapping. In other words, the Web daemon can obtain the same information as the Z-Ware C-API, but the client is required to poll Z-Ware because Z-Ware can't provide callbacks.

| net_cmd_type/prev ||ZWNET_OP_XXX |# |Description || NONE |0 |No operation is executing || INITIALIZE |1 |Initialization operation || ADD_NODE |2 |Add node operation || RM_NODE |3 |Remove node operation || RP_NODE |4 |Replace failed node operation || RM_FAILED_ID |5 |Remove failed node operation || INITIATE |6 |Initiation operation by controller || UPDATE |7 |Update network topology from the SUC/SIS || RESET |8 |Restore to factory default setting || MIGRATE_SUC |9 |Create primary controller by a SUC || MIGRATE |10 |Migrate primary controller operation || LOAD_NW_INFO |12 |Load network information || NODE_UPDATE |13 |Update node info || SEND_NIF |14 |Send node information frame || ADD_NODE_ON_BEHALF |15 |Add node on-half operation (For /zwnet_add Web API only. Operation status will still be ZWNET_OP_ADD_NODE) || RP_NODE_ON_BEHALF |16 |Replace failed node on-half operation (For /zwnet_fail Web API only. Operation status will still be ZWNET_OP_RP_NODE) || FW_UPDT |19 |Firmware Update (For CMD_FIRMWARE_UPDATE_REQ_GET Web A only) || HEALTH_CHK |20 |Network health check || NODE_RESET |21 |Process node reset locally notification || FW_DOWNLD |22 |Firmware Backup (For CMD_FIRMWARE_BACKUP_REQ_GET Web API only) | |||net_cmd_status Progress status of the ongoing network operation. Some are generic while others are specific to the operation. | net_cmd_status ||# |Description || OP_STS_NONE |0 |No status to report || OP_STS_ERROR |-1 |Error || OP_STS_NO_NET |-4 |Network uninitialized || OP_STS_ABORTED |-5 | Network operation aborted || OP_ADD_NODE_xxx ||Add node status || PROTOCOL_DONE |1 |Protocol part of adding node done || GET_NODE_INFO |2 |Getting node detailed information || PROTOCOL_START |3 |Smart Start add node Z-Wave protocol started || SEC_REQ_KEY_READY |11 |(S2 inclusion only) Device requested key info is ready. Client may use /zwnet_add_s2_get_req_keys to query the requested key information. || SEC_DSK_READY |12 |(S2 inclusion only) Device DSK info is ready. Client may use /zwnet_add_s2_get_dsk to query for the DSK information. || ON_BEHALF_SEC_REQ_KEY_READY |21 |(S2 inclusion only) Device requested key info is ready. Client may use /zwnet_add_s2_get_req_keys to query the requested key information. || ON_BEHALF_SEC_DSK_READY |22 |(S2 inclusion only) Device DSK info is ready. Client may use /zwnet_add_s2_get_dsk to query for the DSK information. || OP_RM_NODE_XXX ||Remove node status || LEARN_READY |1 |Ready to remove a node || FOUND |2 |Found a node || REMOVING |3 |Removing the node || OP_RP_NODE_XXX || Replace node status || READY 1 Ready to replace a node || PROTOCOL_DONE |2 |Protocol part of replacing node done || SEC_INCD |3 |Adding node securely || GET_NODE_INFO |4 |Getting node detailed information || OP_INI_XXX ||Initiate status || STARTED |1 |Initiating started, ready to be added/removed to/from network || PROTOCOL_DONE |2 |Protocol part of initiating done || SEC_INCD |3 |Trying to be included securely || GET_NODE_INFO |4 |Getting node detailed information || SEC_OWN_DSK_READY |11 |(S2 inclusion only) Local node DSK info is ready. Client may use /zwnet_initiate_local_dsk_get to query for the DSK information. || OP_NU_XXX ||Network update status || TOPOLOGY |1 |Network topology update started || NEIGHBOR |2 |Node neighbor update started || GET_NODE_INFO |3 |Node information update started || OP_FW_XXX ||Firmware update status || UPLOAD_STARTED |1 |Uploading firmware to device started || UPLOADING |2 |Uploading firmware to device in progress with additional percentage info in op_total_nodes and op_cmplt_nodes || OP_FW_XXX ||Firmware download/backup status || DOWNLOAD_STARTED |1 |Download/backup firmware from device started || DOWNLOADING |2 |Download/backup firmware from device in progress with additional percentage info in op_total_nodes and op_cmplt_nodes || OP_HEALTH_CHK_XXX|| Network health check status || STARTED |1 |Network health check started || PROGRESS |2 |Network health check in progress || CMPLT |3 |Network health check completed

Helper functions, Gateway Discovery#

When initializing the network, the Z-Ware Web portal relies on helper functions. For example, the Z-Ware interface shows a list of Z/IP Gateways in the network. For this purpose, the Z-Ware C-API has the function zwnet_gw_discvr_start and an associated ..._stop function that start and stop gateway discovery. Z-Ware can perform Z/IP Gateway discovery using both mDNS and broadcast methodology based on the parameters given to the functions, and of course the Z-Ware Web interface has a similar function.

zwnet_gw_discvr_start

Callbacks, zwnet_notify_fn#

Returning to the topic of callback functions, now examine the notify callback in detail. This callback receives status of network operations in progress and provides status on what Z-Ware (and Z/IP Gateway) are doing. This callback indicates state information such as: none (no ongoing operations); whether nodes are added, or removed; and if network updates are ongoing, and so on. These callbacks provide confirmation when initiating network management operations.

Callbacks, zwnet_node_fn#

Next, examine the node callback, which provides status for individual nodes. This callback indicates if node was added or removed from the network, and is typically encountered when a node becomes unreachable. For always-on nodes that typically have the status "alive," the status will change to "down" if for some reason transmissions fail. This allows the application to register that the node is currently unresponsive and there might be an error in the network. Similarly, for battery-powered sensor nodes, a "sleeping" status is typically returned. No notifications are provided when sleeping devices wake up, but the status will eventually change to "down" if a node does not wake up within the wake-up interval.

Callbacks, zwnet_appl_fn#

The appl or "application TX" callback provides status of frame transmissions. When transmitting to a Z-Wave device, information about the transmission status is provided, such as: if the transmission was successful, no ACK was received from the destination, or if the frame was queued for a sleeping destination.

Callbacks, s2_unsolicited_cb#

The S2 unsolicited callback is called when another controller has added a node to the network, indicating that the Z-Ware based controller, as the SIS in the network, needs to grant this device S2 keys. This callback allows a client application in an advanced configuration with multiple controllers, such as the Web Portal, to display a pop up prompting the user to input the device DSK and grant or deny the joining device its requested network keys.

Setting Unsolicited Destination#

The API for setting the unsolicited destination is essential for managing unsolicited traffic and lifeline reporting, such as notifications or sensor reports from nodes in the network. The API is called with the listening port, specified when initializing the network, and the IP address of the host running Z-Ware. Because Z/IP Gateway runs its own network stack, the unsolicited address is often confused with Z/IP Gateway's, which should remain separate. Effectively, the unsolicited addres is the IP address of Z-Ware, although the two hosts may be running on the same piece of physical hardware.

zwif_gw_unsolicit_set

Helper Functions Setting Unsolicited Destination#

Two helper functions are used to configure the unsolicited destination: local_address_get and zwnet_listen_port. The first, local_address_get, returns the IP address currently in use to communicate with Z/IP Gateway. If the host has multiple IP addresses on multiple interfaces, this API will return the address used to communicate with the Z/IP Gateway, which is by default the correct address. The second, zwnet_listen_port, returns the port specified when initializing Z-Ware.

zwnet_local_addr_get
zwnet_listen_port_get

Structure#

An application evaluate the Z-Wave network structure either network-first or endpoint first.

Beginning network-first, Z-Ware provides a handle to the network, and the application can use zwnet_get_node to get a handle to the first node in the network, which will typically, but not always, be the Z/IP Gateway itself. With the handle to the first node, the application can then iterate through the nodes in the network by using zwnode_get_next to get the next node, and so on.

From each node, the application can then use zwnode_get_ep, which will get the first endpoint on a node and then can then iterate through the remaining endpoints by using zwep_get_next. If, for example, the second node in the figure has multiple endpoints, the application can use zwnode_get_ep to get the first endpoint and then zwep_get_next to get the next endpoint, and so on.

zwep_if_get is an API to get the first handle to an interface or command class on the endpoint. To get the next interface, call zwif_get_next.

This allows the application to iterate through nodes and, for each node, through endpoints and, for each endpoint, through the command class. In this way, it can build a full Z-Wave network tree in application with all controllable resources. It works slightly differently on the Web interface using the lab interface on the Z-Ware Web portal but the Z-Ware C-API sample apps that work like this are covered in next session.

• Network-first traversal

  • zwnet_get_node

  • zwnode_get_next

  • zwnode_get_ep

  • zwep_get_next

  • zwep_get_if

  • zwif_get_next

Alternatively, with a handle to an interface and a command class, the network can be traversed backwards. Starting with an interface, use zwif_get_ep to get the endpoint relating to the interface, then use zwep_get_node to get the node related to the interface, and zwnode_get_net, to get the net that is hosting the node that is hosting the endpoint, that is hosting the interface.

• Endpoint-first traversal

  • zwif_get_ep

  • zwep_get_node

  • zwnode_get_net

Getting Information about Nodes (zwnoded_t) and Endpoints (zwepd_t)#

For each Z-Wave node in the network, a handle details the Z-Wave node and its capabilities. If watching the Zniffer while including nodes with Z-Ware, a lot of traffic follows the initial inclusion, as Z-Ware performs the mandatory interview on all supported command classes and populates a data structure containing this information about node. This interview is especially important for battery-powered nodes because Z-Ware will not be able to interview the device later when the device is sleeping. However, even for mains powered nodes, you must have the data cached.

This is the data structure for a Z-Wave node. At the top is the Z-Wave node ID in the Z-Wave network. Then, there are some properties that are not described here. The vendor ID, vendor type, device ID this are collected from the device during inclusion. The same goes for the Z-Wave protocol version, the application version, and the library type. Flags specify whether the device is battery-powered and whether it is FLIRS or a sleeping device. Additional version numbers can be stored on the device. Flags indicate which, if any, S2 keys have been granted to the device. Finally, the DSK of the node is a unique identifier.

Similarly, each endpoint has a corresponding data structure, zwepd, which has information about: the specific and generic device classes; application level functionality of the endpoint; the endpoint number, and the node ID; and whether aggregated endpoints are present.

Registering Callbacks, Example for Basic, zwif_basic_rpt_set#

This section contains an example on registering a callback for the basic command class. The application calls zwif_basic_rpt_set and provides a pointer to the callback function. Returned data contains the data value and a timestamp that indicates when data was received, and a state number that is incremented whenever the cached value changes.

Callback Limitations#

A callback is triggered only when incoming data is different from cached data. However, the timestamps on the cached data are always updated for each incoming frame.

Consider a sensor sending temperature readings at a fixed interval. When the temperature does not change between readings, a received report may not trigger a callback, but the timestamp is still updated for each incoming frame.

• Z-Ware caches all incoming data

• Callbacks trigger only when incoming data mismatches cached data

Polling, Example for Basic, C-api zwif_basic_get#

Z-Ware allows polling an interface, where the application actively requests data from an interface. Polling has two modes, “cache," and "live."

In the "cache" get mode, Z-ware provides a callback with the latest cached value and timestamp for the interface and nothing transmitted to the Z-Wave device.

For a “live” get mode, Z-Ware actively transmits a get command to the device and the device responds with a report. Note that the “live” get updates the timestamp of the cached value, but only provides a callback if the interface value was different from the value already cached.

As a result, in order to to get the absolute latest value from a device, first do an “active” get to update the cached value and then do a “cache” get to force a callback.

• Values are returned through registered callback functions

• Two modes:

  • ZWIF_GET_BMSK_CACHE

    • Does not actively get value from device

    • Always returns a callback

  • ZWIF_GET_BMSK_LIVE

    • Actively request the value from the end device

    • Only returns a callback if the value is different from the cached value

    • Time stamps are always updated

Device Database#

Z-Ware includes a device database. The device database allows modifications how a Z-Wave device appears to the application built on top of Z-Ware. For a legacy device, which only supports basic command class, you can remap basic command class into something more meaningful for the device. Basic command class could, for example, be mapped to the alarm interface. As a result, seen from the application level, the device would appear like a modern Z-Wave Plus a notification sensor, sending notification command class, while in fact being a very old device with very limited command class functionality. The catch is that the developer needs to know the devices that are intended to be included in the network in advance. This limits use and functionality of the database, which is generally most used by customers like service providers that have a large install base, and know which devices will be deployed in the network. It is prohibitive to populate the database for all Z-Wave devices that have been made historically. The release note for Z-Ware has a list of devices with which Z-Ware has been tested and known to work with. In general, the device database is populated with these devices to provide the best possible functionality. The programmers guide has a large example on the Z-Ware device database, showing how to remap command classes and manipulate Z-Wave devices in other ways to make it appear more uniform for the application.

• Allows reconfiguring a device based on information that cannot be polled from the device itself.

• Based on manufacturer and device ID

  • Existing command classes can be:

    • Modified

    • Remapped to other command classes