Session 6: Inner Workings of Z-Ware#
The previous session examined the Z-Ware portal and how it is included in the SDK. The Z-Ware portal is a Z-Wave certified controller. It is used internally for software qualification of not only Z-Ware itself, but also of the Z/IP Gateway, a proof of concept illustrating that the Z-Wave controller is fully Z-Wave certifiable. However, Z-Ware can also serve as the foundation of a custom controller. To explore Z-ware in this capacity, this session covers a part of the C-API, some of the central APIs, and the general structure of Z-Ware. Additionally, it covers some of the similar APIs on the HTTPS REST API. Finally, it includes an exercise in enumerating a Z-Wave network and exercising control using the REST API instead of the Portal interface.
Network Initialization, zwnet_init, c-api#
The first requirement for a Z-Ware based controller is to initialize the network. With initialization, Z-Ware connects to the Z/IP Gateway and configures the required connections. The Z-Ware C-API populates an input structure and then passes it to the API. The network initialization,
zwnet_init, has this input structure:
User context used in callbacks.
Host listening and sending port.
Flag to indicate whether to use IPv4 as transport IP protocol. 1=use IPv4; 0=use IPv6.
ZIPGW (gateway) IPv4/IPv6 address in numeric format.
Network operation notification.
Node add/delete/status callback.
Application transmit data status callback.
Unsolicited included node information frame (INIF) callback when a Smart Start device (which has joined a foreign network but is listed in the local provisioning list) is powering up.
Unhandled command callback.
Print text function pointer.
Network file descriptor to connect to ZIPGW using TLS.
SSL object pointer to connect to ZIPGW using TLS.
Display context for the print_txt_fn.
Unrecoverable network error callback, application should close this instance of Z-Ware object, i.e. call zwnet_exit().
Profile of the ZIPGW that is connected to the portal.
const char *
Full path of directory for storing network and node information file (a.k.a. network persistent storage). The file is generated by the library with filename in the format “nifXXXXXXXX.dat”, where XXXXXXXX denotes Z-Wave network home ID.
const char *
Full path of directory for storing network/user preference files.
const char *
Device-specific configurations file (a.k.a device database) in JSON format. If it is NULL, device-specific configurations will be managed by user application. In this case, dev_cfg_usr must be valid.
const char *
Optional CC configuration file. Enable specific CC probing after a new node inclusion and during background polling. If NULL, ALL supported CCs are enabled. The current supported CC configuration file is distributed in the “config” folder of the source distribution with the name “cmd_class.cfg”.
Device-specific configurations (managed by user application). If it is NULL, device-specific configurations will be managed by Z-Ware Library internally. In this case, dev_cfg_file must be valid.
Error location while parsing device-specific configuration file. Note: The error is not due to JSON format parsing error.
User application implemented CCs.NOTE: if the controller has already implemented the CC, the user's request for that CC will be ignored.
User application implemented CCs count.
Callback to report unsolicited joining device requested keys and/or status of Device Specific Key (DSK)
DTLS pre-shared key in binary format, MUST be at least 16 bytes.
DTLS pre-shared key length (bytes). If length is zero, no DTLS will be used, i.e., communication will be unsecured.
Some of the values are familiar because they are also exposed on the Web interface and used by other Z/IP clients. The host_port is the port which Z-Ware is listening to. The unsolicited destination, discussed previously, is where the Z/IP client binds to a port and listens for unsolicited frames, such as lifeline reporting. Then, a define indicates whether to use IPv4 or IPv6. The address of the Z/IP Gateway is the address that Z-Ware should connect to. Then, there are several callback functions, as follows:
The notify function that notifies about network operations in progress.
The node callback that informs about changes to the nodes in the network.
The application TX which has information about the status of outgoing frame, so when transmitting a frame, the application can get information about whether the frame was delivered.
Some functions in bold relate to the portal mode, with Z-Ware typically running in the cloud and Z/IP Gateway connected through a TLS tunnel. These functions are mainly related to the portal connection and some SSL functions. A file descriptor contains information about the certificates required for the SSL link. When installing the Web portal, the installer prompts for this information and subsequently populates the structure. As a result, when using the portal, these are handled automatically.
Next, a pointer references a directory where Z-Ware saves information about the network. This structure is not quite up to date, as it is shown as a .dat, while the latest release has it changed to a json file that is human readable. Then, there is the device configuration which is covered later. Z-Ware has a device database, which can be used to manipulate how a device behaves in a Z-Wave network. It also has the option to disable command classes, which is typically not used. The next entries are not described.
Then, there is the S2 unsolicited callback, typically used Z-Wave networks with multiple controllers, where Z-Ware is the SIS. In this case, the other controllers in the network, the inclusion controllers, may include a node and hand over the security key exchange to Z-Ware. In this case, Z-Ware will relay some information about the key exchange to the application, which is promoted to the user to prompt input of required information.
Finally, some information is required when using Z-Ware running in local mode, as in the lab exercise, where it relies on the DTLS key.
Network Initialization, /zw_gw_set, REST API#
As outlined above, network initialization is a lot simpler when using the Z-Ware Web portal because a lot of this has been configured automatically by the Z-Ware daemon. When doing the same initialization on the Z-Ware Web portal, it looks a lot simpler, like this:
Z-Ware is called, providing a gateway name, URL, or an IP address of Z/IP Gateway, and an unsolicited destination port and the DTLS key. All remaining entries are provided by the Z-Ware Web daemon.
Polling for Network Status on REST API, /zwnet_get_operation#
While the C-API has an option to configure required callbacks, the Web portal lacks this capability. To receive this information at runtime, the Z-Ware client needs to poll zwnet_get_operation at a relatively short interval. This function returns the same information as multiple C-API callback functions: the notify, node, application TX and S2 unsolicited callbacks. If taking a look at the top of the list, the options are the same as the callbacks. The Z-Ware Web daemon provides information about: adding nodes into the network, network operations, status of the nodes, and S2 bootstrapping. In other words, the Web daemon can obtain the same information as the Z-Ware C-API, but the client is required to poll Z-Ware because Z-Ware can't provide callbacks.
current network operation
| net_cmd_type/prev ||ZWNET_OP_XXX |# |Description || NONE |0 |No operation is executing || INITIALIZE |1 |Initialization operation || ADD_NODE |2 |Add node operation || RM_NODE |3 |Remove node operation || RP_NODE |4 |Replace failed node operation || RM_FAILED_ID |5 |Remove failed node operation || INITIATE |6 |Initiation operation by controller || UPDATE |7 |Update network topology from the SUC/SIS || RESET |8 |Restore to factory default setting || MIGRATE_SUC |9 |Create primary controller by a SUC || MIGRATE |10 |Migrate primary controller operation || LOAD_NW_INFO |12 |Load network information || NODE_UPDATE |13 |Update node info || SEND_NIF |14 |Send node information frame || ADD_NODE_ON_BEHALF |15 |Add node on-half operation (For /zwnet_add Web API only. Operation status will still be ZWNET_OP_ADD_NODE) || RP_NODE_ON_BEHALF |16 |Replace failed node on-half operation (For /zwnet_fail Web API only. Operation status will still be ZWNET_OP_RP_NODE) || FW_UPDT |19 |Firmware Update (For CMD_FIRMWARE_UPDATE_REQ_GET Web A only) || HEALTH_CHK |20 |Network health check || NODE_RESET |21 |Process node reset locally notification || FW_DOWNLD |22 |Firmware Backup (For CMD_FIRMWARE_BACKUP_REQ_GET Web API only) | |||net_cmd_status Progress status of the ongoing network operation. Some are generic while others are specific to the operation. | net_cmd_status ||# |Description || OP_STS_NONE |0 |No status to report || OP_STS_ERROR |-1 |Error || OP_STS_NO_NET |-4 |Network uninitialized || OP_STS_ABORTED |-5 | Network operation aborted || OP_ADD_NODE_xxx ||Add node status || PROTOCOL_DONE |1 |Protocol part of adding node done || GET_NODE_INFO |2 |Getting node detailed information || PROTOCOL_START |3 |Smart Start add node Z-Wave protocol started || SEC_REQ_KEY_READY |11 |(S2 inclusion only) Device requested key info is ready. Client may use /zwnet_add_s2_get_req_keys to query the requested key information. || SEC_DSK_READY |12 |(S2 inclusion only) Device DSK info is ready. Client may use /zwnet_add_s2_get_dsk to query for the DSK information. || ON_BEHALF_SEC_REQ_KEY_READY |21 |(S2 inclusion only) Device requested key info is ready. Client may use /zwnet_add_s2_get_req_keys to query the requested key information. || ON_BEHALF_SEC_DSK_READY |22 |(S2 inclusion only) Device DSK info is ready. Client may use /zwnet_add_s2_get_dsk to query for the DSK information. || OP_RM_NODE_XXX ||Remove node status || LEARN_READY |1 |Ready to remove a node || FOUND |2 |Found a node || REMOVING |3 |Removing the node || OP_RP_NODE_XXX || Replace node status || READY 1 Ready to replace a node || PROTOCOL_DONE |2 |Protocol part of replacing node done || SEC_INCD |3 |Adding node securely || GET_NODE_INFO |4 |Getting node detailed information || OP_INI_XXX ||Initiate status || STARTED |1 |Initiating started, ready to be added/removed to/from network || PROTOCOL_DONE |2 |Protocol part of initiating done || SEC_INCD |3 |Trying to be included securely || GET_NODE_INFO |4 |Getting node detailed information || SEC_OWN_DSK_READY |11 |(S2 inclusion only) Local node DSK info is ready. Client may use /zwnet_initiate_local_dsk_get to query for the DSK information. || OP_NU_XXX ||Network update status || TOPOLOGY |1 |Network topology update started || NEIGHBOR |2 |Node neighbor update started || GET_NODE_INFO |3 |Node information update started || OP_FW_XXX ||Firmware update status || UPLOAD_STARTED |1 |Uploading firmware to device started || UPLOADING |2 |Uploading firmware to device in progress with additional percentage info in op_total_nodes and op_cmplt_nodes || OP_FW_XXX ||Firmware download/backup status || DOWNLOAD_STARTED |1 |Download/backup firmware from device started || DOWNLOADING |2 |Download/backup firmware from device in progress with additional percentage info in op_total_nodes and op_cmplt_nodes || OP_HEALTH_CHK_XXX|| Network health check status || STARTED |1 |Network health check started || PROGRESS |2 |Network health check in progress || CMPLT |3 |Network health check completed
Helper functions, Gateway Discovery#
When initializing the network, the Z-Ware Web portal relies on helper functions. For example, the Z-Ware interface shows a list of Z/IP Gateways in the network. For this purpose, the Z-Ware C-API has the function
zwnet_gw_discvr_start and an associated
..._stop function that start and stop gateway discovery. Z-Ware can perform Z/IP Gateway discovery using both mDNS and broadcast methodology based on the parameters given to the functions, and of course the Z-Ware Web interface has a similar function.
Returning to the topic of callback functions, now examine the
notify callback in detail. This callback receives status of network operations in progress and provides status on what Z-Ware (and Z/IP Gateway) are doing. This callback indicates state information such as: none (no ongoing operations); whether nodes are added, or removed; and if network updates are ongoing, and so on. These callbacks provide confirmation when initiating network management operations.
Initializing local Z/IP controller.
Adding a node.
Removing a node.
Replacing a failed node.
Removing a failed node.
Initiating in response to or anticipation of some operation.
Updating network topology from SUC/SIS.
Restoring ZIPGW attached controller to factory default settings.
Updating node information.
Migrating primary controller.
Sending node information frame.
Network change detection.
Update node cached info. (Internally used; application won’t receive this notification.)
Save network and node information to persistent storage. (Internally used; application won’t receive this notification.)
Sleeping device node information polling. (Internally used; application won’t receive this notification.)
Network health check.
Remove node which has been reset.
Next, examine the
node callback, which provides status for individual nodes. This callback indicates if node was added or removed from the network, and is typically encountered when a node becomes unreachable. For always-on nodes that typically have the status "alive," the status will change to "down" if for some reason transmissions fail. This allows the application to register that the node is currently unresponsive and there might be an error in the network. Similarly, for battery-powered sensor nodes, a "sleeping" status is typically returned. No notifications are provided when sleeping devices wake up, but the status will eventually change to "down" if a node does not wake up within the wake-up interval.
Node was added. Application can retrieve detailed node information now.
Node was removed.
Node was updated. Application can retrieve updated detailed node information now.
Node status has changed to "alive"
Node status has changed to "down"
Node status has changed to "sleeping"
Received notification from node that it has been reset
appl or "application TX" callback provides status of frame transmissions. When transmitting to a Z-Wave device, information about the transmission status is provided, such as: if the transmission was successful, no ACK was received from the destination, or if the frame was queued for a sleeping destination.
Send frame timeout due to no ACK received.
System error, the program should exit or restart.
Message has not timed out yet. The destination host may have a long response time (e.g., sleeping node).
Frame failed to reach destination host.
S2 unsolicited callback is called when another controller has added a node to the network, indicating that the Z-Ware based controller, as the SIS in the network, needs to grant this device S2 keys. This callback allows a client application in an advanced configuration with multiple controllers, such as the Web Portal, to display a pop up prompting the user to input the device DSK and grant or deny the joining device its requested network keys.
Setting Unsolicited Destination#
The API for setting the unsolicited destination is essential for managing unsolicited traffic and lifeline reporting, such as notifications or sensor reports from nodes in the network. The API is called with the listening port, specified when initializing the network, and the IP address of the host running Z-Ware. Because Z/IP Gateway runs its own network stack, the unsolicited address is often confused with Z/IP Gateway's, which should remain separate. Effectively, the unsolicited addres is the IP address of Z-Ware, although the two hosts may be running on the same piece of physical hardware.
Helper Functions Setting Unsolicited Destination#
Two helper functions are used to configure the unsolicited destination:
zwnet_listen_port. The first,
local_address_get, returns the IP address currently in use to communicate with Z/IP Gateway. If the host has multiple IP addresses on multiple interfaces, this API will return the address used to communicate with the Z/IP Gateway, which is by default the correct address. The second,
zwnet_listen_port, returns the port specified when initializing Z-Ware.
An application evaluate the Z-Wave network structure either network-first or endpoint first.
Beginning network-first, Z-Ware provides a handle to the network, and the application can use
zwnet_get_node to get a handle to the first node in the network, which will typically, but not always, be the Z/IP Gateway itself. With the handle to the first node, the application can then iterate through the nodes in the network by using
zwnode_get_next to get the next node, and so on.
From each node, the application can then use
zwnode_get_ep, which will get the first endpoint on a node and then can then iterate through the remaining endpoints by using
If, for example, the second node in the figure has multiple endpoints, the application can use
zwnode_get_ep to get the first endpoint and then
zwep_get_next to get the next endpoint, and so on.
zwep_if_get is an API to get the first handle to an interface or command class on the endpoint. To get the next interface, call
This allows the application to iterate through nodes and, for each node, through endpoints and, for each endpoint, through the command class. In this way, it can build a full Z-Wave network tree in application with all controllable resources. It works slightly differently on the Web interface using the lab interface on the Z-Ware Web portal but the Z-Ware C-API sample apps that work like this are covered in next session.
Alternatively, with a handle to an interface and a command class, the network can be traversed backwards. Starting with an interface, use
zwif_get_ep to get the endpoint relating to the interface, then use
zwep_get_node to get the node related to the interface, and
zwnode_get_net, to get the net that is hosting the node that is hosting the endpoint, that is hosting the interface.
Getting Information about Nodes (zwnoded_t) and Endpoints (zwepd_t)#
For each Z-Wave node in the network, a handle details the Z-Wave node and its capabilities. If watching the Zniffer while including nodes with Z-Ware, a lot of traffic follows the initial inclusion, as Z-Ware performs the mandatory interview on all supported command classes and populates a data structure containing this information about node. This interview is especially important for battery-powered nodes because Z-Ware will not be able to interview the device later when the device is sleeping. However, even for mains powered nodes, you must have the data cached.
Z-Wave Node ID
Properties of the node (bit-mask): NODE_PROPTY_XXX.
Z-Wave Vendor ID.
Z-Wave Product ID.
Z-Wave Protocol Version.
Z-Wave Library Type.
Device category, DEV_XXX.
Flag to indicate whether the node is a sensor (FLIRS).
Flag to indicate the node is capable to sleep (i.e. non-listening and support Wake up CC).
Flag to indicate the node is always listening.
Number of valid Z-Wave software version types stored in the zwsw_ver array.
Z-Wave software version information.
Flag to indicate whether s2_grnt_keys is valid.
Security 2 granted keys (bit-mask), see SEC_KEY_BITMSK_XXX
NOTE: This is valid only s2_keys_valid = 1.
S2 DSK. If s2_dsk == '\0', the DSK is unavailable for this node.
This is the data structure for a Z-Wave node. At the top is the Z-Wave node ID in the Z-Wave network. Then, there are some properties that are not described here. The vendor ID, vendor type, device ID this are collected from the device during inclusion. The same goes for the Z-Wave protocol version, the application version, and the library type. Flags specify whether the device is battery-powered and whether it is FLIRS or a sleeping device. Additional version numbers can be stored on the device. Flags indicate which, if any, S2 keys have been granted to the device. Finally, the DSK of the node is a unique identifier.
Similarly, each endpoint has a corresponding data structure, zwepd, which has information about: the specific and generic device classes; application level functionality of the endpoint; the endpoint number, and the node ID; and whether aggregated endpoints are present.
Z-Wave Generic device class
Z-Wave Specific device class
Total number of end point members that are represented by this aggregated end point. Zero means this endpoint is NOT an aggregated end point
Buffer to store the end point members that are represented by this aggregated end point
User configured name string of the endpoint
User configured location string of the endpoint
Z-Wave Plus information CC
Registering Callbacks, Example for Basic, zwif_basic_rpt_set#
This section contains an example on registering a callback for the basic command class. The application calls
zwif_basic_rpt_set and provides a pointer to the callback function. Returned data contains the data value and a timestamp that indicates when data was received, and a state number that is incremented whenever the cached value changes.
Basic interface handle.
Report callback function.
ZW_ERR_NONE on success; else ZW_ERR_XXX.
Time stamp. If this is zero, the callback has no data and hence other parameter values should be ignored.
State number that is incremented by one whenever a cache change is detected
A callback is triggered only when incoming data is different from cached data. However, the timestamps on the cached data are always updated for each incoming frame.
Consider a sensor sending temperature readings at a fixed interval. When the temperature does not change between readings, a received report may not trigger a callback, but the timestamp is still updated for each incoming frame.
Z-Ware caches all incoming data
Callbacks trigger only when incoming data mismatches cached data
Polling, Example for Basic,
Z-Ware allows polling an interface, where the application actively requests data from an interface. Polling has two modes, “cache," and "live."
In the "cache" get mode, Z-ware provides a callback with the latest cached value and timestamp for the interface and nothing transmitted to the Z-Wave device.
For a “live” get mode, Z-Ware actively transmits a get command to the device and the device responds with a report. Note that the “live” get updates the timestamp of the cached value, but only provides a callback if the interface value was different from the value already cached.
As a result, in order to to get the absolute latest value from a device, first do an “active” get to update the cached value and then do a “cache” get to force a callback.
Values are returned through registered callback functions
Does not actively get value from device
Always returns a callback
Actively request the value from the end device
Only returns a callback if the value is different from the cached value
Time stamps are always updated
Z-Ware includes a device database. The device database allows modifications how a Z-Wave device appears to the application built on top of Z-Ware. For a legacy device, which only supports basic command class, you can remap basic command class into something more meaningful for the device. Basic command class could, for example, be mapped to the alarm interface. As a result, seen from the application level, the device would appear like a modern Z-Wave Plus a notification sensor, sending notification command class, while in fact being a very old device with very limited command class functionality. The catch is that the developer needs to know the devices that are intended to be included in the network in advance. This limits use and functionality of the database, which is generally most used by customers like service providers that have a large install base, and know which devices will be deployed in the network. It is prohibitive to populate the database for all Z-Wave devices that have been made historically. The release note for Z-Ware has a list of devices with which Z-Ware has been tested and known to work with. In general, the device database is populated with these devices to provide the best possible functionality. The programmers guide has a large example on the Z-Ware device database, showing how to remap command classes and manipulate Z-Wave devices in other ways to make it appear more uniform for the application.
Allows reconfiguring a device based on information that cannot be polled from the device itself.
Based on manufacturer and device ID
Existing command classes can be:
Remapped to other command classes