Session 6: Inner Workings of Z-Ware#

The previous session examined the Z-Ware portal and how it is included in the SDK. The Z-Ware portal is a Z-Wave certified controller. It is used internally for software qualification of not only Z-Ware itself, but also of the Z/IP Gateway, a proof of concept illustrating that the Z-Wave controller is fully Z-Wave certifiable. However, Z-Ware can also serve as the foundation of a custom controller. To explore Z-ware in this capacity, this session covers a part of the C-API, some of the central APIs, and the general structure of Z-Ware. Additionally, it covers some of the similar APIs on the HTTPS REST API. Finally, it includes an exercise in enumerating a Z-Wave network and exercising control using the REST API instead of the Portal interface.

Network Initialization, zwnet_init, c-api#

The first requirement for a Z-Ware based controller is to initialize the network. With initialization, Z-Ware connects to the Z/IP Gateway and configures the required connections. The Z-Ware C-API populates an input structure and then passes it to the API. The network initialization, zwnet_init, has this input structure:

Attribute

Type

I/O

Description

user

void *

I

User context used in callbacks.

host_port

uint16_t

I

Host listening and sending port.

use_ipv4

Int

I

Flag to indicate whether to use IPv4 as transport IP protocol. 1=use IPv4; 0=use IPv6.

zip_router

uint8_t[16]

I

ZIPGW (gateway) IPv4/IPv6 address in numeric format.

notify

zwnet_notify_fn

I

Network operation notification.

node

zwnet_node_fn

I

Node add/delete/status callback.

appl_tx

zwnet_appl_fn

I

Application transmit data status callback.

inif_cb

pl_info_fn

I

Unsolicited included node information frame (INIF) callback when a Smart Start device (which has joined a foreign network but is listed in the local provisioning list) is powering up.

unhandled_cmd

zwnet_unhandled_cmd_fn

I

Unhandled command callback.

print_txt_fn

print_fn

I

Print text function pointer.

portal_fd#

int

I

Network file descriptor to connect to ZIPGW using TLS.

portal_ssl#

void *

I

SSL object pointer to connect to ZIPGW using TLS.

display_ctx#

void *

I

Display context for the print_txt_fn.

net_err#

zwnet_net_err_fn

I

Unrecoverable network error callback, application should close this instance of Z-Ware object, i.e. call zwnet_exit().

portal_prof#

clnt_prof_t

I

Profile of the ZIPGW that is connected to the portal.

net_info_dir

const char *

I

Full path of directory for storing network and node information file (a.k.a. network persistent storage). The file is generated by the library with filename in the format “nifXXXXXXXX.dat”, where XXXXXXXX denotes Z-Wave network home ID.

pref_dir

const char *

I

Full path of directory for storing network/user preference files.

dev_cfg_file

const char *

I

Device-specific configurations file (a.k.a device database) in JSON format. If it is NULL, device-specific configurations will be managed by user application. In this case, dev_cfg_usr must be valid.

cmd_cls_cfg_file

const char *

I

Optional CC configuration file. Enable specific CC probing after a new node inclusion and during background polling. If NULL, ALL supported CCs are enabled. The current supported CC configuration file is distributed in the “config” folder of the source distribution with the name “cmd_class.cfg”.

dev_cfg_usr

dev_cfg_usr_t *

I

Device-specific configurations (managed by user application). If it is NULL, device-specific configurations will be managed by Z-Ware Library internally. In this case, dev_cfg_file must be valid.

err_loc

dev_cfg_error_t

O

Error location while parsing device-specific configuration file. Note: The error is not due to JSON format parsing error.

sup_cmd_cls

sup_cmd_cls_t *

I

User application implemented CCs.NOTE: if the controller has already implemented the CC, the user's request for that CC will be ignored.

sup_cmd_cls_cnt

uint8_t

I

User application implemented CCs count.

s2_unsolicited_cb

add_node_sec2_fn

I

Callback to report unsolicited joining device requested keys and/or status of Device Specific Key (DSK)

dtls_psk*

uint8_t[32]

I

DTLS pre-shared key in binary format, MUST be at least 16 bytes.

dtls_psk_len*

uint8_t

I

DTLS pre-shared key length (bytes). If length is zero, no DTLS will be used, i.e., communication will be unsecured.

Some of the values are familiar because they are also exposed on the Web interface and used by other Z/IP clients. The host_port is the port which Z-Ware is listening to. The unsolicited destination, discussed previously, is where the Z/IP client binds to a port and listens for unsolicited frames, such as lifeline reporting. Then, a define indicates whether to use IPv4 or IPv6. The address of the Z/IP Gateway is the address that Z-Ware should connect to. Then, there are several callback functions, as follows:

  • The notify function that notifies about network operations in progress.

  • The node callback that informs about changes to the nodes in the network.

  • The application TX which has information about the status of outgoing frame, so when transmitting a frame, the application can get information about whether the frame was delivered.

Some functions in bold relate to the portal mode, with Z-Ware typically running in the cloud and Z/IP Gateway connected through a TLS tunnel. These functions are mainly related to the portal connection and some SSL functions. A file descriptor contains information about the certificates required for the SSL link. When installing the Web portal, the installer prompts for this information and subsequently populates the structure. As a result, when using the portal, these are handled automatically.

Next, a pointer references a directory where Z-Ware saves information about the network. This structure is not quite up to date, as it is shown as a .dat, while the latest release has it changed to a json file that is human readable. Then, there is the device configuration which is covered later. Z-Ware has a device database, which can be used to manipulate how a device behaves in a Z-Wave network. It also has the option to disable command classes, which is typically not used. The next entries are not described.

Then, there is the S2 unsolicited callback, typically used Z-Wave networks with multiple controllers, where Z-Ware is the SIS. In this case, the other controllers in the network, the inclusion controllers, may include a node and hand over the security key exchange to Z-Ware. In this case, Z-Ware will relay some information about the key exchange to the application, which is promoted to the user to prompt input of required information.

Finally, some information is required when using Z-Ware running in local mode, as in the lab exercise, where it relies on the DTLS key.

Network Initialization, /zw_gw_set, REST API#

As outlined above, network initialization is a lot simpler when using the Z-Ware Web portal because a lot of this has been configured automatically by the Z-Ware daemon. When doing the same initialization on the Z-Ware Web portal, it looks a lot simpler, like this:

zip_gw_name={zip_gw_name}&unsol_rpt_port={unsol_rpt_port}&key={preshared_key}

Z-Ware is called, providing a gateway name, URL, or an IP address of Z/IP Gateway, and an unsolicited destination port and the DTLS key. All remaining entries are provided by the Z-Ware Web daemon.

Polling for Network Status on REST API, /zwnet_get_operation#

While the C-API has an option to configure required callbacks, the Web portal lacks this capability. To receive this information at runtime, the Z-Ware client needs to poll zwnet_get_operation at a relatively short interval. This function returns the same information as multiple C-API callback functions: the notify, node, application TX and S2 unsolicited callbacks. If taking a look at the top of the list, the options are the same as the callbacks. The Z-Ware Web daemon provides information about: adding nodes into the network, network operations, status of the nodes, and S2 bootstrapping. In other words, the Web daemon can obtain the same information as the Z-Ware C-API, but the client is required to poll Z-Ware because Z-Ware can't provide callbacks.

Values

Description

net_cmd_type

current network operation

| net_cmd_type/prev ||ZWNET_OP_XXX |# |Description || NONE |0 |No operation is executing || INITIALIZE |1 |Initialization operation || ADD_NODE |2 |Add node operation || RM_NODE |3 |Remove node operation || RP_NODE |4 |Replace failed node operation || RM_FAILED_ID |5 |Remove failed node operation || INITIATE |6 |Initiation operation by controller || UPDATE |7 |Update network topology from the SUC/SIS || RESET |8 |Restore to factory default setting || MIGRATE_SUC |9 |Create primary controller by a SUC || MIGRATE |10 |Migrate primary controller operation || LOAD_NW_INFO |12 |Load network information || NODE_UPDATE |13 |Update node info || SEND_NIF |14 |Send node information frame || ADD_NODE_ON_BEHALF |15 |Add node on-half operation (For /zwnet_add Web API only. Operation status will still be ZWNET_OP_ADD_NODE) || RP_NODE_ON_BEHALF |16 |Replace failed node on-half operation (For /zwnet_fail Web API only. Operation status will still be ZWNET_OP_RP_NODE) || FW_UPDT |19 |Firmware Update (For CMD_FIRMWARE_UPDATE_REQ_GET Web A only) || HEALTH_CHK |20 |Network health check || NODE_RESET |21 |Process node reset locally notification || FW_DOWNLD |22 |Firmware Backup (For CMD_FIRMWARE_BACKUP_REQ_GET Web API only) | |||net_cmd_status Progress status of the ongoing network operation. Some are generic while others are specific to the operation. | net_cmd_status ||# |Description || OP_STS_NONE |0 |No status to report || OP_STS_ERROR |-1 |Error || OP_STS_NO_NET |-4 |Network uninitialized || OP_STS_ABORTED |-5 | Network operation aborted || OP_ADD_NODE_xxx ||Add node status || PROTOCOL_DONE |1 |Protocol part of adding node done || GET_NODE_INFO |2 |Getting node detailed information || PROTOCOL_START |3 |Smart Start add node Z-Wave protocol started || SEC_REQ_KEY_READY |11 |(S2 inclusion only) Device requested key info is ready. Client may use /zwnet_add_s2_get_req_keys to query the requested key information. || SEC_DSK_READY |12 |(S2 inclusion only) Device DSK info is ready. Client may use /zwnet_add_s2_get_dsk to query for the DSK information. || ON_BEHALF_SEC_REQ_KEY_READY |21 |(S2 inclusion only) Device requested key info is ready. Client may use /zwnet_add_s2_get_req_keys to query the requested key information. || ON_BEHALF_SEC_DSK_READY |22 |(S2 inclusion only) Device DSK info is ready. Client may use /zwnet_add_s2_get_dsk to query for the DSK information. || OP_RM_NODE_XXX ||Remove node status || LEARN_READY |1 |Ready to remove a node || FOUND |2 |Found a node || REMOVING |3 |Removing the node || OP_RP_NODE_XXX || Replace node status || READY 1 Ready to replace a node || PROTOCOL_DONE |2 |Protocol part of replacing node done || SEC_INCD |3 |Adding node securely || GET_NODE_INFO |4 |Getting node detailed information || OP_INI_XXX ||Initiate status || STARTED |1 |Initiating started, ready to be added/removed to/from network || PROTOCOL_DONE |2 |Protocol part of initiating done || SEC_INCD |3 |Trying to be included securely || GET_NODE_INFO |4 |Getting node detailed information || SEC_OWN_DSK_READY |11 |(S2 inclusion only) Local node DSK info is ready. Client may use /zwnet_initiate_local_dsk_get to query for the DSK information. || OP_NU_XXX ||Network update status || TOPOLOGY |1 |Network topology update started || NEIGHBOR |2 |Node neighbor update started || GET_NODE_INFO |3 |Node information update started || OP_FW_XXX ||Firmware update status || UPLOAD_STARTED |1 |Uploading firmware to device started || UPLOADING |2 |Uploading firmware to device in progress with additional percentage info in op_total_nodes and op_cmplt_nodes || OP_FW_XXX ||Firmware download/backup status || DOWNLOAD_STARTED |1 |Download/backup firmware from device started || DOWNLOADING |2 |Download/backup firmware from device in progress with additional percentage info in op_total_nodes and op_cmplt_nodes || OP_HEALTH_CHK_XXX|| Network health check status || STARTED |1 |Network health check started || PROGRESS |2 |Network health check in progress || CMPLT |3 |Network health check completed

Helper functions, Gateway Discovery#

When initializing the network, the Z-Ware Web portal relies on helper functions. For example, the Z-Ware interface shows a list of Z/IP Gateways in the network. For this purpose, the Z-Ware C-API has the function zwnet_gw_discvr_start and an associated ..._stop function that start and stop gateway discovery. Z-Ware can perform Z/IP Gateway discovery using both mDNS and broadcast methodology based on the parameters given to the functions, and of course the Z-Ware Web interface has a similar function.

zwnet_gw_discvr_start

Callbacks, zwnet_notify_fn#

Returning to the topic of callback functions, now examine the notify callback in detail. This callback receives status of network operations in progress and provides status on what Z-Ware (and Z/IP Gateway) are doing. This callback indicates state information such as: none (no ongoing operations); whether nodes are added, or removed; and if network updates are ongoing, and so on. These callbacks provide confirmation when initiating network management operations.

XXX

Description

NONE

Idle.

INITIALIZE

Initializing local Z/IP controller.

ADD_NODE

Adding a node.

RM_NODE

Removing a node.

RP_NODE

Replacing a failed node.

RM_FAILED_ID

Removing a failed node.

INITIATE

Initiating in response to or anticipation of some operation.

UPDATE

Updating network topology from SUC/SIS.

RESET

Restoring ZIPGW attached controller to factory default settings.

NODE_UPDATE

Updating node information.

MIGRATE

Migrating primary controller.

SEND_NIF

Sending node information frame.

NW_CHANGED

Network change detection.

NODE_CACHE_UPT

Update node cached info. (Internally used; application won’t receive this notification.)

SAVE_NW

Save network and node information to persistent storage. (Internally used; application won’t receive this notification.)

SLP_NODE_POLL

Sleeping device node information polling. (Internally used; application won’t receive this notification.)

FW_UPDT

Firmware update.

HEALTH_CHK

Network health check.

NODE_RESET

Remove node which has been reset.

FW_DOWNLD

Firmware download

Callbacks, zwnet_node_fn#

Next, examine the node callback, which provides status for individual nodes. This callback indicates if node was added or removed from the network, and is typically encountered when a node becomes unreachable. For always-on nodes that typically have the status "alive," the status will change to "down" if for some reason transmissions fail. This allows the application to register that the node is currently unresponsive and there might be an error in the network. Similarly, for battery-powered sensor nodes, a "sleeping" status is typically returned. No notifications are provided when sleeping devices wake up, but the status will eventually change to "down" if a node does not wake up within the wake-up interval.

ZWNET_NODE_XXX

Description

ADDED

Node was added. Application can retrieve detailed node information now.

REMOVED

Node was removed.

UPDATED

Node was updated. Application can retrieve updated detailed node information now.

STATUS_ALIVE

Node status has changed to "alive"

STATUS_DOWN

Node status has changed to "down"

STATUS_SLEEP

Node status has changed to "sleeping"

RESET

Received notification from node that it has been reset

Callbacks, zwnet_appl_fn#

The appl or "application TX" callback provides status of frame transmissions. When transmitting to a Z-Wave device, information about the transmission status is provided, such as: if the transmission was successful, no ACK was received from the destination, or if the frame was queued for a sleeping destination.

Transmission status

ZWNET_TX_XXX

OK

Successful.

NO_ACK

Send frame timeout due to no ACK received.

SYS_ERR

System error, the program should exit or restart.

DEST_BUSY

Message has not timed out yet. The destination host may have a long response time (e.g., sleeping node).

NOROUTE

Frame failed to reach destination host.

Callbacks, s2_unsolicited_cb#

The S2 unsolicited callback is called when another controller has added a node to the network, indicating that the Z-Ware based controller, as the SIS in the network, needs to grant this device S2 keys. This callback allows a client application in an advanced configuration with multiple controllers, such as the Web Portal, to display a pop up prompting the user to input the device DSK and grant or deny the joining device its requested network keys.

Setting Unsolicited Destination#

The API for setting the unsolicited destination is essential for managing unsolicited traffic and lifeline reporting, such as notifications or sensor reports from nodes in the network. The API is called with the listening port, specified when initializing the network, and the IP address of the host running Z-Ware. Because Z/IP Gateway runs its own network stack, the unsolicited address is often confused with Z/IP Gateway's, which should remain separate. Effectively, the unsolicited addres is the IP address of Z-Ware, although the two hosts may be running on the same piece of physical hardware.

zwif_gw_unsolicit_set

Helper Functions Setting Unsolicited Destination#

Two helper functions are used to configure the unsolicited destination: local_address_get and zwnet_listen_port. The first, local_address_get, returns the IP address currently in use to communicate with Z/IP Gateway. If the host has multiple IP addresses on multiple interfaces, this API will return the address used to communicate with the Z/IP Gateway, which is by default the correct address. The second, zwnet_listen_port, returns the port specified when initializing Z-Ware.

zwnet_local_addr_get
zwnet_listen_port_get

Structure#

An application evaluate the Z-Wave network structure either network-first or endpoint first.

Beginning network-first, Z-Ware provides a handle to the network, and the application can use zwnet_get_node to get a handle to the first node in the network, which will typically, but not always, be the Z/IP Gateway itself. With the handle to the first node, the application can then iterate through the nodes in the network by using zwnode_get_next to get the next node, and so on.

From each node, the application can then use zwnode_get_ep, which will get the first endpoint on a node and then can then iterate through the remaining endpoints by using zwep_get_next.

Z-Ware Network EnumerationZ-Ware Network Enumeration
If, for example, the second node in the figure has multiple endpoints, the application can use zwnode_get_ep to get the first endpoint and then zwep_get_next to get the next endpoint, and so on.

zwep_if_get is an API to get the first handle to an interface or command class on the endpoint. To get the next interface, call zwif_get_next.

This allows the application to iterate through nodes and, for each node, through endpoints and, for each endpoint, through the command class. In this way, it can build a full Z-Wave network tree in application with all controllable resources. It works slightly differently on the Web interface using the lab interface on the Z-Ware Web portal but the Z-Ware C-API sample apps that work like this are covered in next session.

  • Network-first traversal

    • zwnet_get_node

    • zwnode_get_next

    • zwnode_get_ep

    • zwep_get_next

    • zwep_get_if

    • zwif_get_next

Alternatively, with a handle to an interface and a command class, the network can be traversed backwards. Starting with an interface, use zwif_get_ep to get the endpoint relating to the interface, then use zwep_get_node to get the node related to the interface, and zwnode_get_net, to get the net that is hosting the node that is hosting the endpoint, that is hosting the interface.

  • Endpoint-first traversal

    • zwif_get_ep

    • zwep_get_node

    • zwnode_get_net

Getting Information about Nodes (zwnoded_t) and Endpoints (zwepd_t)#

For each Z-Wave node in the network, a handle details the Z-Wave node and its capabilities. If watching the Zniffer while including nodes with Z-Ware, a lot of traffic follows the initial inclusion, as Z-Ware performs the mandatory interview on all supported command classes and populates a data structure containing this information about node. This interview is especially important for battery-powered nodes because Z-Ware will not be able to interview the device later when the device is sleeping. However, even for mains powered nodes, you must have the data cached.

Attribute

Type

I/O

Description

nodeid

uint8_t

O

Z-Wave Node ID

propty

uint8_t

O

Properties of the node (bit-mask): NODE_PROPTY_XXX.

Vid

uint16_t

O

Z-Wave Vendor ID.

type

uint16_t

O

Vendor type.

Pid

uint16_t

O

Z-Wave Product ID.

Net

zwnet_t *

O

Network handle.

dev_id

dev_id_t

O

Device ID.

proto_ver

uint16_t

O

Z-Wave Protocol Version.

app_ver

uint16_t

O

Application Version.

lib_type

uint8_t

O

Z-Wave Library Type.

category

uint8_t

O

Device category, DEV_XXX.

sensor

uint8_t

O

Flag to indicate whether the node is a sensor (FLIRS).

sleep_cap

uint8_t

O

Flag to indicate the node is capable to sleep (i.e. non-listening and support Wake up CC).

listen

uint8_t

O

Flag to indicate the node is always listening.

zwsw_ver_cnt

uint8_t

O

Number of valid Z-Wave software version types stored in the zwsw_ver array.

zwsw_ver

zwsw_ver_t[]

O

Z-Wave software version information.

s2_keys_valid

uint8_t

O

Flag to indicate whether s2_grnt_keys is valid.

s2_grnt_keys

uint8_t

O

Security 2 granted keys (bit-mask), see SEC_KEY_BITMSK_XXX

NOTE: This is valid only s2_keys_valid = 1.

s2_dsk

char[]

O

S2 DSK. If s2_dsk[0] == '\0', the DSK is unavailable for this node.

This is the data structure for a Z-Wave node. At the top is the Z-Wave node ID in the Z-Wave network. Then, there are some properties that are not described here. The vendor ID, vendor type, device ID this are collected from the device during inclusion. The same goes for the Z-Wave protocol version, the application version, and the library type. Flags specify whether the device is battery-powered and whether it is FLIRS or a sleeping device. Additional version numbers can be stored on the device. Flags indicate which, if any, S2 keys have been granted to the device. Finally, the DSK of the node is a unique identifier.

Similarly, each endpoint has a corresponding data structure, zwepd, which has information about: the specific and generic device classes; application level functionality of the endpoint; the endpoint number, and the node ID; and whether aggregated endpoints are present.

Attribute

Type

I/O

Description

generic

uint8_t

O

Z-Wave Generic device class

specific

uint8_t

O

Z-Wave Specific device class

epid

uint8_t

O

Endpoint number

nodeid

uint8_t

O

Node ID

aggr_ep_cnt

uint8_t

O

Total number of end point members that are represented by this aggregated end point. Zero means this endpoint is NOT an aggregated end point

aggr_members

uint8_t[126]

O

Buffer to store the end point members that are represented by this aggregated end point

net

zwnet_t *

O

Network handle

name

char [33]

O

User configured name string of the endpoint

loc

char [33]

O

User configured location string of the endpoint

zwplus_info

zwplus_info_t

O

Z-Wave Plus information CC

Registering Callbacks, Example for Basic, zwif_basic_rpt_set#

This section contains an example on registering a callback for the basic command class. The application calls zwif_basic_rpt_set and provides a pointer to the callback function. Returned data contains the data value and a timestamp that indicates when data was received, and a state number that is incremented whenever the cached value changes.

Attribute

Type

I/O

Description

ifd

zwifd_t *

I

Basic interface handle.

rpt_cb

zwrep_basic_fn

I

Report callback function.

return

int

O

ZW_ERR_NONE on success; else ZW_ERR_XXX.

Attribute

Type

I/O

Description

ifd

zwifd_t *

I

Interface handle.

val

zwbasic_t *

I

Basic data.

ts

time_t

I

Time stamp. If this is zero, the callback has no data and hence other parameter values should be ignored.

stat_num

uint16_t

I

State number that is incremented by one whenever a cache change is detected

Callback Limitations#

A callback is triggered only when incoming data is different from cached data. However, the timestamps on the cached data are always updated for each incoming frame.

Consider a sensor sending temperature readings at a fixed interval. When the temperature does not change between readings, a received report may not trigger a callback, but the timestamp is still updated for each incoming frame.

  • Z-Ware caches all incoming data

  • Callbacks trigger only when incoming data mismatches cached data

Polling, Example for Basic, C-api zwif_basic_get#

Z-Ware allows polling an interface, where the application actively requests data from an interface. Polling has two modes, “cache," and "live."

In the "cache" get mode, Z-ware provides a callback with the latest cached value and timestamp for the interface and nothing transmitted to the Z-Wave device.

For a “live” get mode, Z-Ware actively transmits a get command to the device and the device responds with a report. Note that the “live” get updates the timestamp of the cached value, but only provides a callback if the interface value was different from the value already cached.

As a result, in order to to get the absolute latest value from a device, first do an “active” get to update the cached value and then do a “cache” get to force a callback.

  • Values are returned through registered callback functions

  • Two modes:

    • ZWIF_GET_BMSK_CACHE

      • Does not actively get value from device

      • Always returns a callback

    • ZWIF_GET_BMSK_LIVE

      • Actively request the value from the end device

      • Only returns a callback if the value is different from the cached value

      • Time stamps are always updated

Device Database#

Z-Ware includes a device database. The device database allows modifications how a Z-Wave device appears to the application built on top of Z-Ware. For a legacy device, which only supports basic command class, you can remap basic command class into something more meaningful for the device. Basic command class could, for example, be mapped to the alarm interface. As a result, seen from the application level, the device would appear like a modern Z-Wave Plus a notification sensor, sending notification command class, while in fact being a very old device with very limited command class functionality. The catch is that the developer needs to know the devices that are intended to be included in the network in advance. This limits use and functionality of the database, which is generally most used by customers like service providers that have a large install base, and know which devices will be deployed in the network. It is prohibitive to populate the database for all Z-Wave devices that have been made historically. The release note for Z-Ware has a list of devices with which Z-Ware has been tested and known to work with. In general, the device database is populated with these devices to provide the best possible functionality. The programmers guide has a large example on the Z-Ware device database, showing how to remap command classes and manipulate Z-Wave devices in other ways to make it appear more uniform for the application.

  • Allows reconfiguring a device based on information that cannot be polled from the device itself.

  • Based on manufacturer and device ID

    • Existing command classes can be:

      • Modified

      • Remapped to other command classes