Network Core Troubleshooting#

• Network Interface

• Socket Operations

• Performance Issues

  • Number of RX & TX Buffers to Configure

  • Number of DMA Descriptors to Configure

  • Configuring Window Sizes

  • Reducing the Number of Transitory Errors

• Wi-Fi Interface

  • An RTOS_ERR_IO error is returned when I try to join a Wi-Fi network

  • I am getting an RTOS_ERR_INVALID_HANDLE in SPI_SlaveOpen() when adding a Wi-Fi interface

  • I am getting an RTOS_ERR_NOT_FOUND when adding a Wi-Fi interface

Network Interface#

When adding a new network interface with NetIF_xx_Add(), I never return from the function.

• Verify that your controller Base Address is correct (see section).

• Could also be a problem with your BSP Clock or GPIO configuration (see section).

When adding a new network interface with NetIF_xx_Add(), the target crashes before returning.

• Verify that your controller Base Address is correct (the target should crash in the driver initialization function NetDev_Init() if the Base Address is incorrect).

• Could be a problem in your BSP (the target should crash in NetDev_Init() after a BSP call).

• Your task stack size could be insufficient.

When adding a new network interface with NetIF_xx_Add(), I get an error.

These are the more common errors:

• RTOS_ERR_NOT_FOUND : Before creating a new network interface, you need to have previously registered the network controller associated with this network interface inside the BSP_OS_Init() function.

• RTOS_ERR_NO_MORE_RSRC : Be sure to have set the maximum network interface (NET_IF_CFG_MAX_NBR_IF inside file net_cfg.h) according to your application needs.

• RTOS_ERR_SEG_OVF : The memory space required for the new network interface is insufficient. You can increase your general HEAP memory size or the memory segment used for the network module. You can also try to reduce the number of buffers configured for this network interface.

When starting a network interface with NetIF_xx_Start(), I end up in a debug assert with the error code RTOS_ERR_NOT_AVAIL.

You are trying to setup the network interface with a module that is not enabled in your configuration; e.g., your NET_IF_xx_CFG argument enables an IPv6 address configuration process but the NET_IPv6_CFG_EN inside your net_cfg.h file is set to DEF_DISABLED.

When starting a network interface with NetIF_xx_Start(), I set the NET_IF_xx_CFG argument to enable DHCP client, but then I just exit NetIF_xx_Start() without knowing if the DHCP process was successful. (This can also apply to the IPv6 setup)

The NetIF_xx_Start() function is non-blocking. If you want to be notified when the DHCP client process is done, you need to define a callback function and refer to it in the DHCPc_ON_COMPLETE_HOOK field of the NET_IF_xx_CFG argument. The program call the hook function when the DHCP process has completed. You can also use the API function NetIF_WaitSetupReady() that allows waiting for all the setup of the network interface to be done after the Start function. When using this API the different callback functions inside the NET_IF_xx_CFG argument are not mandatory.

When using the function NetIF_WaitSetupReady () to wait for the setup completion after the network interface start, the function returns with the error code RTOS_ERR_TIMEOUT.

By default, the function NetIF_WaitSetupReady() will time out after a delay of 30 seconds even if not all the setup steps have been completed. This delay should be enough on most networks. You can pass a bigger timeout value if you feel that your network needs more time to complete all the address setups. You can also use the argument of type NET_IF_APP_INFO to get more information on the setup progress. By passing a pointer to a NET_IF_APP_INFO variable, the function will populate the structure with the information of the different address setups gradually. You can, therefore, see which ones was completed and the ones still in progress. If no address setups seem to have happened during the waiting process, you should validate that your network interface is well configured because it could mean that the interface is never getting is link state to UP.

Socket Operations#

When binding a socket to an IP address using the NetSock_Bind() function, I get an error.

These are the more common errors:

• RTOS_ERR_INVALID_HANDLE : The socket ID passed to the function is invalid. Check if you did not previously close the socket.

• RTOS_ERR_NOT_FOUND : The IP address must have previously been configured on the network interface.You can validate if an IP address is configured with the function NetIPv4_IsAddrHostCfgd() or NetIPv6_IsAddrHostCfgd().

• RTOS_ERR_ALREADY_EXISTS : A socket binding with the same IP local/remote addresses and local/remote ports already exist. The network stack doesn't support to have multiple sockets with the exact same binding of local/remote addresses/ports.

When calling NetSock_Conn() to open a TCP connection to a remote host, I get an error.

These are the more common errors:

• RTOS_ERR_INVALID_HANDLE : The socket ID passed to the function is invalid. Check if you did not previously close the socket.

• RTOS_ERR_ALREADY_EXISTS : A socket binding with the same IP local/remote addresses and local/remote ports already exist. The network stack doesn't support to have multiple sockets with the exact same binding of local/remote addresses/ports.

• RTOS_ERR_NET_INVALID_ADDR_SRC : No valid configured IP address was found to send the connection request. You should check that you have a valid IP address configured on each of your network interfaces.

Transitory errors are also possible. Most of the time, in those cases, you can retry the operation. It is recommended to add a short delay before the retry especially if your socket is in a non-blocking mode:

• RTOS_ERR_POOL_EMPTY : You are referring to a memory pool that doesn't have any element left inside. In this context, it will refer most of the time to the network buffer pool, therefore no more buffer is available to send the TCP SYN message.

When my TCP socket is in listening mode and I call NetSock_Accept() to accept new incoming connections, I get an error.

These are the more common errors:

• RTOS_ERR_INVALID_HANDLE : The socket ID passed to the function is invalid. Check if you did not previously close the socket.

Transitory errors are also possible. Most of the time, in those cases, you can retry the operation. It is recommended to add a short delay before the retry especially if your socket is in a non-blocking mode:

1. RTOS_ERR_POOL_EMPTY : You are referring to a memory pool that doesn't have any element left inside. In this context, it will refer most of the time to the socket pool since a child socket will be created for the new connection. Therefore be sure that your socket configuration includes enough TCP socket type for your needs.

2. RTOS_ERR_TIMEOUT : When the socket is in blocking mode, this means that no incoming connections requests were received even after the waiting inside the NetSock_Accept() function. The timeout value can be set by the macro NET_SOCK_DFLT_TIMEOUT_CONN_ACCEPT_MS in the net_cfg.h file or can be set for via API functions (see NetSock_CfgTimeoutConnAcceptDflt() and NetSock_CfgTimeoutConnAcceptSet()).

3. RTOS_ERR_WOULD_BLOCK : When the socket is in non-blocking mode, this means that no incoming connection requests were received at the time NetSock_Accept() was called.

When my TCP socket is in listening mode and multiple new incoming connection requests are sent to it, not all connections seem to be received by the socket.

Different reasons can explain why some TCP connection requests are dropped by the stack. The two main reasons are:

• Your socket accept queue is already full when a new incoming connection request is received. You can increase the size of your socket accept queue with the configuration macro NET_SOCK_CFG_CONN_ACCEPT_Q_SIZE_MAX in net_cfg.h file.

• You don't have any TCP connection objects left when a new incoming connection request is received. The number of TCP connection objects you have comes from the configuration macro NET_SOCK_CFG_SOCK_NBR_TCP, but you can also have additional TCP connection objects from the configuration macro NET_TCP_CFG_NBR_CONN.

To validate if either of those cases are happening, you can check the error counters Net_ErrCtrs.TCP.RxListenQ_FullCtr and Net_ErrCtrs.TCP.NoneAvailCtr.

When calling NetSock_RxDataFrom() / NetSock_RxData() to receive data, I get an error.

These are the more common errors:

• RTOS_ERR_INVALID_HANDLE : The socket ID passed to the function is invalid. Check if you did not previously close the socket.

• RTOS_ERR_NOT_FOUND : The destination address passed as argument when using API NetSock_RxDataFrom() is not associated with any known connections.

• RTOS_ERR_NET_CONN_CLOSE_RX : In case of a TCP socket, this means a FIN was received from the other side of the connection. Therefore, when ready, you should close your socket.

• RTOS_ERR_NET_CONN_CLOSED_FAULT : In case of a TCP socket, this means that a RESET was received. Therefore you should close your socket as soon as possible.

• RTOS_ERR_WOULD_OVF : In case of a UDP socket, the size of the reception buffer is not enough for the size of the datagram received.

Transitory errors are also possible. Most of the time, in those cases, you can retry the operation. It is recommended to add a short delay before the retry especially if your socket is in a non-blocking mode:

• RTOS_ERR_TIMEOUT : When the socket is in blocking mode, this means that no data was received even after the waiting inside the receive function. The timeout value can be set by the macro NET_SOCK_DFLT_TIMEOUT_RX_Q_MS in the net_cfg.h file or can be set for via API functions (see NetSock_CfgTimeoutRxQ_Dflt() and NetSock_CfgTimeoutRxQ_Set()).

• RTOS_ERR_WOULD_BLOCK : When the socket is in non-blocking mode, this means that no data was received at the time the receive function was called.

When calling NetSock_TxDataTo() / NetSock_TxData() to transmit data, I get an error.

These are the more common errors:

• RTOS_ERR_INVALID_HANDLE : The socket ID passed to the function is invalid. Check if you did not previously close the socket.

• RTOS_ERR_NOT_FOUND : The destination address passed as argument when using API NetSock_TxDataTo() is not associated with any known connections.

• RTOS_ERR_NET_CONN_CLOSE_RX : In case of a TCP socket, this means a FIN was received from the other side of the connection. Therefore, when ready, you should close your socket.

• RTOS_ERR_NET_CONN_CLOSED_FAULT : In case of a TCP socket, this means that a RESET was received. Therefore you should close your socket as soon as possible.

• RTOS_ERR_NET_INVALID_ADDR_SRC : In case of a UDP socket, no valid configured IP address was found to sent the data. You should check that you have a valid IP address configured on each of your network interfaces.

• RTOS_ERR_NET_NEXT_HOP : No valid next hop was found to sent the packet to. You should check that a valid gateway was configured if you wish to send data outside the local network.

• RTOS_ERR_WOULD_OVF : In case of a UDP socket, this means that the length of the data to send is too big for the size of the transmit buffers.

Transitory errors are also possible. Most of the time, in those cases, you can retry the operation. It is recommended to add a short delay before the retry especially if your socket is in a non-blocking mode:

• RTOS_ERR_TIMEOUT : When the socket is in blocking mode and for TCP socket, this means that the transmit window is full and no data can be queued for transmission. The timeout value can be set by the macro NET_TCP_DFLT_TIMEOUT_CONN_TX_Q_MS in the net_cfg.h file or can be set for via API functions (see NetSock_CfgTimeoutTxQ_Dflt() and NetSock_CfgTimeoutTxQ_Set()).

• RTOS_ERR_WOULD_BLOCK : When the socket is in non-blocking mode, this means that the transmit window is full at the time the transmit function was called.

• RTOS_ERR_POOL_EMPTY : Refer to a memory pool who doesn't have any element left inside. In this context, it will refer most of the time to the network buffer pool, therefore no more buffer is available to send the data.

• RTOS_ERR_NET_ADDR_UNRESOLVED : The destination MAC address of the next hop is not resolved. It could be that the ARP or NDP process was just not completed yet.

Performance Issues#

Number of RX & TX Buffers to Configure#

The number of large receive, small transmit and large transmit buffers you will configure for a specific interface depends on several factors:

1. Desired level of performance.

2. Amount of data to be either transmitted or received.

3. Ability of the target application to either produce or consume transmitted or received data.

4. Average CPU utilization.

5. Average network utilization.

6. Type of connection (UDP or TCP)

7. Number of simultaneous connection.

8. Application/connection priorities

The discussion on the bandwidth-delay product is always valid. In general, the more buffers the better. However, the number of buffers can be tailored based on the application. For example, if an application receives a lot of data but transmits very little, then it may be sufficient to define a number of small transmit buffers for operations such as TCP acknowledgments and allocate the remaining memory to large receive buffers. Similarly, if an application transmits and receives little, then the buffer allocation emphasis should be on defining more transmit buffers. However, there is a caveat:

If the application is written such that the task that consumes receive data runs infrequently or the CPU utilization is high and the receiving application task(s) becomes starved for CPU time, then more receive buffers will be required.

To ensure the highest level of performance possible, it makes sense to define as many buffers as possible and use the interface and pool statistics data in order to refine the number after having run the application for a while. A busy network will require more receive buffers in order to handle the additional broadcast messages that will be received.

In general, at least two large and two small transmit buffers should be configured. This assumes that neither the network or CPU are very busy.

Many applications will receive properly with four or more large receive buffers. However, for TCP applications that move a lot of data between the target and the peer, this number may need to be higher.

Specifying too few transmit or receive buffers may lead to stalls in communication and possibly even dead-lock. Care should be taken when configuring the number of buffers. The Network module is often tested with configurations of 10 or more small transmit, large transmit, and large receive buffers.

Number of DMA Descriptors to Configure#

If the hardware device is an Ethernet MAC that supports DMA, then the number of configured receive descriptors will play an important role in determining overall performance for the configured interface.

For applications with 10 or less large receive buffers, it is desirable to configure the number of receive descriptors to that of 60% to 70% of the number of configured receive buffers.

In this example, 60% of 10 receive buffers allows for four receive buffers to be available to the stack waiting to be processed by application tasks. While the application is processing data, the hardware may continue to receive additional frames up to the number of configured receive descriptors.

There is, however, a point at which configuring additional receive descriptors no longer greatly impacts performance. For applications with 20 or more buffers, the number of descriptors can be configured to 50% of the number of configured receive buffers. After this point, only the number of buffers remains a significant factor; especially for slower or busy CPUs and networks with higher utilization.

In general, if the CPU is not busy and the Network Receive task has the opportunity to run often, the ratio of receive descriptors to receive buffers may be reduced further for very high numbers of available receive buffers (e.g., 50 or more).

The number of transmit descriptors should be configured such that it is equal to the number of small plus the number of large transmit buffers.

These numbers only serve as a starting point. The application and the environment that the device will be attached to will ultimately dictate the number of required transmit and receive descriptors necessary for achieving maximum performance.

Specifying too few descriptors can cause communication delays. See section Network Interface Controller Configuration for descriptors configuration.

Configuring Window Sizes#

Receive and transmit queue size must be properly configured to optimize performance. It represents the number of bytes that can be queued by one socket. It's important that all socket are not able to queue more data than what the device can hold in its buffers. The size should be also a multiple of the maximum segment size (MSS) to optimize performance. UDP MSS is 1470 and TCP MSS is 1460.

RX and TX maximum queue size are configured using #defines in net_cfg.h , see Network Core Compile-Time Configurations .

RX and TX queue size can be reduced at runtime using socket option API (NetTCP_ConnCfgRxWinSize() and NetTCP_ConnCfgTxWinSize()).

The following listing shows a calculation example:

Number of TCP connection  : 2
    Number of UDP connection  : 0
    Number of RX large buffer : 10
    Number of TX Large buffer : 6
    Number of TX small buffer : 2
    Size of RX large buffer   : 1518
    Size of TX large buffer   : 1518
    Size of TX small buffer   : 60

    TCP MSS RX                = 1460
    TCP MSS TX large buffer   = 1460
    TCP MSS TX small buffer   = 0

    Maximum receive  window   = (10 * 1460)           = 14600 bytes
    Maximum transmit window   = (6  * 1460) + (2 * 0) = 8760  bytes

    RX window size per socket = (14600 / 2) =  7300 bytes
    TX window size per socket = (8760  / 2) =  4380 bytes

Reducing the Number of Transitory Errors#

The number of transmit buffer should be increased. Additionally, it may be helpful to add a short delay between successive calls to socket transmit functions.

Wi-Fi Interface#

An RTOS_ERR_IO error is returned when I try to join a WiFi network#

Make sure the SSID and Password sent to NetIF_WiFi_Join() are correct. Also note that some WiFi adapters require you to perform a scan before attempting to join an access point.

I am getting an RTOS_ERR_INVALID_HANDLE in SPI_SlaveOpen() when adding a WiFi interface#

Make sure you are specifying the correct SPI bus handle when registering the WiFi controller in the BSP. Also make sure that it corresponds to the handle of the serial controller that was registered beforehand in the BSP. If you are using the provided examples, don't forget to update the value of EX_SPI_CTRLR_NAME in the SPI initialization example file.

I am getting an RTOS_ERR_NOT_FOUND when adding a WiFi interface#

Make sure that the WiFi controller is properly registered at the BSP level (bsp_os.c). If multiple WiFi controllers are supported on a development board you are using, you may have to select it in bsp_net.h.