File System

Most connected applications require the ability to store configuration information or cache local data. Gecko OS provides a reliable read/write filesystem and access to extended and bulk flash storage to satisfy application storage requirements.

Silicon Labs IoT modules contain a microprocessor with internal flash and an extended (on module) serial flash. An additional (optional) bulk serial flash is also supported. See Flash Storage. For third-party devices running Gecko OS, an extended serial flash must be provided to store temporary files and to enable wireless OTA updates.

Commands are available to create, delete and open files, and file contents can be easily accessed using one or more stream commands.

Here's a quick example showing how to create, manipulate then delete a file:

Gecko OS Commands Description

> file_create     hello.txt 12
Hello World!

Success
> file_open       hello.txt
0
> stream_read     0 3
Hel
> stream_read     0 50
lo World!
> file_delete     hello.txt
Success

<- Create file, length 12 bytes
<- Enter file contents
  (echoed if system.cmd.echo = 1)

<- Open file for reading
(Gecko OS returns file stream handle = 0)
<- Read no more than 3 bytes

<- Read no more than 50 bytes
  (close file if EOF reached)
<- Delete file

When a file is open, Gecko OS creates a file stream and returns a file stream handle to be used as an argument of the stream_read command. See Network Connections and Streams. The stream type is FILE.

File System Limits

Prohibited Files and Directories

Files cannot be created under the sys virtual directory. That is, a file cannot be created where the file name begins with sys/. This prevents deleting and replacing 'system' files.

Minimum File Size

Files are created 4KByte aligned; that is, no file can occupy less flash space than 4K.

Maximum File Size

There is no limit to the size of an individual file other than the size of the extended or bulk serial flash connected to the Gecko OS hardware. However the method of creating the file may impose a limit.

When creating files via the file_create command without the -o option, or via the Web App, which uses file_create in the background, file size is limited by available RAM at file creation.

Maximum File Count

The minimum flash space occupied by files is <number of files> * 4K. For example, in 20K of flash you can create no more than 5 files .

Listing Files

To list files, use the ls command.

To obtain a verbose listing of all files on the MCU-internal flash, bulk and extended serial flash, use the ls command with the -v (verbose) or -l (long) option. The -v and -l options result in the same listing. Users are permitted to write files only to serial flash.

ls -l
!   # L Hnd Type Flag O Perm    Size    Created   Version Filename
#  0 e  61   45 0401 d FFFF     743 1528250046     1.0.0 favicon.ico.gz
#  1 e  44   41 0401 d FFFF    1236 1528250046     1.0.0 geotrust_ca.pem
#  2 i   0    1 0607 d 0000  327424 1528250046     0.0.0 sys/kernel.bin
#  3 i  50    7 0607 d 0000  254768 1528250046     1.0.0 sys/plugins.bin
#  4 e   0    2 0605 d 0000  276552 1528250046     1.2.6 sys/wfm_wf200.sec
#  5 e  C4    4 0201 u FFFF    2048         28     1.0.0 sys/wl_table.bin
#  6 e  45   45 0401 d FFFF    2357 1528250046     1.0.0 webapp/index.html
#  7 e  5E   45 0401 d FFFF    9745 1528250046     1.0.0 webapp/unauthorized.html
#  8 e  46   45 0401 d FFFF   22720 1528250046     1.0.0 webapp/gecko_os.css.gz
#  9 e  4C   45 0401 d FFFF   72947 1528250046     1.0.0 webapp/gecko_os.js.gz

The meaning of the columns is summarised here and covered in detail in the sections below:

File Location

The ls -l file listing L column indicates the location of the file in internal, bulk or extended flash:

See Flash Storage below.

File Handle

The ls -l file listing Hnd column indicates the file handle. This is a file identifier that corresponds to the file's base sector in flash. The handle can be used with the ls command -b option for paging the file listing, and with the file_open, file_delete and file_stat <memory>-<handle> parameters as a file identifier to distinguish files with the same file name.

File Types

The ls file listing Type column indicates the file type in hex. Gecko OS uses file type internally for file management.

File types are enumerated in the following table.

File typeID (hex)ID (dec)Access
UNKNOWN0x000Kernel
KERNEL0x011Kernel
WIFI_FW0x022Kernel
NVM_DEFAULTS0x033Kernel
WEAR_LEVELING_TABLE0x044Kernel
STARTUP_HANDLER0x055Kernel
SAFEMODE_APP0x066Kernel
PLUGINS_APP0x077Kernel
USER_APP0x088Kernel
TEMPORARY0x099Kernel
NVM_USER_DEFAULTS0x0A10Kernel
... Reserved ...0x0B-03F11 - 63Kernel
USER_TYPES_START0x4064User
LOG0x4064User
TLS_CERTIFICATE0x4165User
GPIO_CONFIG0x4266User
SETTINGS_CSV0x4367User
GENERAL0x4468User
WEB_APP0x4569User
... Reserved ...0x46-0x7F70-127Kernel
CUSTOM_START0x80128User
... Custom ...0x80-0xFE128-254User
CUSTOM_END0xFE254User
INVALID0xFF255

Custom User Types

The type range from 0x80 (decimal 128) to 0xFE (decimal 254) is set aside for custom types. Use types in this range for file management specific to your application.

File Flags

The ls file listing Flag column indicates the file flags, in hex format, without a 0x prefix.

Flag bits 0 - 7 are for Kernel use only and cannot be set by the application.

Flags correspond to bits as follows:

FlagBitNotesAccess
Valid0File is valid on the File System. This flag can be set or cleared only by the kernel.Kernel
Executable1File is an executable (i.e. is an application).Kernel
Kernel-only2File is accessible only by the kernel. It is not readable by a customer application.Kernel
Reserved3 - 7Kernel
Encrypted80: Not encrypted; 1: Encrypted
See The Encrypted Flag.
User
Essential9This flag can be set explicitly with the -e flag with the file_create and http_download commands. See The Essential Flag.User
Checksum Valid10The file's checksum (CRC) field is valid. If this flag is set when creating a file, the checksum is validated against the supplied checksum before setting the file Valid flag.User
Pre-encrypted11The file's data is pre-encrypted. Gecko OS is not to encrypt the file as it is written.User

Examples:

Flags columnFlags Set
0101Valid Encrypted
0201Valid Essential
0301Valid Encrypted Essential

The Encrypted Flag

A file marked with the Encrypted flag is encrypted, either with the system.security_key or with the internal device key (AES key).

A file is encrypted on creation with the file_create and http_download commands, when you use the -u, -d or -p options.

See also Encrypting Gecko OS Files.

The Essential Flag

The Essential flag indicates that a file must not be deleted during an OTA. An OTA may delete files not marked Essential in order to make space for new or changed files.

You set the Essential flag on creating a file, using the -e option with the file_create and http_download commands.

File Owner

The ls file listing O column indicates the file owner. In the Gecko OS context, the file owner is the product, device, or user. The owner determines the encryption key to be used if file encryption is required.

ls codeOwnerEncryption Key
pProductDMS product's encryption key
dDeviceDevice's internal encryption key
uUsersystem.security_key

File Permissions

The ls file listing Perm column indicates the file permissions. File permissions are displayed as a bit mask, specified as a hex string. The permissions attribute indicates which interfaces may read or delete the file. Note that if the 'Kernel-Only' file flag is set then this attribute is ignored.

The default permissions bitmask is 0xFFFF: ALL

InterfaceBitNotes
Serial Bus0Any UART, SPI, USB that has a command parser enabled
Remote Terminal1Remote telnet terminal
HTTP API2HTTP Server REST API or websocket
Bluetooth3Bluetooth interface
Zigbee4Zigbee interface

File Version

For files that form part of the Gecko OS system, such as services.bin and kernel.bin, the file version follows the scheme:

<major>.<minor>.<patch>

When creating a user file, the user can choose the versioning scheme. The file_create and http_download commands provide a version option to set the version.

File Checksum

The checksum or CRC used on individual files is calculated using a CRC32 algorithm.

The file_create and http_download commands provide an option to set the CRC.

You can view a file CRC with the file_stat command. The CRC is the second item in the comma-separated list.

The following python fragment shows how the binascii.crc32 function can be used to calculate the CRC.

import binascii

def crc32(data):
    return binascii.crc32(data) & 0xffffffff

See also the File System application note.

File Names

File names are case sensitive, and can contain a maximum of 95 characters from the set:

A-Za-z0-9~!@#$%^&*()_+#.{}[]`=:;|\/'-.

That is, a file name can contain any character except for the double quote character. A double quote results in a parse error when creating a file with the file_create command.

It may be possible to create a file with a double quote character using other means, then download the file to the device flash.

The Gecko OS file system does not treat the . character or the / character differently to any other. These can be used to create pseudo extensions and pseudo folders. The web app recognizes the / character for folder emulation, which makes some file manipulations more convenient. See Gecko OS Web App, Folder Emulation.

Web app folder emulation creates some restrictions on where you can place a / character in a file name. These apply only to files manipulated in the Web App Files tab.

Special Files

Some files with special functions are listed in the following table:

File NameDescriptionNotes
default_config.csvVariable configuration, loaded after a successful OTASee Configuration and Setup, Variable Configuration Script.
factory_config.csvFactory variable configuration, loaded if found after a hard or soft factory reset.See Configuration and Setup, Variable Configuration Script.

Writing Files

There are two Gecko OS commands that can write to the file system:

There are several ways to invoke these commands:

Writing with the Web App File Browser

When you open the Gecko OS Web App in a web browser and select the Files tab, you can load files to the device file system using click and browse, or drag and drop.

In the background, the Gecko OS Web App uses the file_create command to write files to the device file system, using the HTTP Server RESTful API.

There are several ways to activate the Gecko OS Web App. See Gecko OS Web App.

Writing with the Terminal

In the Gecko OS terminal or remote terminal, use the file_create command to create the file. Immediately after issuing the command, type the file contents, or alternatively copy and paste the file contents into the terminal after issuing the fcr command. The length of the file must be supplied. The character count includes any line termination characters.

In the example below we create a script that can be run using the command setup cmd -v test.script. In this case the character count is 35.

> fcr test.script 35
help,setup,\r\n#This is a comment

File created
Success

Writing File Contents in Chunks

You can write the file in chunks by leaving the file open after issuing the file_create with the -o option. Write the file chunks to the stream returned by the file_create command. You must know the total size of the file to be created in advance.

When you open the file, Gecko OS returns a stream handle, to be used as an argument of the stream_write command. See Network Connections and Streams.

In the example below a small file is created in two chunks:

Commands and ResponsesDescription

> file_create hello.txt 11 -o
[Opened: 0]
0
> stream_write 0 6
Hello
Success
> stream_write 0 5
World
File created
[Closed: 0]
Success
> file_open hello.txt
[Opened: 0]
0
> stream_read 0 1000
Hello World
[Closed: 0]

Create file; leave stream open; specify total length

Response: open stream and return stream handle
Write first chunk; specify chunk length
Send chunk content (includes trailing space)

Write second chunk; specify chunk length
Send chunk content

Stream closed

Open new file

Response: open stream and return stream handle
Read contents

Stream closed

HTTP Download

You can also provide the file for download from a web server accessible to the device, and use the http_download command. For example:

> http_download http://www.google.com.au/images/srpr/logo11w.png test1.png
Downloading: test1.png to flash file system
Request GET /images/srpr/logo11w.png
Connecting (http): www.google.com.au:80
HTTP response: 200
Success
> ls
!  #   Size   Version  Filename
…
#  4  14022     1.0.0  test1.png
…

OTA

OTA wireless updates allow you to update the Gecko OS system files automatically using the Zentri Device Management Service. Please contact Silicon Labs if you need custom hosting services for your application firmware or files.

Reading Files

The following Gecko OS commands can read from the file system:

There are several ways to invoke these commands:

Reading with the Web App File Browser

When you open the Gecko OS Web App in a web browser and select the Files tab, you can download a file from the device file system by clicking the file name.

In the background, the Gecko OS Web App uses the file_open command to read the files from the device file system, using the HTTP Server RESTful API.

There are several ways to activate the Gecko OS Web App. See Gecko OS Web App.

Reading from a Terminal

You can read text files directly from the Gecko OS Terminal or remote terminal. Read the file, using a file_open followed by a stream_read command, specifying the stream index returned from the file open, e.g.:

> fop default_setup.script
[Opened: 0]
0
> read 0 1000
network_up,-s    ,Configuration network credentials
set wlan.auto_join.enabled,true,Enable network auto-join
save,-,Saving settings

[Closed: 0]

HTTP Upload

You can upload the file to an available web server with file upload capability with the http_upload command.

Flash Storage

A Gecko OS device contains internal flash inside the microprocessor and an extended flash chip for user storage.

Extended flash is serial flash. Internal flash cannot be manipulated by Gecko OS commands.

Flash Wear Leveling

The flash wear leveling feature ensures that all sectors on the extended/bulk flash are evenly used. This allows for maximum file writes to the extended/bulk flash. This system works seamlessly in the background, and requires no user activity.

The file sys/wl_table.bin is used by the flash wear leveling feature.

Bulk Flash

In addition, an external serial flash can be connected to the device using the existing sflash GPIOs. This is referred to as bulk flash. To enable access to bulk flash, configure the chip select GPIO with the system.bflash.cs_gpio variable.

By default Gecko OS can support up to 128MB bulk flash.

All OTA files and config files (created by the save command) are always stored on extended flash regardless of whether the bulk flash is enabled.

The following commands operate on bulk flash if available, otherwise on extended flash:

Extended and bulk flash can be formatted with the format_flash command.

Supported Bulk Flash Chips

Serial flash chips currently supported by Gecko OS are as follows:

Model IDStorage Capacity
(Megabytes)
SFLASH_ID_EN25Q80B1
SFLASH_ID_EN25QH162
SFLASH_ID_MX25L1606E2
SFLASH_ID_MX25L6406E8
SFLASH_ID_MX25L8035E1
SFLASH_ID_SST25VF080B1
SFLASH_ID_SST26VF016B2

If you require support for a serial flash chip not listed in the table above, please contact Silicon Labs.

Connecting Bulk Serial Flash to the Device

The Gecko OS device is the SPI master. The bulk sflash is the SPI slave.

Connecting bulk sflash to the device requires 4 signals. The GPIO numbers vary depending on the device.

Use the format_flash bulk command to erase the contents of the sflash.

The bulk sflash uses the same SPI bus as the extended serial flash on-board the module. Thus the SPI parameters (clock speed, polarity, etc) are not configurable.

Encrypting Gecko OS Files

Gecko OS maintains three different AES encryption keys. The keys are as follows:

NameProgrammed/UpdatedLocationPersistenceUniquenessDescription
ProductManufacturing or DFUKernel binaryLifetime of product on DMSCommon for all devices of a given product typeThe 'product' encryption key is used to encrypt DFU images. The Kernel uses this key to decrypt DFU image data as it's read. The 'product' encryption key resides in the Kernel binary.
When an encrypted file 'owned' by the product is written to extended/bulk flash it is encrypted with this key.
DeviceFirst time its neededNVMLifetime of internal flashUnique per deviceThe device internally generates this. It is never externally accessible.
When an encrypted file 'owned' by the device is written to extended/bulk flash it is encrypted with this key.
UserManufacturing, DFU, 'set' commandNVMUser specifiedUser specifiedThe user generates this. It is set during DFU or the command 'set system.user_key'.
When an encrypted file 'owned' by the user is written to extended/bulk flash it is encrypted with this key.

DFU File Encryption Sequence

The DFU (Device Firmware Update) process uses the product and device keys.

When performing a DFU, the DFU images are encrypted by the DMS with the device's product key. As the device reads the DFU image data, it decrypts the data using the product key embedded in the Kernel and writes the bundle files to their corresponding memory location.

IF a bundle file has its Encrypted flag set AND the file's 'owner' field is set to 'device', THEN the file is encrypted with the device's AES as it is written to device memory.

IF a bundle file has its Encrypted flag set AND the file's 'owner' field is set to 'product' or 'user', THEN the file is written AS-IS to device memory (the file is assumed to be encrypted with the corresponding key before sending to the device).

Encrypted File Format

A Gecko OS encrypted file has the following format:

So an encrypted file is always 32 bytes longer than a file with same contents unencrypted.

Encryption Sequence

The sequence for encrypting a file is as follows:

  1. Generate Initialization Vector (IV)
  2. Initialize AES-128 context with the user AES key
  3. Calculate the Cipher Block Chaining (CBC) of the entire file, IV=0, and pad with 0s to 16 bytes if necessary, the last 16 bytes of the encrypted output is used as the HMAC
  4. Write the initial IV (the IV from step 1) to the output file/buffer
  5. Using the IV from step 1 and AES-128 context from step 2, using CTR mode encrypt HMAC and write to output file/buffer
  6. Using IV from step 5 and AES-128 context from step 2, using CTR mode encrypt file contents and write to output file/buffer

Decryption Sequence

The sequence for decrypting a file is as follows:

  1. Initialize AES-128 context with the user AES key
  2. Read the 16 byte IV from the beginning of the encrypted file
  3. Read the 16 bytes encrypted HMAC from the file
  4. Using the AES context from Step 1 and the IV from Step 2, decrypt the HMAC using AES CTR mode
  5. Using the AES context and IV from step 4, decrypt the rest of the file using AES CTR mode
  6. Calculate the Cipher Block Chaining (CBC) of the decrypted file data from step 5, IV=0, and pad with 0s to 16 bytes if necessary
  7. The final block from step 6 should match the HMAC from step 4. If the two 16 byte values don't match then either the given user key is invalid or the encrypted file is corrupted.

Preventing File Corruption

The Gecko OS file system is designed to be resistant to file corruption. A file does not appear in the file system until it is marked as valid. See File Flags. A file is marked as valid only after all file data is written and validated. When a file is not valid, it cannot be listed with the ls command or opened with the file_open command. A valid file may not be modified. That is, file size and contents cannot be modified after the file is created.

This ensures the file system does not become populated with corrupt files due to sudden power loss, reset, or interruption of the file stream during writing.

Although normal files cannot be modified, you can add contents to a log file after it is created.

The steps in file creation are as follows:

If the module is reset or power cycled, or the file stream is manually closed before the valid flag is set, the created file and any written contents are lost.

Once the valid flag is set, the file may not be modified.

Log Files

There is no command API interface to this feature. Log files are available only through the Native API File System API:

Log files are used when the file contents are not known in advance, and when small amounts of data are to be written to the file in future.

For example, an application that wakes, writes sensor readings, then goes to sleep or shuts down, can use a log file to accumulate sensor data over a period of time.

If all the data is immediately available, normal files are more appropriate than log files.

A log file is created with a specified total size. The contents of the file are left blank. Logs may then be written to the file at any point thereafter.

Each log has its own header which indicates the log's size and has a valid flag. When a log is appended, a header is created and the log data is written. Once all the log data is written, the log's valid flag is set.

Log File Format

When a log file is generated, its entire contents is erased (i.e. the contents is set to 0xFF).

When you call gos_log_file_append(), Gecko OS appends an individual log entry to the first blank (i.e. first 0xFFFF) in the file.

The log entry starts with a 16bit header. The first 15bits contain the log's length in bytes (little-endian). The MSb is the log entry valid flag. When the MSb of the log's header is cleared (i.e. 0) the log entry is valid. After the 16bit header is the log's contents with length specified in the header.

After the log entry data is written, the MSb of the header is cleared to indicate that the log is valid.

The general format of a log file is:

<header><log data><header><log data>....<header><log data>0xFFFF...

where <header> is:

<1bit valid flag><15bit log data length, little-endian>