Image Feature Generator#

The image feature generator process input image captured from camera and extracts required features to use with machine learning image classification applications using a camera as an image source.

Feature Generation#

The captured image pre-processed using crop and mean-std normalization based on model configuration used from application.

            Image
              │
      ┌───────▼───────┐
      │      Crop     │
      └───────┬───────┘
              │
      ┌───────▼───────┐
      │  Check mean   │
      └───────┬───────┘
              │
      ┌───────▼───────┐
      │ Normalization │
      └───────┬───────┘
              │
              ▼
            Model

Usage#

sl_ml_image_feature_generation_init() initializes the feature generation based on the configuration in sl_ml_image_feature_generation_config.h. It also initializes and starts the camera in streaming mode, which places the image samples into a ping-pong-buffer.

The image is pre-processed and filled in input tensor when sl_ml_image_feature_generation_fill_tensor() is called.

To retrieve the generated features sl_ml_image_feature_generation_fill_tensor() must be called.

Example#

When used with TensorFlow Lite Micro, the image feature generator can be used to fill a tensor directly by using sl_ml_image_feature_generation_fill_tensor(). However, the model has to be trained using the same feature generator configurations as used for inference, configured in sl_ml_image_feature_generation_config.h.

#include "sl_tflite_micro_init.h"
#include "sl_ml_image_feature_generation.h"

void main(void)
{
  sl_ml_image_feature_generation_init();

  while(1){ *
    if(do_inference){
      sl_ml_image_feature_generation_fill_tensor(sl_tflite_micro_get_input_tensor());
      sl_tflite_micro_get_interpreter()->Invoke();
    }

    ...

  }
}

Functions#

sl_status_t

sl_ml_image_feature_generation_init()

Set up the camera as an image source for feature generation.

sl_status_t

sl_ml_image_feature_generation_fill_tensor(TfLiteTensor *input_tensor)

Fill a TensorFlow tensor with feature data. Data type of input image for model will be selected based on model configuration.

sl_status_t

sl_ml_initialize_arducam_camera()

Set up the aurducam as an image source for feature generation and initialize the camera with configuration.

void

sl_ml_dump_image(const uint8_t *image_data, uint32_t image_length)

Set up the JLink stream as output node. Image data streamed over JLink to python interface.

void

sl_ml_retrieve_next_camera_image(uint8_t **image_data, uint32_t *image_size)

Retrieve image from ping-pog buffer captured from aurducam. Store image data to destination pointer.

void

sl_ml_image_crop_lut_init()

Initializes Look up table for croping image.

sl_status_t

sl_ml_image_feature_generation_get_image_scaled(uint8_t *image_data, float *buffer, size_t num_elements, float scaler)

Retrieve the features, scales them by the given scaler, and fills float buffer.

sl_status_t

sl_ml_get_image_mean(uint8_t *in_cam_image, float *mean, float *mean2, size_t num_elements)

Computes mean and mean square of input image.

sl_status_t

sl_ml_image_feature_generation_get_image_mean_std_normalized(uint8_t *image_data, float *buffer, float mean, float mean2, size_t num_elements)

Retrieve the features, normalizes them by centering about their mean and standard deviation and fills float buffer.

sl_status_t

sl_ml_image_feature_generation_get_image_raw_float32(uint8_t *image_data, float *buffer, size_t num_elements)

Retrieve the features as type float32 and copy them to the provided buffer.

void

sl_ml_image_feature_generation_reset()

Reset the state of the image feature generator.

void

sl_ml_convert_rgb565_to_gray(uint8_t *img)

Function to convert image color space to gray.

Function Documentation#

sl_ml_image_feature_generation_init#

sl_status_t sl_ml_image_feature_generation_init ()

Set up the camera as an image source for feature generation.

Returns

SL_STATUS_OK for success SL_STATUS_FAIL

sl_ml_image_feature_generation_fill_tensor#

sl_status_t sl_ml_image_feature_generation_fill_tensor (TfLiteTensor * input_tensor)

Fill a TensorFlow tensor with feature data. Data type of input image for model will be selected based on model configuration.

Parameters

Type	Direction	Argument Name	Description
TfLiteTensor *	[in]	input_tensor	The input tensor to fill with features.

Note

This function overwrites the entire input tensor.

Returns

SL_STATUS_OK for success SL_STATUS_INVALID_PARAMETER Tensor type or size does not correspond with configuration

sl_ml_initialize_arducam_camera#

sl_status_t sl_ml_initialize_arducam_camera ()

Set up the aurducam as an image source for feature generation and initialize the camera with configuration.

Returns

SL_STATUS_OK for success SL_STATUS_FAIL

sl_ml_dump_image#

void sl_ml_dump_image (const uint8_t * image_data, uint32_t image_length)

Set up the JLink stream as output node. Image data streamed over JLink to python interface.

Parameters

Type	Direction	Argument Name	Description
const uint8_t *	N/A	image_data
uint32_t	N/A	image_length

Returns

SL_STATUS_OK for success SL_STATUS_FAIL

sl_ml_retrieve_next_camera_image#

void sl_ml_retrieve_next_camera_image (uint8_t ** image_data, uint32_t * image_size)

Retrieve image from ping-pog buffer captured from aurducam. Store image data to destination pointer.

Parameters

Type	Direction	Argument Name	Description
uint8_t **	N/A	image_data
uint32_t *	N/A	image_size

sl_ml_image_crop_lut_init#

void sl_ml_image_crop_lut_init ()

Initializes Look up table for croping image.

sl_ml_image_feature_generation_get_image_scaled#

sl_status_t sl_ml_image_feature_generation_get_image_scaled (uint8_t * image_data, float * buffer, size_t num_elements, float scaler)

Retrieve the features, scales them by the given scaler, and fills float buffer.

Parameters

Type	Direction	Argument Name	Description
uint8_t *	[out]	image_data	Pointer to the buffer to store the scaled feature data
float *	[in]	buffer	The number of elements corresponding to the size of the buffer; If this is not large enough to store the entire feature buffer the function will return with an error.
size_t	[in]	num_elements	The scaling factor to apply to each feature value.
float	N/A	scaler

buffer = (float)image_data * scaler

Note

This function overwrites the entire buffer.

Returns

SL_STATUS_OK for success SL_STATUS_INVALID_PARAMETER num_elements too small

sl_ml_get_image_mean#

sl_status_t sl_ml_get_image_mean (uint8_t * in_cam_image, float * mean, float * mean2, size_t num_elements)

Computes mean and mean square of input image.

Parameters

Type	Direction	Argument Name	Description
uint8_t *	[out]	in_cam_image	Pointer to the mean to store the mean value of image.
float *	[out]	mean	Pointer to the mean2 to store the mean square value of image.
float *	[in]	mean2	The number of elements corresponding to the size of the buffer; If this is not large enough to store the entire feature buffer the function will return with an error.
size_t	N/A	num_elements

mean = ((float)image_data / (width*length)

Returns

SL_STATUS_OK for success SL_STATUS_INVALID_PARAMETER num_elements too small

sl_ml_image_feature_generation_get_image_mean_std_normalized#

sl_status_t sl_ml_image_feature_generation_get_image_mean_std_normalized (uint8_t * image_data, float * buffer, float mean, float mean2, size_t num_elements)

Retrieve the features, normalizes them by centering about their mean and standard deviation and fills float buffer.

Parameters

Type	Direction	Argument Name	Description
uint8_t *	[out]	image_data	Pointer to the buffer to store the normalized feature data
float *	[in]	buffer	float mean, mean value of image.
float	[in]	mean	float mean2, mean square value of image.
float	[in]	mean2	The number of elements corresponding to the size of the buffer; If this is not large enough to store the entire feature buffer the function will return with an error.
size_t	N/A	num_elements

buffer = ((float)image_data - mean(image_data)) / std(image_data)

Note

This function overwrites the entire buffer.

Returns

SL_STATUS_OK for success SL_STATUS_INVALID_PARAMETER num_elements too small

sl_ml_image_feature_generation_get_image_raw_float32#

sl_status_t sl_ml_image_feature_generation_get_image_raw_float32 (uint8_t * image_data, float * buffer, size_t num_elements)

Retrieve the features as type float32 and copy them to the provided buffer.

Parameters

Type	Direction	Argument Name	Description
uint8_t *	[out]	image_data	Pointer to the buffer to store the feature data
float *	[in]	buffer	The number of elements corresponding to the size of the buffer; If this is not large enough to store the entire feature buffer the function will return with an error.
size_t	N/A	num_elements

Note

This function overwrites the entire buffer.

Returns

SL_STATUS_OK for success SL_STATUS_INVALID_PARAMETER num_elements too small

sl_ml_image_feature_generation_reset#

void sl_ml_image_feature_generation_reset ()

Reset the state of the image feature generator.

sl_ml_convert_rgb565_to_gray#

void sl_ml_convert_rgb565_to_gray (uint8_t * img)

Function to convert image color space to gray.

Parameters

Type	Direction	Argument Name	Description
uint8_t *	[out]	img	Pointer holding reference to image data.