The function of this project use AMG8833 infrared sensor to recognize different gesture states.

The principle: On the SP7021 platform, AMG8833 recognizes the temperature under different gestures and then converts the temperature into an image, performs grayscale processing on the image and converts it to black and white. Under the PC, the image of different gestures is obtained based on tensorflow to achieve CNN model training, and the model file is obtained . Then on the SP7021 platform, the input image is recognized based on the model file.

1、Model training (Windows PC)

The train gesture pictures and obtain tensorflow model files can be implement in the Windows PC..

1.1   PC work environment setup

a)      Download and run python-3.7.7-amd64.exe to install python
b)     Download and run opencv-4.3.0-vc14_vc15.exe to install opencv
c)      After install python, run "pip3 install tensorflow" to install tensorflow
After the installation complete, execute python->import tensorflow/import cv2 in the dos command window to check whether the installation is correct, as shown below：

1.2   The train.py file

a)      num: the gesture number. If you want add the gesture type please modify this variable.
b)     The name of the folder storing the gesture pictures must be increased in sequence with "a", "b", "c"...
c)      The image suffix must end in .jpg

1.3   Model training

Put the pictures of different gestures obtained through the SP7021 platform to together with train.py. The default requirement is to store 6 folders of a/b/c/d/e/f. The gesture pictures stored in the folder a/b/c/d/e/f and the corresponding pictures are list as below:
Folder a: forward action
Folder b: intermediate action
Folder c: backward action
Folder d: idle state (no gesture detected)
Folder e: upward movement
Folder f: upward preparation


The 6 folders of pictures put together with train.py.

Execute "python train.py" in dos command window to train data. After the training is completed, the model file hand.tflite will be generated.
 

2、SP7021 Platform Environment Setup

a) Install raspberry image，Download the latest image file from the Raspberry Pi website and write it into sdcard by Win32DiskImager.
Download path: https://www.raspberrypi.org/downloads/raspberry-pi-os/
The Win32DiskImager tool shown as below figure.

b) Enable i2c port in sp7021-bpi-f2s.dts of linux source code

Please connect to Sunplus Yocto server to get SP7021 source bin file and package. PLUS1 Yocto. (http://113.196.136.132/)

c) Copy the 5 boot files of SP7021 to the boot partition of the sdcard

d) After SP7021 boot, run "sudo raspi-config --expand-rootfs" to expand rootfs partition

e) Run "sudo apt install cmake libopencv-dev" to install cmake and opencv

3、 Download test code and compile

a)     Download test code from below icon.

There is already a tensorflow library file in the test code. The main operation code of thermal cam also inside.
 

b) Compile thermal_cam

1. Copy and decompressed the thermal_cam.tar.gz from sdcard to work folder
   
2. run "make" to compile thermal_cam.cpp.
   
3. After the compilation is complete, the thermal_cam file will be generated
   

4、Run thermal_cam：

a)      Test mode :run   ./themal_cam temp_L temp_H hand.tflite

Parameter Description：
·        temp_L：Minimum temperature
·        temp_H：Maximum temperature
·        hand.tflite：Model files generated by training on the PC side based on the collected pictures

Note1: Because the AMG8833 thermal imaging sensor can only detect the temperature value of the 8*8 array, then we will correspond the temperature value to the color (green to red) according to the set size range. In this way, different colors can be used to indicate different temperature ranges. For better identification, it is necessary to set the appropriate temperature range value according to the current ambient temperature

Note2: The figure change rule define as below.
1->0  light to blue bar
0->1 or 2->1  light to green bar
1->2  light to red bar
5->4  bar become higher
4-> 0, 1, 2  bar become lower

 
b)     Collection mode: run  ./themal_cam temp_L temp_H

Run the above command, the screen will display 2 screens, ori_image and new_image
·        ori_image：AMG8833 Get the temperature and convert the corresponding color image
·        New_image：The image obtained by ori_image after grayscale processing, black and white is convenient for model training and recognition

At this time, you can input the'a','b','c','d','e','f' buttons through the keyboard to save the gesture action images of different actions to the corresponding folder. Then copy it to the corresponding directory of the PC environment for data training. (Each input saves 1000 images)
KEY a: forward action
KEY b: intermediate action
KEY c: backward action
KEY d: idle state (no gesture detected)
KEY e: upward movement
KEY f: upward preparation

5、Operation flow chart