Infrared Gesture Detection
The function of this project use AMG8833 infrared sensor to recognize different gesture states.
The principle: On the SP7021 platform, AMG8833 recognizes the temperature under different gestures and then converts the temperature into an image, performs grayscale processing on the image and converts it to black and white. Under the PC, the image of different gestures is obtained based on tensorflow to achieve CNN model training, and the model file is obtained . Then on the SP7021 platform, the input image is recognized based on the model file.
1、Model training (Windows PC)
The train gesture pictures and obtain tensorflow model files can be implement in the Windows PC..
1.1 PC work environment setup
a) Download and run python-3.7.7-amd64.exe to install python
b) Download and run opencv-4.3.0-vc14_vc15.exe to install opencv
c) After install python, run "pip3 install tensorflow" to install tensorflow
After the installation complete, execute python->import tensorflow/import cv2 in the dos command window to check whether the installation is correct, as shown below：
1.2 The train.py file
a) num: the gesture number. If you want add the gesture type please modify this variable.
b) The name of the folder storing the gesture pictures must be increased in sequence with "a", "b", "c"...
c) The image suffix must end in .jpg
1.3 Model training
Put the pictures of different gestures obtained through the SP7021 platform to together with train.py. The default requirement is to store 6 folders of a/b/c/d/e/f. The gesture pictures stored in the folder a/b/c/d/e/f and the corresponding pictures are list as below:
Folder a: forward action
Folder b: intermediate action
Folder c: backward action
Folder d: idle state (no gesture detected)
Folder e: upward movement
Folder f: upward preparation
The 6 folders of pictures put together with train.py.
Execute "python train.py" in dos command window to train data. After the training is completed, the model file hand.tflite will be generated.
2、SP7021 Platform Environment Setup
a) Install raspberry image，Download the latest image file from the Raspberry Pi website and write it into sdcard by Win32DiskImager.
Download path: https://www.raspberrypi.org/downloads/raspberry-pi-os/
The Win32DiskImager tool shown as below figure.
b) Enable i2c port in sp7021-bpi-f2s.dts of linux source code
Please connect to Sunplus Yocto server to get SP7021 source bin file and package. PLUS1 Yocto. (http://113.196.136.132/)
c) Copy the 5 boot files of SP7021 to the boot partition of the sdcard
d) After SP7021 boot, run "sudo raspi-config --expand-rootfs" to expand rootfs partition
e) Run "sudo apt install cmake libopencv-dev" to install cmake and opencv
3、 Download test code and compile
a) Download test code from below icon.
There is already a tensorflow library file in the test code. The main operation code of thermal cam also inside.
b) Compile thermal_cam
Copy and decompressed the thermal_cam.tar.gz from sdcard to work folder
run "make" to compile thermal_cam.cpp.
After the compilation is complete, the thermal_cam file will be generated
4、Run thermal_cam：
a) Test mode :run ./themal_cam temp_L temp_H hand.tflite
Parameter Description：
· temp_L：Minimum temperature
· temp_H：Maximum temperature
· hand.tflite：Model files generated by training on the PC side based on the collected pictures
Note1: Because the AMG8833 thermal imaging sensor can only detect the temperature value of the 8*8 array, then we will correspond the temperature value to the color (green to red) according to the set size range. In this way, different colors can be used to indicate different temperature ranges. For better identification, it is necessary to set the appropriate temperature range value according to the current ambient temperature