YUV Conversion and Scaling by ARM NEON Instructions

Owned by zengping
Last updated: Jul 19, 2024 by Wells Lu
9 min read

Scenario introduction

When we use camera streaming to record (display) and CNN at the same time, usually what we get from the camera in the system is YUV streaming. CNN requires BGR format. When we use OpenCV to scale and convert the FHD YUV image The BGR format of 320*180 usually takes more than 10ms, which is already relatively long for the scene where we hope to achieve 30FPS. At this time, we will hope for a more efficient way.

OpenCV Code

Below is an example using OpenCV that converts a YUYV image to BGR format and scales it down to half of its original size:

/* cvt color & scale down to width / 2, height / 2 */ if (FILE_FORMAT == YUV_FMT_YUYV) { cv::cvtColor(yuvImage, bgrImage, cv::COLOR_YUV2BGR_YUYV); } else { cv::cvtColor(yuvImage, bgrImage, cv::COLOR_YUV2BGR_UYVY); } cv::resize(bgrImage, resizedBgrImage, cv::Size(FILE_WIDTH / 2, FILE_HEIGHT / 2), 0, 0, cv::INTER_LINEAR);

Performance comparison(1080P)

We named the efficient YUV -> BGR resize & convert method `YUV Converter`

We use a BGR format that converts YUV from 1080P to 320*180 as an example. The measured efficiency comparison is as follows:

image-20240515-170635.png

  • The "OpenCV" row represents the use of OpenCV entirely for format conversion and scaling operations.

  • The "YUV Converter" row represents the use of the interface provided in this example entirely for format conversion and scaling operations.

YUV Converter Introduction

YUV Conveter is a sample code for YUV to BGR conversion and scaling specifically for the C3V platform. Its key features are as follows:

  1. Supports YUV to BGR conversion.

  2. Allows scaling the image size to a specified ratio while converting from YUV to BGR.

  3. Supports YUYV and UYVY formats, which are commonly used on the C3V platform.

  4. Utilizes ARM NEON acceleration, requiring only the inclusion of relevant .c and .h files, without the need for installing a bulky OpenCV library.

  5. The efficiency of conversion plus scaling is higher than OpenCV. For example, using OpenCV to convert a 1080P YUV image to BGR and scale it to half the original size takes approximately 14ms, while the sample code can complete the same task in about 3ms.

How to use

  1. Introduction to Core Files

sunplus@ubuntu:~/workspace/neon/optimize_samples$ ls -l sources/converter/ total 44 -rw-rw-r-- 1 sunplus sunplus 9253 May 15 03:32 YUVConverter.c -rw-rw-r-- 1 sunplus sunplus 732 May 15 03:32 YUVConverter.h -rw-rw-r-- 1 sunplus sunplus 13852 May 15 03:25 YUVConverterScale.c -rw-rw-r-- 1 sunplus sunplus 1077 May 15 03:25 YUVConverterScale.h -rw-rw-r-- 1 sunplus sunplus 492 May 15 03:25 YUVConverterTypes.h

  • YUVConverter.c/h: This file converts YUV images to BGR format without changing their size.

  • YUVConverterScale.c/h: This file converts YUV images to BGR format while scaling them down to a specified ratio.

  • YUVConverterTypes.h: This header file defines the supported YUV formats. The example code only supports two commonly used formats on c3v: YUYV and UYVY.

  1. Introduction to Test Files

sunplus@ubuntu:~/workspace/neon/optimize_samples$ ls sources/examples/ -l total 32 -rw-rw-r-- 1 sunplus sunplus 2303 May 15 05:24 converter_cv.cpp -rw-rw-r-- 1 sunplus sunplus 4643 May 15 05:22 main.cpp -rw-rw-r-- 1 sunplus sunplus 4875 May 15 03:25 MainTestRunner.cpp -rw-rw-r-- 1 sunplus sunplus 1197 May 15 03:25 MainTestRunner.h -rw-rw-r-- 1 sunplus sunplus 1359 May 15 03:25 MainTestUtil.cpp -rw-rw-r-- 1 sunplus sunplus 612 May 15 03:25 MainTestUtil.h

  • converter_cv.cpp: An example of conversion and scaling based on OpenCV.

  • main.cpp, MainTestRunner.cpp/h, MainTestUtil.cpp: Test files for the interfaces in YUVConverter.h and YUVConverterScale.h.

  • FHD_face.yuv: A 1080P YUV image in the YUYV format.

  • yuv2.uyvy: A 720P image in the UYVY format.

  1. As mentioned above, the interface provides test code, test files, and a makefile for reference, which can be used to compile and run the code.

  • If you are performing cross-compilation, please use the gcc version arm64-9.2 or arm64-10.2 (none-linux-gnu). The download links are available on the ARM official website. Additionally, if you do not have an OpenCV library for arm64, you may need to make some modifications to the makefile.

 

 

As shown in the figure, there are two things that need to be done: 1. Disable OpenCV, 2. Specify the cross-compiler path.

  • After executing the make command in the command line and completing the compilation, the generated binary file will be located under _out/bins.

  • Execution Example:

  1. API Introduction

  • YUVConverter.h

  • yuvToBgrByNeon: Converts YUV to BGR with the same size, accelerated by NEON.

  • yuvToBgrByNorm: Converts YUV to BGR with the same size, without NEON acceleration.

  • yuvToGrayByNeon: Converts YUV to grayscale with the same size, accelerated by NEON.

  • Parameters and Return Values for the Above Three Functions:

  • yuvFormat: Specifies the image format, either YUYV or UYVY.

  • width: The width of the image.

  • height: The height of the image.

  • yuvBuffer: The content of the YUV image.

  • rgbBuffer: The converted RGB image where RGB channel data is stored separately. The format in memory is as follows:

  1. bbbbb.....bbbb <-- width * height (blue channel)

  2. ggggg.....gggg <-- width * height (green channel)

  3. rrrrrrrr......rrrrrrr <-- width * height (red channel)

  • rgbBufferInterleaved: The converted RGB image where RGB pixels are stored interleaved. The format in memory is as follows:

  1. bgrbgrbgr.....bgrbgrbgr <-- width * height * 3 (interleaved BGR pixels)

  • Return Value: The size of the converted image. If the conversion fails, it returns 0.

YUVConverterScale.h

  • yuvToBgrByNeonScale: Converts YUV to BGR and scales the width and height to a specified factor, accelerated by NEON.

  • yuvToBgrByNeonWHScale: Converts YUV to BGR and allows scaling the width and height to different factors, accelerated by NEON.

  • yuvToBgrByNormScale: Converts YUV to BGR and scales the width and height to a specified factor, without NEON acceleration.

  • yuvToBgrByNormWHScale: Converts YUV to BGR and allows scaling the width and height to different factors, without NEON acceleration.

  • The return values of the above four functions are consistent with those in YUVConverter.h, and the meanings of the parameters with the same names are also the same. The different named parameters are as follows:

    • scaleFactor: This parameter determines the scaling factor for both width and height, which can be an integer multiple of 2, such as 2, 4, 6, ... 16, etc.

    • scaleFactorW: This parameter determines the scaling factor for the width, which can be an integer multiple of 2, such as 2, 4, 6, ... 16, etc.

    • scaleFactorH:This parameter determines the scaling factor for the height, which can be an integer multiple of 2, such as 2, 4, 6, ... 16, etc.

The API usage sample

  1. Please refer to the code of the test function in MainTestRunner.cpp:

  1. The caller code is as follows (located in main.cpp):

Code

Please refer to the attachment for the sample code mentioned above and the implementation code of YUV Converter.

Â