Scenario introduction

When we use camera streaming to record (display) and CNN at the same time, usually what we get from the camera in the system is YUV streaming. CNN requires BGR format. When we use OpenCV to scale and convert the FHD YUV image The BGR format of 320*180 usually takes more than 10ms, which is already relatively long for the scene where we hope to achieve 30FPS. At this time, we will hope for a more efficient way.

OpenCV Code

Below is an example using OpenCV that converts a YUYV image to BGR format and scales it down to half of its original size:

/* cvt color & scale down to width / 2, height / 2 */
if (FILE_FORMAT == YUV_FMT_YUYV) {
    cv::cvtColor(yuvImage, bgrImage, cv::COLOR_YUV2BGR_YUYV);
}
else {
    cv::cvtColor(yuvImage, bgrImage, cv::COLOR_YUV2BGR_UYVY);
}
cv::resize(bgrImage, resizedBgrImage, cv::Size(FILE_WIDTH / 2, FILE_HEIGHT / 2), 0, 0, cv::INTER_LINEAR);

Performance comparison(1080P)

We named the efficient YUV -> BGR resize & convert method `YUV Converter`

...

• The "OpenCV" row represents the use of OpenCV entirely for format conversion and scaling operations.

• The "YUV Converter" row represents the use of the interface provided in this example entirely for format conversion and scaling operations.

YUV Converter Introduction

YUV Conveter is a sample code for YUV to BGR conversion and scaling specifically for the C3V platform. Its key features are as follows:

1. Supports YUV to BGR conversion.

2. Allows scaling the image size to a specified ratio while converting from YUV to BGR.

3. Supports YUYV and UYVY formats, which are commonly used on the C3V platform.

4. Utilizes ARM NEON acceleration, requiring only the inclusion of relevant .c and .h files, without the need for installing a bulky OpenCV library.

5. The efficiency of conversion plus scaling is higher than OpenCV. For example, using OpenCV to convert a 1080P YUV image to BGR and scale it to half the original size takes approximately 14ms, while the sample code can complete the same task in about 3ms.

How to use

1. Introduction to Core Files

...

• yuvToBgrByNeonScale: Converts YUV to BGR and scales the width and height to a specified factor, accelerated by NEON.

• yuvToBgrByNeonWHScale: Converts YUV to BGR and allows scaling the width and height to different factors, accelerated by NEON.

• yuvToBgrByNormScale: Converts YUV to BGR and scales the width and height to a specified factor, without NEON acceleration.

• yuvToBgrByNormWHScale: Converts YUV to BGR and allows scaling the width and height to different factors, without NEON acceleration.

• The return values of the above four functions are consistent with those in YUVConverter.h, and the meanings of the parameters with the same names are also the same. The different named parameters are as follows:

  • scaleFactor: This parameter determines the scaling factor for both width and height, which can be an integer multiple of 2, such as 2, 4, 6, ... 16, etc.

  • scaleFactorW: This parameter determines the scaling factor for the width, which can be an integer multiple of 2, such as 2, 4, 6, ... 16, etc.

  • scaleFactorH:This parameter determines the scaling factor for the height, which can be an integer multiple of 2, such as 2, 4, 6, ... 16, etc.

The API usage sample

1. Please refer to the code of the test function in MainTestRunner.cpp:

...

/* real test */
if (testType != MAIN_TEST_NEON_SCALE)
{
    printf("testType: %d(%s) vs %d(%s)\n",
           testType, MainTestRunner::getType(testType).c_str(),
           MAIN_TEST_CALC_COMM, MainTestRunner::getType(MAIN_TEST_CALC_COMM).c_str());

    auto testRunner1 = make_shared<MainTestRunner>(MAIN_TEST_CALC_COMM);
    auto testRunner2 = make_shared<MainTestRunner>(testType);

    outputDataSize = testRunner1->test(dumpFile, FRAME_FMT, FRAME_WIDTH, FRAME_HEIGHT, yuvBuffer, nullptr, outputBuffer1);
    outputDataSize = testRunner2->test(dumpFile, FRAME_FMT, FRAME_WIDTH, FRAME_HEIGHT, yuvBuffer, nullptr, outputBuffer2);

    if (outputBuffer1 != nullptr && outputBuffer2 != nullptr) {
        compaire_data(0, outputBuffer1, outputBuffer2, 0, outputDataSize);
    }
}
else
{
    printf("testType: %d(%s) vs %d(%s)\n",
           testType, MainTestRunner::getType(testType).c_str(),
           MAIN_TEST_CALC_SCALE, MainTestRunner::getType(MAIN_TEST_CALC_SCALE).c_str());

    auto testRunner1 = make_shared<MainTestRunner>(MAIN_TEST_CALC_SCALE);
    auto testRunner2 = make_shared<MainTestRunner>(testType);

    auto scaleFactors = vector<int>({2, 4, 6, 8, 10, 12, 14, 16});
    for (auto scaleFactorW : scaleFactors)
    {
        for (auto scaleFactorH : scaleFactors)
        {
            outputDataSize = testRunner1->test(scaleFactorW, scaleFactorH, dumpFile, FRAME_FMT, FRAME_WIDTH, FRAME_HEIGHT, yuvBuffer, nullptr, outputBuffer1);
            outputDataSize = testRunner2->test(scaleFactorW, scaleFactorH, dumpFile, FRAME_FMT, FRAME_WIDTH, FRAME_HEIGHT, yuvBuffer, nullptr, outputBuffer2);

            if (outputBuffer1 != nullptr && outputBuffer2 != nullptr) {
                compaire_data(0, outputBuffer1, outputBuffer2, 0, outputDataSize);
            }
        }
    }
}

Code

Please refer to the attachment for the sample code mentioned above and the implementation code of YUV Converter.

...