[ESP32-Korvo Review] (3) Compilation of ESP-Skainet Project

cruelfox

[ESP32-Korvo Review] (3) Compilation of ESP-Skainet Project [Copy link]

 

　　Since ESP-Korvo has hardware for voice processing, it is naturally necessary to have official software support. The focus of this review is the open software package ESP-Skainet provided by Espressif, which provides several API supports for audio and voice recognition, as well as the hardware driver library for this development board.
　　I cloned esp-skainet from github to my local computer. (There may be other more convenient ways to download) Because ESP-Skainet comes with a version of ESP-IDF, there is no need to download IDF separately. What is ESP-IDF? This is the software development framework for ESP32, which includes many components and is not a simple hardware support library. For example, IDF provides a bootloader, which does a lot more work than the STM32 MCU startup code; for example, IDF has FreeRTOS at the bottom, and the entry function for writing an application is app_main() instead of main(). IDF gives me the feeling that it is something closer to an operating system, hiding the ESP32 startup process, and it has all the BLE, network support, and file system support you can think of. Without IDF, it would be very difficult to develop for ESP32 (Lexin said: We have built the framework for you, just use it...)
　　With ESP-IDF, you can develop ESP32, so what to compile? Naturally, you need the gcc compiler tool for Xtensa processors, which can be downloaded from Lexin's website. If you want to tinker, you can also compile it yourself with the GCC source code. But the compiler alone is not enough, because each IDF project also needs to be configured, so you can't avoid using Lexin's Python scripts and auxiliary programs. This is very different from developing STM32/LPC/ATSAM..., right? No wonder, because IDF is really too complicated. If it is a Linux system, the problem of software tools depends on whether the system is installed enough. If it is a Windows system? You need an operating environment similar to Linux, such as cygwin, msys, and then use the ported software. Lexin gave another golden idea: Is it difficult to install? I have prepared a set for you, just download it and you can use it!
　　This time I used the compressed package esp32_win32_msys2_environment_and_esp2020r2_toolchain-20200601.zip which I downloaded directly. It contains the software in the msys2 environment and the Xtensa compilation tool. You can use it directly by unzipping it. The disadvantage is that it is not friendly to Windows XP. This is a problem with the msys2 version.

　　Start the msys2 terminal program (mingw32.exe) and try to compile the project examples/get_started in esp-skainet. Since there is already a Makefile, just make it directly.

#
# This is a project Makefile. It is assumed the directory this Makefile resides in is a
# project subdirectory.
#

PROJECT_NAME := get_started
EXTRA_COMPONENT_DIRS += ../../components/

include $(IDF_PATH)/make/project.mk

　　This Makefile contains the actual compilation instructions used in IDF, so you need to set the path of IDF:
export IDF_PATH=../../esp-idf
　　and then make it all the way to compile... It actually took a long time to compile, it doesn't look like an MCU project. After the end, you can find more than 100 megabytes of files in the generated build directory. I haven't configured the IDF component yet, I use the original default esp-skainet. If you want to configure, then execute make menuconfig , and a character interface similar to configuring the Linux kernel will appear:

　　After the compilation is complete, the following prompt will be displayed, explaining how to download to the board:
To flash all build output, run 'make flash' or:
python /d/esp-skainet/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 115200 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect 0x1000 /d/esp-skainet/examples/get_started/build/bootloader/bootloader.bin 0x10000 /d/esp-skainet/examples/get_started/build/get_started.bin 0x8000 /d/esp-skainet/examples/get_started/build/partitions.bin
　　There are three files to be burned: bootloader.bin, get_started.bin, partitions.bin
　　I used another machine connected to the board to download:

　　After reset, it runs normally, and the content printed by the serial port is different from before. It seems that the get_started example has started the voice detection, so let's wake it up and try it - it works.

　　Look at what is written in main.c. The main function is app_main()

void app_main()
{
    codec_init();
    rec_rb = rb_init(BUFFER_PROCESS, 8 * 1024, 1, NULL);
#ifdef CONFIG_ESP32_KORVO_V1_1_BOARD
    mase_rb = rb_init(BUFFER_PROCESS, 8 * 1024, 1, NULL);
#else
    ns_rb = rb_init(BUFFER_PROCESS, 8 * 1024, 1, NULL);
#endif
    agc_rb = rb_init(BUFFER_PROCESS, 8 * 1024, 1, NULL);

    model_iface_data_t *model_data = wakenet->create(model_coeff_getter, DET_MODE_90);
    model_data_mn = multinet->create(&MULTINET_COEFF, 4000);

    xTaskCreatePinnedToCore(&recsrcTask, "rec", 2 * 1024, NULL, 8, NULL, 1);
#ifdef CONFIG_ESP32_KORVO_V1_1_BOARD
    xTaskCreatePinnedToCore(&maseTask, "mase", 2 * 1024, NULL, 8, NULL, 1);
#else
    xTaskCreatePinnedToCore(&nsTask, "ns", 2 * 1024, NULL, 8, NULL, 1);
#endif
    xTaskCreatePinnedToCore(&agcTask, "agc", 2 * 1024, NULL, 8, NULL, 1);
    xTaskCreatePinnedToCore(&wakenetTask, "wakenet", 2 * 1024, (void*)model_data, 5, NULL, 0);

    printf("-----------awaits to be waken up-----------\n");
}

　　app_main() initializes and creates several tasks related to speech processing and recognition, and then ends. In the wakenetTask task, receive and process the speech:

void wakenetTask(void *arg)
{
    model_iface_data_t *model_data = arg;
    int frequency = wakenet->get_samp_rate(model_data);
    int audio_chunksize = wakenet->get_samp_chunksize(model_data);
    int chunk_num = multinet->get_samp_chunknum(model_data_mn);
    printf("chunk_num = %d\n", chunk_num);
    int16_t *buffer = malloc(audio_chunksize * sizeof(int16_t));
    assert(buffer);
    int chunks = 0;
    int mn_chunks = 0;
    bool detect_flag = 0;

    while (1) {
        rb_read(agc_rb, (uint8_t *)buffer, audio_chunksize * sizeof(int16_t), portMAX_DELAY);
        if (detect_flag == 0) {
            int r = wakenet->detect(model_data, buffer);
            if (r) {
                float ms = (chunks * audio_chunksize * 1000.0) / frequency;
                printf("%.2f: %s DETECTED.\n", (float)ms / 1000.0, wakenet->get_word_name(model_data, r));
                detect_flag = 1;
                printf("-----------------LISTENING-----------------\n\n");
                rb_reset(rec_rb);
                rb_reset(ns_rb);
                rb_reset(agc_rb);
            }
        } else {
            int command_id = multinet->detect(model_data_mn, buffer);
            mn_chunks++;
            if (mn_chunks == chunk_num || command_id > -1) {
                mn_chunks = 0;
                detect_flag = 0;
                if (command_id > -1) {
                    speech_commands_action(command_id);
                } else {
                    printf("can not recognize any speech commands\n");
                }

                printf("\n-----------awaits to be waken up-----------\n");
                rb_reset(rec_rb);
                rb_reset(ns_rb);
                rb_reset(agc_rb);
            }
        }
        chunks++;
    }
    vTaskDelete(NULL);
}

　　The code and the content printed by the serial port do correspond. You can try to change the code later.

freebsder

Thanks for sharing!

Jacktang

The test tool software can be downloaded from Espressif's website. Espressif's ecosystem is good.