ESP-WHO Face Recognition with WebSocket Communication

Face Capture Interface

Using the ESP-WHO library to record faces with names and then display the name when a face is recognised.

This project uses the ArduinoWebsockets library for two way communication between the ESP32 and the browser. All the face detection, capturing and recognising are done on the ESP32. The browser sends instructions and receives notifications via WebSockets for updating the interface. The same WebSocket library is used to send the camera data to the browser as binary blobs.

Video showing the interface in action

The Interface

The interface consists a camera feed plus the following elements:

A status area, showing the current status of the ESP32:

Interface Status

A form field to enter the name of the person:

Interface Name

Four buttons that control the ESP32. They are STREAM to just stream the frames from the camera, DETECT for detecting faces in the stream, CAPTURE for capturing the current face and RECOGNISE for matching a face from the camera to a previous captured face:

Interface Buttons

If a face has been captured it can be seen in the list under Captured Faces. A face can be deleted by clicking the X next to it:

Interface Captured Face

DELETE ALL will delete all faces stored on the ESP32:

Interface Delete

Setting Up

If you’ve not set up or tested your ESP32 Camera in the Arduino IDE yet then please follow this tutorial first: You will also need to set up persistent storage on your board. Follow the steps under Persistent Storage Partition Scheme in this tutorial:

This application needs the latest version of the ESP32 package in the Arduino IDE. Update the ESP32 board library to 1.0.2 or higher. Tools > Board > Board Manager:

Arduino Board Manager

You also need to install the WebSockets library in the IDE by navigating Tools > Manage Libraries and searching for arduinowebsockets and installing it. Version 0.4.5 works for me.

Library Manager showing ArduinoWebsockets

Copy and paste the Sketch below and save it as a new Sketch. Add to the folder where the Sketch has been saved these two files: camera_index.h and camera_pins.h . camera_index.h is the HTML for the interface and camera_pins.h is the camera definitions.

In the pasted Sketch, edit the ssid and password to match your WiFi and uncomment the define for the camera you are using.

The Code

#include <ArduinoWebsockets.h>
#include "esp_http_server.h"
#include "esp_timer.h"
#include "esp_camera.h"
#include "camera_index.h"
#include "Arduino.h"
#include "fd_forward.h"
#include "fr_forward.h"
#include "fr_flash.h"

const char* ssid = "NSA";
const char* password = "orange";


// Select camera model
#include "camera_pins.h"

using namespace websockets;
WebsocketsServer socket_server;

camera_fb_t * fb = NULL;

long current_millis;
long last_detected_millis = 0;

void app_facenet_main();
void app_httpserver_init();

typedef struct
  uint8_t *image;
  box_array_t *net_boxes;
  dl_matrix3d_t *face_id;
} http_img_process_result;

static inline mtmn_config_t app_mtmn_config()
  mtmn_config_t mtmn_config = {0};
  mtmn_config.type = FAST;
  mtmn_config.min_face = 80;
  mtmn_config.pyramid = 0.707;
  mtmn_config.pyramid_times = 4;
  mtmn_config.p_threshold.score = 0.6;
  mtmn_config.p_threshold.nms = 0.7;
  mtmn_config.p_threshold.candidate_number = 20;
  mtmn_config.r_threshold.score = 0.7;
  mtmn_config.r_threshold.nms = 0.7;
  mtmn_config.r_threshold.candidate_number = 10;
  mtmn_config.o_threshold.score = 0.7;
  mtmn_config.o_threshold.nms = 0.7;
  mtmn_config.o_threshold.candidate_number = 1;
  return mtmn_config;
mtmn_config_t mtmn_config = app_mtmn_config();

face_id_name_list st_face_list;
static dl_matrix3du_t *aligned_face = NULL;

httpd_handle_t camera_httpd = NULL;

typedef enum
} en_fsm_state;
en_fsm_state g_state;

typedef struct
  char enroll_name[ENROLL_NAME_LEN];
} httpd_resp_value;

httpd_resp_value st_name;

void setup() {

  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  //init with high specs to pre-allocate larger buffers
  if (psramFound()) {
    config.frame_size = FRAMESIZE_UXGA;
    config.jpeg_quality = 10;
    config.fb_count = 2;
  } else {
    config.frame_size = FRAMESIZE_SVGA;
    config.jpeg_quality = 12;
    config.fb_count = 1;

  pinMode(13, INPUT_PULLUP);
  pinMode(14, INPUT_PULLUP);

  // camera init
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);

  sensor_t * s = esp_camera_sensor_get();
  s->set_framesize(s, FRAMESIZE_QVGA);

  s->set_vflip(s, 1);
  s->set_hmirror(s, 1);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
  Serial.println("WiFi connected");


  Serial.print("Camera Ready! Use 'http://");
  Serial.println("' to connect");

static esp_err_t index_handler(httpd_req_t *req) {
  httpd_resp_set_type(req, "text/html");
  httpd_resp_set_hdr(req, "Content-Encoding", "gzip");
  return httpd_resp_send(req, (const char *)index_ov2640_html_gz, index_ov2640_html_gz_len);

httpd_uri_t index_uri = {
  .uri       = "/",
  .method    = HTTP_GET,
  .handler   = index_handler,
  .user_ctx  = NULL

void app_httpserver_init ()
  httpd_config_t config = HTTPD_DEFAULT_CONFIG();
  if (httpd_start(&camera_httpd, &config) == ESP_OK)
    httpd_register_uri_handler(camera_httpd, &index_uri);

void app_facenet_main()
  face_id_name_init(&st_face_list, FACE_ID_SAVE_NUMBER, ENROLL_CONFIRM_TIMES);
  aligned_face = dl_matrix3du_alloc(1, FACE_WIDTH, FACE_HEIGHT, 3);

static inline int do_enrollment(face_id_name_list *face_list, dl_matrix3d_t *new_id)
  int left_sample_face = enroll_face_id_to_flash_with_name(face_list, new_id, st_name.enroll_name);
  ESP_LOGD(TAG, "Face ID %s Enrollment: Sample %d",
           ENROLL_CONFIRM_TIMES - left_sample_face);
  return left_sample_face;

void send_face_list(WebsocketsClient &client)
  client.send("delete_faces"); // tell browser to delete all faces
  face_id_node *head = st_face_list.head;
  char add_face[64];
  for (int i = 0; i < st_face_list.count; i++) // loop current faces
    sprintf(add_face, "listface:%s", head->id_name);
    client.send(add_face); //send face to browser
    head = head->next;

void delete_all_faces(WebsocketsClient &client)

void handle_message(WebsocketsClient &client, WebsocketsMessage msg)
  if ( == "stream") {
    g_state = START_STREAM;
  if ( == "detect") {
    g_state = START_DETECT;
  if (, 8) == "capture:") {
    g_state = START_ENROLL;
    char person[FACE_ID_SAVE_NUMBER * ENROLL_NAME_LEN] = {0,};, sizeof(person));
    memcpy(st_name.enroll_name, person, strlen(person) + 1);
  if ( == "recognise") {
    g_state = START_RECOGNITION;
  if (, 7) == "remove:") {
    char person[ENROLL_NAME_LEN * FACE_ID_SAVE_NUMBER];, sizeof(person));
    delete_face_id_in_flash_with_name(&st_face_list, person);
    send_face_list(client); // reset faces in the browser
  if ( == "delete_all") {

void loop() {
  auto client = socket_server.accept();
  dl_matrix3du_t *image_matrix = dl_matrix3du_alloc(1, 320, 240, 3);
  http_img_process_result out_res = {0};
  out_res.image = image_matrix->item;


  while (client.available()) {

    fb = esp_camera_fb_get();

    if (g_state == START_DETECT || g_state == START_ENROLL || g_state == START_RECOGNITION)
      out_res.net_boxes = NULL;
      out_res.face_id = NULL;

      fmt2rgb888(fb->buf, fb->len, fb->format, out_res.image);

      out_res.net_boxes = face_detect(image_matrix, &mtmn_config);

      if (out_res.net_boxes)
        if (align_face(out_res.net_boxes, image_matrix, aligned_face) == ESP_OK)

          out_res.face_id = get_face_id(aligned_face);
          last_detected_millis = millis();
          if (g_state == START_DETECT) {
            client.send("FACE DETECTED");

          if (g_state == START_ENROLL)
            int left_sample_face = do_enrollment(&st_face_list, out_res.face_id);
            char enrolling_message[64];
            sprintf(enrolling_message, "SAMPLE NUMBER %d FOR %s", ENROLL_CONFIRM_TIMES - left_sample_face, st_name.enroll_name);
            if (left_sample_face == 0)
              ESP_LOGI(TAG, "Enrolled Face ID: %s", st_face_list.tail->id_name);
              g_state = START_STREAM;
              char captured_message[64];
              sprintf(captured_message, "FACE CAPTURED FOR %s", st_face_list.tail->id_name);


          if (g_state == START_RECOGNITION  && (st_face_list.count > 0))
            face_id_node *f = recognize_face_with_name(&st_face_list, out_res.face_id);
            if (f)
              char recognised_message[64];
              sprintf(recognised_message, "RECOGNISED %s", f->id_name);
              client.send("FACE NOT RECOGNISED");

        if (g_state != START_DETECT) {
          client.send("NO FACE DETECTED");

      if (g_state == START_DETECT && millis() - last_detected_millis > 500) { // Detecting but no face detected


    client.sendBinary((const char *)fb->buf, fb->len);

    fb = NULL;


Arduino WebSocket library I used:
ESP-WHO WeChat Example for the face recognition with names code –
Hidden face photo by Honey Yanibel Minaya Cruz on Unsplash

91 Replies to “ESP-WHO Face Recognition with WebSocket Communication”

  1. Peter says:

    I have the same Problem with the the same Code for my ttgo pir board. It recognizes every face as enrolled face.
    After I downgraded to esp-face 0.34, everything is good. My Intension is a face recognation Background Task, wake up by pir and sending Image and message per MQTT. For watching i use a simple webserver.


    1. WordBot says:

      Hi, There’s a couple of issues and a possible solution on the ESP-WHO Github page:

  2. Cobiam says:

    Good day robotzero, first I thank you for your great work to publish and explain to a large extent this project of libre use. I am a manager of technological projects in a Colombian university and we want to know if we can in any way implement or embed this project in Moodle.
    We also need a good manual to emulate the use of this project on a Debian 9 virtual machine and later try to include it in Moodle for the use of facial recognition for the presentation of virtual exams.
    Thank you very much for your collaboration and support.
    Happy day.

    1. WordBot says:

      Hi there. I don’t think you will be able to use the ESP32 like this easily or at all. You are better to look for something in OpenCV or similar libraries.

  3. Felipe Messias says:

    Hello, Congratulations for the code, I added to this code an MQTT client that publishes messages to the Broker whenever it detects new faces, however I noticed 2 problems in the code, if I do not have the web interface of the camera running in browser there is no face detection and a another problem is that if I configure to recognize mode the camera image freezes in the browser in a few minutes, how could I fix these 2 problems?

    1. WordBot says:

      Hi, I’ll take a look tomorrow but it might be the websockets stuff will only work when you are running in a browser. This sketch is really to load up the faces and then you could use another sketch for normal operations (like this: I had a problem with it crashing when it’s been detecting a while but I was hoping it was a bug in the Arduino ESP32 library so I was waiting for the new version to be released. It might also be overheating. I have an ESP-EYE that is on the way out because of the heat generated when streaming the camera over WiFi.

  4. WordBot says:

    Do you see an error in the serial monitor? I’ve been testing and I get crashes but there doesn’t seem to be a pattern. This is the error I see:
    CORRUPT HEAP: multi_heap.c:308 detected at 0x3ffe7264
    I put a new camera in my ESP-EYE today because the heat had damaged the one it comes with.

  5. Felipe Messias says:

    Thanks for the answer, I’ll try this solution joining the two codes, in fact I checked some random errors, sometimes the camera rebooted, sometimes the camera stopped communicating with the camera, sometimes only the image in the recognizable mode crashed, hence I updated all files from esp32 direct from the github directory ( and started using the AI Thinker ESP 32 CAM card, I made the changes to use the partition created by you and practically a good part of the problems are gone, I just notice the camera restarting the times (every start an MQTT message is sent to my broker) and the image freeze in recognize mode, but other than that, all random errors were corrected

  6. Felipe Messias says:

    On the websocket library somehow when the line “auto client = socket_server.accept ();” is read in its code the main loop is “locked”, it is only “freed” after some client access the camera server via browser, still behaves strangely, the loop passes to re-read the function to each action taken in the camera web server interface, I tried to implement a reading of offline faces of the camera, however I was prevented by this, this solution that you suggested to me seems to be promising, I will try to implement and communicate you about the results

  7. David says:

    Hi Robotzero,

    Many thanks for sharing, the project is promised . Unfortunately I cant make it work. when arduino compile it complain with this:
    esp32CAM:48:15: error: ‘struct mtmn_config_t’ has no member named ‘type’
    mtmn_config.type = FAST;
    esp32CAM:51:15: error: ‘struct mtmn_config_t’ has no member named ‘pyramid_times’
    mtmn_config.pyramid_times = 4;

    I did tried with head files from the “ESP who GIT”, still same issue. Very strange since I can see TYPE & pyramid_times in the *.h file. So why did it complain?

    #include “c:\Users\tuan\Documents\Arduino\esp-who\components\esp-face\face_detection\include\fd_forward.h”
    #include “c:\Users\tuan\Documents\Arduino\esp-who\components\esp-face\face_recognition\include\fr_forward.h”
    #include “c:\Users\tuan\Documents\Arduino\esp-who\components\esp-face\face_recognition\include\fr_flash.h”

    My board is “esp32 dev module” , camera CAMERA_MODEL_AI_THINKER. My esp32-cam work fine with this example CameraWebServer.ino . can you see why?

    1. WordBot says:

      Which version of the ESP32 boards do you have in the Arduino IDE? I used 1.0.3rc1 in the tutorial. You shouldn’t need to link like that to include files. They might not be the correct versions. The guys at Espressif are making lots of changes to the code for ESP-WHO.

  8. David says:

    Thanks, it helps with 1.0.3rc1 !. no more error.
    i found the following line at that site
    add this in arduino preferences. Install board 1.0.3rc1 done

    it works fine, except I need to look at my SD card (32GB but only one primary partition 4Gb).

    17:52:45.303 -> ………….
    17:52:51.306 -> WiFi connected
    17:52:51.306 -> httpd_start
    17:52:51.306 -> E (11077) fr_flash: Not found
    17:52:51.306 -> Camera Ready! Use ‘’ to connect


  9. David says:

    a cup of java coffee is on the way, I wish I could buy 2 or 3 for you at once 🙂

  10. david says:

    hi robotzero
    is it the SD card
    I have insert a SD card with 4GB partition (the card is 32GB, I create 1 partition with windows fat32)
    I get this when start esp32
    17:52:51.306 -> httpd_start
    17:52:51.306 -> E (11077) fr_flash: Not found

    why fr_flash? there is a 4GB SD in the slot

    and when I click capture (after enter a name), nothing were saved as it show in your clip.

    1. WordBot says:

      fr_flash isn’t the SD card. It’s a partition on the flash memory on the board. Maybe try with this tutorial first to set up a new partition type..

  11. David says:

    thanks, i am now satisfied with your C-code on , quite stable, fast

  12. Alan says:

    Hi robotzero,
    I have been using your code to capture some faces and it has been working successfully.
    I am developing a different application that uses the recognised faces. To add more faces, I recompiled your code, flashed it to the ESP and I’m getting an error when a browser connects to the ESP32. The error appears immediately after the browser connects. The error shows that the socket has been disconnected. Confusingly, using Safari as the browser on an iMAC, theWeb Inspector reports the error as “WebSocket connection to ‘ws://’ failed: Could not decode a text frame as UTF-8.” whereas using Firefox as the browser the equivalent development tool (Web Console) shows the error as “The connection to ws:// was interrupted while the page was loading.”
    The serial monitor output is shown below:

    21:05:51.758 -> [D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled
    21:05:51.827 ->
    21:05:52.586 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 0 – WIFI_READY
    21:05:52.586 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 2 – STA_START
    21:05:52.796 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 4 – STA_CONNECTED
    21:05:52.831 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 7 – STA_GOT_IP
    21:05:52.831 -> [D][WiFiGeneric.cpp:379] _eventCallback(): STA IP:, MASK:, GW:
    21:05:53.071 -> .
    21:05:53.071 -> WiFi connected
    21:05:53.108 -> httpd_start
    21:05:53.108 -> Camera Ready! Use ‘’ to connect
    21:05:53.108 -> Code file: ESP-who-recognition-with-names
    21:05:58.803 -> [D][WiFiClient.cpp:482] connected(): Disconnected: RES: 0, ERR: 128

    I am using core 1.0.3-rc1 and the hardware works successfully with the example:
    ESP->camera->webserver successfully so I believe the hardware is OK. To make sure that the file “camera_index.h” had not become corrupt, I redownloaded the file and repeated the compilation but I still get the same error(s).
    I am using the most recent Arduino web sockets library (version 0.4.9) and since the error appears to be with the websockets library I downgraded to the original version you had used in your tutorial – version 0.4.0 but still get the same error(s).
    Can you advise how I might track down the error please?

    1. WordBot says:


      I’ve seen errors like this during various project testing but usually after running for a while. I’m not sure if it’s a bug in the WiFiClient library, a bug in my code, trying to process to much data, or just the module overheating and the WiFi quitting. I’ve been waiting for them to release 1.0.3 final to see if it gets better.

      This tools gives more information about what happened during a crash if the module crashes with an exception –

  13. Alan says:

    Hi there, many thanks for your speedy reply. The error occurs when connecting a browser to the ESP and is consistent. The code doesn’t crash so the exception decoder doesn’t help. I have been inserting Serial.print’s to trace exactly where the error occurs. I have narrowed the search down and the error occurs in the function “index_handler” when sending the file “index_ov2640_html_gz”. I have done quite a lot of searching for errors similar to ones reported in the browser and they seem to point to a problem in the WiFiClient library: reference:
    I do not believe there is a bug in your code. I plan to continue to track the problem down and will update you if I find anything meaningful. I too await formal release of 1.0.3.
    Many thanks for your response so far.

  14. MiaNguyen says:

    I’ve seen the errors likes this “face_id_name_list was not declared in this scope”. I installed the library “ArduinoWebsockets”, the camera_index.h and camera_pins.h.
    Please help me.
    Thanks a lot.

    1. WordBot says:

      Hi, Probably the wrong ESP32 Hardware Library installed. Which version do you have?

  15. MiaNguyen says:

    Dear Sir,
    I’ve checked ur reply and i updated the ESP32 Board to the 1.0.3 version. The web loaded ok, but when i try to capture a face (from a picture), it didnt save that face after take 5 samples. Actually , it turns from 1, 2, 3, 4 and 7!

    Thanks alot Sir.

    1. WordBot says:

      Hi. Try 1.0.4 the face recognition was fixed in that version.

  16. MiaNguyen says:

    And how to know that when i used the 8GB sd card, how many people can I capture?
    Capturing a person takes how many gb in the storage?
    Thanks a lot Sir

    1. WordBot says:

      Faces are captured on the board itself. It doesn’t use the SD card. One face is 2KB. This isn’t really meant for professional use. More for fun or possibly home security as an extra layer.

  17. Rudab says:

    i got error every time i delete a face . when i reset/restart the esp The list does not look properly arranged .
    the face is deleted from the flash and the list .
    here were the code disconnected :

    [E][WiFiClient.cpp:365] write(): fail on fd 63, errno: 104, “Connection reset by peer”

    i still did not find a solution .
    static esp_err_t send_face_list(WebsocketsClient &client)

    client.send(“delete_faces”); // tell browser to delete all faces
    face_id_node *head = st_face_list.head;
    char add_face[64];
    for (int i = 0; i id_name);
    client.send(add_face); // here the code crash
    head = head->next;


    1. WordBot says:

      You’ve replaced this line: sprintf(add_face, “listface:%s”, head->id_name);
      with your
      Serial.println(… line.

  18. Rudab says:

    when i copy/paste the code it got deleted in the comment , but i have it on the code . i only add Serial.println( .. to trace the code .

    1. WordBot says:

      I’ve just tested it again and it seems to work with the list when I reboot but… it does crash when I add a new face and then try to recognise it but not straight away…

      [D][CameraWebServerWeChatWebSockets.ino:217] do_enrollment(): Face ID face4 Enrollment: Sample 5
      [I][CameraWebServerWeChatWebSockets.ino:315] loop(): Enrolled Face ID: face4

      It crashes around line 328 here: Serial.println(f->id_name);
      I don’t know why though.

      I made a video:

      1. WordBot says:

        Must have been asleep.. sometimes there is no ‘f’ because a face is detected but not recognised! I’ve removed that line.

  19. Rudab says:

    you can change this line : if (f) ……to if (f != NULL ) to solve the problem ” always finds a matching face even when a face hasn’t been captured “

  20. Ralf says:

    Hi there
    Thanks for the great work. I tried it and it works fine. However, I have a problem that the connected relay only switches if there is an http connection to the esp32-cam modul. If I close the browser, the connected relay no longer works. Can you explain to me where I have to change the program accordingly so that it runs permanently even without an http browser.

    1. WordBot says:

      Hi. This version runs without a browser being connected – Or.. you should just be able to remove or comment out the code for the WebSocket connection once you have saved your faces. Buy me a coffee and I’ll write the code for you.

  21. Ralf says:

    My idea is this. The web module should be preserved in this way, but the detection and switching of the relay should also work if there is no active web session.
    In addition, it would be great if you could send an http call in addition to switching the relay, which could also be used as an argument, e.g. contains the recognized name (e.g. for control via fhem).
    The crowning glory would be if you still had a log (via web front end) in which you can see at what time a face was recognized and attached the picture (storage on an SD card).
    Up to what punk is that a coffee?

    1. WordBot says:

      Hmm.. I’ll take a look tomorrow at this.

    2. WordBot says:

      It’s not the most elegant code but this version has a button that disconnects the interface and then the recognition runs without websockets – I’ll try to work out a way for it to do this on a timeout or disconnection.

      Also this version has the http call you wanted. I’m looking at the log and picture on the SD card.

  22. Ralf says:

    Hi there
    Thank you for your efforts. I just tried it and it works as described.
    1.) However, the module should start with the detection and not first be brought into this state with “Disconnect”. The module should start detection as soon as the power supply is established.
    2.) Can not the “Disconnect” also take place as soon as the browser window is closed?
    Would be great if that could be adjusted.

    Greetings from Germany

    1. WordBot says:

      Hi, The problem with disconnections that no final packet is sent so the board doesn’t know it’s disconnected. I’m going to try and write around this using the http interface. I have saving the image to SD done now.

      I’m thinking of starting a premium section on the site and have basic and premium versions of my projects to make a bit of money. This might be the first one if I can get it working nicely.
      Thanks (from Spain)

  23. Lê Hữu Khoa says:

    How to add
    static void rgb_print(dl_matrix3du_t *image_matrix, uint32_t color, const char * str){
    fb_data_t fb;
    fb.width = image_matrix->w;
    fb.height = image_matrix->h; = image_matrix->item;
    fb.bytes_per_pixel = 3;
    fb.format = FB_BGR888;
    fb_gfx_print(&fb, (fb.width – (strlen(str) * 14)) / 2, 10, color, str);

    static int rgb_printf(dl_matrix3du_t *image_matrix, uint32_t color, const char *format, …){
    char loc_buf[64];
    char * temp = loc_buf;
    int len;
    va_list arg;
    va_list copy;
    va_start(arg, format);
    va_copy(copy, arg);
    len = vsnprintf(loc_buf, sizeof(loc_buf), format, arg);
    if(len >= sizeof(loc_buf)){
    temp = (char*)malloc(len+1);
    if(temp == NULL) {
    return 0;
    vsnprintf(temp, len+1, format, arg);
    rgb_print(image_matrix, color, temp);
    if(len > 64){
    return len;

    static void draw_face_boxes(dl_matrix3du_t *image_matrix, box_array_t *boxes, int face_id){
    int x, y, w, h, i;
    uint32_t color = FACE_COLOR_YELLOW;
    if(face_id 0){
    color = FACE_COLOR_GREEN;
    fb_data_t fb;
    fb.width = image_matrix->w;
    fb.height = image_matrix->h; = image_matrix->item;
    fb.bytes_per_pixel = 3;
    fb.format = FB_BGR888;
    for (i = 0; i len; i++){
    // rectangle box
    x = (int)boxes->box[i].box_p[0];
    y = (int)boxes->box[i].box_p[1];
    w = (int)boxes->box[i].box_p[2] – x + 1;
    h = (int)boxes->box[i].box_p[3] – y + 1;
    fb_gfx_drawFastHLine(&fb, x, y, w, color);
    fb_gfx_drawFastHLine(&fb, x, y+h-1, w, color);
    fb_gfx_drawFastVLine(&fb, x, y, h, color);
    fb_gfx_drawFastVLine(&fb, x+w-1, y, h, color);
    #if 0
    // landmark
    int x0, y0, j;
    for (j = 0; j landmark[i].landmark_p[j];
    y0 = (int)boxes->landmark[i].landmark_p[j+1];
    fb_gfx_fillRect(&fb, x0, y0, 3, 3, color);
    This code into your code

    1. WordBot says:

      You want to write onto the image frames so you see the green boxes etc in the stream? I didn’t use that code because it slows down the frame rate. It would be tricky to put it back but you can see how to do some of it in this sketch:

  24. Tuan says:

    Thank you for sharing the code with detailed instruction.
    It worked well in the first time with new partition as you suggested.
    However, after a couple of times, the video didn’t show anymore with that new partition.
    There’s no error when compiling and uploading.

    When I tried to reuse the existing partition “Huge APP”, there was video shown but the recognition function did not work since there is no “fr” in that partition.

    Please can you advise how to resolve this issue.
    Many thanks in advance.

    1. WordBot says:

      Hi, Is there still a problem? The video shouldn’t just stop appearing. Do you see anything in the serial monitor when the video isn’t showing?

  25. Tuan says:

    Still the same problem though I have tried to reset it many times. There is no warning/error shown on serial monitor apart from “Camera Ready! Use ‘’ to connect”

  26. Anton says:

    Where in this example is the code http pages, CSS styles? Where is it formed, where does it come from? I was unable to find anything related to this.

  27. Anton says:

    Thank you!

  28. Anton says:

    I downloaded the code from this example. It works great! Thanks for this!
    Then I started to refine it and as a result it turned out that even if I download the “CameraWebServer”, I get a web page exactly as in this code, but there is no image and status in the green field, there is also no information about the saved faces. This happens if “FaceRecognition(2621440 bytes with OTA)” is enabled. If you use “CameraWebServer” then “Huge APP(3MB no OTA/1MB SPIFFS)” works fine.
    I think there is some kind of problem with the new section “FaceRecognition(2621440 bytes with OTA)”, which manifests itself over time. The code for clearing the memory, which is given here: , it doesn’t solve the problem, data for the example from this page is not deleted. Even if I use a different partition scheme, for example Huge APP(3MB no OTA/1MB SPIFFS), And then back to the scheme FaceRecognition(2621440 bytes with OTA) old data remains unaffected.
    How to erase all data and start life from an empty disk?

    1. WordBot says:

      It sounds like you’ve over-written the camera_index.h file for the CameraWebServer example with the code from the camera_index.h for this tutorial. In other words.. the camera_index.h file contains the HTML for current project. It doesn’t get saved to the ESP32 anywhere.

  29. mike says:

    hi !
    i ‘ve uploaded the sketch complete but i can not enroll the humun face . I don t know reason why. is there something wrong ?

    => I will buy the coffee for you if you help me to code completely .

    i did detail as below :
    i used board mananger => esp32 1.0.4 new updated .
    rzo_partitions additional
    modified the boars.txt


    1. WordBot says:

      Hi, Do you see anything in the serial monitor?

  30. mike says:

    i can see detail serial monitor as below :
    22:06:00.034 -> dl_matrix3dqq_fc_with_bias, value > DL_QTP_MAX
    22:06:00.034 -> dl_matrix3dqq_fc_with_bias, value dl_matrix3dqq_fc_with_bias, value > DL_QTP_MAX
    => i can see the video steaming but When a click the capture button => no face detected .

    thank for your helps .

    1. WordBot says:

      Does the normal CameraWebServer example work for you?

  31. mike says:

    Camerawebserver example work without face recognition . Is there something wrong ? Plz help me to understand what happen ? May be missing something ?

    1. WordBot says:

      So you can detect faces when the Face Detection button is on but when you press the enrol button it doesn’t start the enrolling sequence? You can see what should happen in this video:

  32. Ivan A. says:

    Hi robotzero, I’ve an issue with this project. It works perfect but after some connections (I think between 20 and 30), the web page doesn’t display anymore the video and the users doesn’t show anymore, however, if I upload the CameraWebServer example from the Arduino IDE, the video shows perfectly, so it isn’t an issue of hardware. I think someone asked the same thing, but I couldn’t find the reason for this bug, I will buy you some coffees if you can help me 🙂

    1. WordBot says:

      Hi, So after 20 to 30 ‘recognitions’ it stops streaming? So if I want to test this I should add myself and step in front of the camera 30 times? I guess you don’t have it connected to serial to see what happens on the ESP32? Maybe in the browser console (F12) there is something. I’m guessing the Websocket fails and doesn’t start up again. I’ll run a test here tomorrow.

      1. Ivan A says:

        I´ve tested a new ESP32 Cam, and it took like 10 iterations to show this symptom. I don’t know if it’s something related to the partition, or erasing users and adding new ones.

        I’ve tried some things, and if I select the partition: “Huge APP” it does stream again in the page, but I can´t add faces and the one’s that were saved doesn’t appear.

        1. WordBot says:

          When you say iterations, Do you mean adding 10 people to the system or recognising the same person 10 times broke it?

  33. Ivan says:

    Adding, for example 3 users. And then eliminating 1 or 2 and adding other. Also connecting the camera, using it for like 5 minutes and turning it off. Reconnect again and using the recognition.
    I tried what you said about the console in the web page, and always say:

    HTTP error: status code 404, net ::ERR_UNKNOWN_URL_SCHEME

    1. WordBot says:

      Is it the code pasted from the page above? Or the code on Github?

      1. Ivan says:

        I will change the function and see what happens.

  34. Ivan says:

    From this blog. I didn’t know there was a GitHub link.

    1. WordBot says:

      Github has a version with a lock as seen here – but I’m testing with the code above at the moment.

    2. WordBot says:

      Can you try changing the send_face_list function to this:

      and look in the serial monitor for the listface: items. After doing some testing I found one of the names saved on the ESP32 was corrupted. Not sure why atm.

  35. Ivan says:

    Finally, it works!! Also I noticed that my custom PCB wasn´t giving enough current to the CAM, so when I started the web page it reset itself. Just that now with the corrected function the names that are shown in the interface are the same, also I can’t erase them individually,it just works if I click in the “DELETE ALL” button. Any advice?

    1. WordBot says:

      This is just a temporary fix to try and work out what is causing the occasional corruption of the names. Next time you get a problem, it you connect it to serial you will see listface:xxxxx for the names and one of them will be junk characters. The code also needs something to check if the Wi-Fi and Websocket connections are active and reconnect (or just reset the board) if not.

      1. Ivan says:

        I´ve been a little busy, but I I have bought you some coffees. Thanks a lot!

        1. WordBot says:

          I’ve updated the tutorial to reflect the new way of uploading partitions in release 1.0.5. I’ve created a new partition scheme as well, maybe this won’t have the corruption issue.

  36. Gerard Sanchez says:

    Hello, I have the same problem as Ivan. The code has worked fine until today. Today the streaming is not working.
    With the example code of streaming, my ESP32 is still working, so it’s not a hardware problem.

    In the serial monitor appears this:

    I’m using 1.0.5 version.

    1. WordBot says:

      I think there is a problem with the usernames getting corrupted somehow on the flash memory. Try the change here: and see if it starts working again. It’s just a test to see if that’s the problem.

      1. Gerard Sanchez says:

        Now it works, although the listface only says anonymous. But the camera works, and when I click recognise it recognises well.

        1. WordBot says:

          I think there’s a bug somewhere in the ESP library that causes it to corrupt the storage where the face data and name is stored.

  37. Abdelrahman Ahmed says:

    hey thank you for your explaining , but i follow every think as you said but iam getting this error below :

    C:\Users\Abdulrahman Ahmed\AppData\Local\Temp\arduino_build_865522\sketch\app_httpd.cpp: In function ‘esp_err_t index_handler(httpd_req_t*)’:

    app_httpd.cpp:587:51: error: ‘index_ov3660_html_gz’ was not declared in this scope

    return httpd_resp_send(req, (const char *)index_ov3660_html_gz, index_ov3660_html_gz_len);


    app_httpd.cpp:587:73: error: ‘index_ov3660_html_gz_len’ was not declared in this scope

    return httpd_resp_send(req, (const char *)index_ov3660_html_gz, index_ov3660_html_gz_len);


    Multiple libraries were found for “WiFi.h”
    Used: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi
    Not used: C:\Program Files (x86)\Arduino\libraries\WiFi
    Using library ArduinoWebsockets at version 0.5.0 in folder: C:\Users\Abdulrahman Ahmed\Documents\Arduino\libraries\ArduinoWebsockets
    Using library WiFi at version 1.0 in folder: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi
    Using library HTTPClient at version 1.2 in folder: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\HTTPClient
    Using library WiFiClientSecure at version 1.0 in folder: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFiClientSecure
    exit status 1
    ‘index_ov3660_html_gz’ was not declared in this scope

    1. WordBot says:

      Did you add the two other files to your sketch folder? “Add to the folder where the Sketch has been saved these two files: camera_index.h and camera_pins.h” from the tutorial.

  38. tim says:

    hi WordBot
    should these 2 files( camera_index.h and camera_pins.h ) be add to
    sketch folder? i was unable to find this sketch folder,
    where is this sketch folder? need some help, please

    1. WordBot says:

      Hi, Should look something like this:

  39. Vivek says:

    Hello , in your gitgub , you have two partitions files , which one to use , i get fr_flash error in serial monitor.
    You say in GitHub we must use partitions.csv , but in video you use the other partitions file ? so which one is it?
    I cannot stream the video in the web interface 🙁

    1. WordBot says:

      Depends which version of the ESP32 hardware library you have. partions.csv works with versions after 1.0.4

  40. Hexer says:

    Great project, I adjust it to my needs, added buttons for the doorbell and for manual open door. Takes a picture when doorbell pressed and stores it on the SD card. Every event (a button was pressed…) is stored on the SD card as log file for info or debugging. Using static AP address. Doorbell and manual opener works always, with or without connection to smartphone. Added buttons in WebSockets to take pics manually from smartphone, open door by smartphone button. Several Leds are indicating: door is open/closed, door lock activated/deaktivated. Working at the moment to activate a buzzer as warning indicator when door longer open than 10 min. On smartphone screen it shows the total space and used space of the SD card, here I have to fix the apperance. When the project is completed I will post it in youtube. The project is for my tiny house, having no internet connection yet. Keep going

  41. Rusky says:

    Hi, I have a question can I find somwhere the library fd_forward?

    1. WordBot says:

      Hi, which version of the ESP32 Hardware Library are you using?

      1. Rusky says:

        Hi thanks for answering so fast.
        If you mean the esp32 board version, I have installed the 1.0.6

        1. WordBot says:

          Can you try with 1.0.4 or 1.0.5. I don’t know if 1.0.6 works.

          1. Rusky says:

            With 1.0.5 I have the same error and with 1.0.4 I get another error :

            xtensa-esp32-elf-g++: error: unrecognized command line option ‘-mfix-esp32-psram-cache-strategy=memw’

            exit status 1

            Error compiling for board AI Thinker ESP32-CAM.

            1. WordBot says:

              Are you using version 1.x.x of the Arduino IDE?

              1. Rusky says:

                Yes I have the version : arduino-1.8.19-windows

            2. WordBot says:

              Do you these settings in the IDE: (although this doesn’t show the custom partition) ?
              I just tried compiling and I saw a couple of different errors from I think a change in the compiler so I updated a couple of functions (new code in tutorial) and it compiles. I have it working in the browser but I didn’t change the partitions so it doesn’t save the face data.
              You might find this one a bit easier because with 1.0.5 you don’t need to create the custom partition. The difference with this one is it controls a electronic door lock but you can leave that part out.
              I wonder if you have a problem with your installation it’s still not working. Does the CameraWebServer example work?

              1. Rusky says:

                Yes the CameraWebServer works, I will try the access control.


                1. Rusky says:

                  I’m having the same error about the fd_forward.h . My Arduino IDE can’t find this file. Is it an arduino library or one file you created?

                  1. Rusky says:

                    I’ve seen that in the CameraWebServer example don’t use the libraries “fd_forward.h”, “fr_forward.h” and “fr_flash.h” that are the libraries that my IDE can’t find

                    1. WordBot says:

                      Do you have this path on your computer?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

scroll to top