Using the ESP-WHO library to record faces with names and then display the name when a face is recognised.
This project uses the ArduinoWebsockets library for two way communication between the ESP32 and the browser. All the face detection, capturing and recognising are done on the ESP32. The browser sends instructions and receives notifications via WebSockets for updating the interface. The same WebSocket library is used to send the camera data to the browser as binary blobs.
The Interface
The interface consists a camera feed plus the following elements:
A status area, showing the current status of the ESP32:
A form field to enter the name of the person:
Four buttons that control the ESP32. They are STREAM to just stream the frames from the camera, DETECT for detecting faces in the stream, CAPTURE for capturing the current face and RECOGNISE for matching a face from the camera to a previous captured face:
If a face has been captured it can be seen in the list under Captured Faces. A face can be deleted by clicking the X next to it:
DELETE ALL will delete all faces stored on the ESP32:
Setting Up
If you’ve not set up or tested your ESP32 Camera in the Arduino IDE yet then please follow this tutorial first: https://robotzero.one/esp32-cam-arduino-ide/. You will also need to set up persistent storage on your board. Follow the steps under Persistent Storage Partition Scheme in this tutorial: https://robotzero.one/esp32-face-door-entry/
This application needs the latest version of the ESP32 package in the Arduino IDE. Update the ESP32 board library to 1.0.2 or higher. Tools > Board > Board Manager:
You also need to install the WebSockets library in the IDE by navigating Tools > Manage Libraries and searching for arduinowebsockets and installing it. Version 0.4.5 works for me.
Copy and paste the Sketch below and save it as a new Sketch. Add to the folder where the Sketch has been saved these two files: camera_index.h and camera_pins.h . camera_index.h is the HTML for the interface and camera_pins.h is the camera definitions.
In the pasted Sketch, edit the ssid and password to match your WiFi and uncomment the define for the camera you are using.
The Code
#include <ArduinoWebsockets.h> #include "esp_http_server.h" #include "esp_timer.h" #include "esp_camera.h" #include "camera_index.h" #include "Arduino.h" #include "fd_forward.h" #include "fr_forward.h" #include "fr_flash.h" const char* ssid = "NSA"; const char* password = "orange"; #define ENROLL_CONFIRM_TIMES 5 #define FACE_ID_SAVE_NUMBER 7 // Select camera model //#define CAMERA_MODEL_WROVER_KIT //#define CAMERA_MODEL_ESP_EYE //#define CAMERA_MODEL_M5STACK_PSRAM //#define CAMERA_MODEL_M5STACK_WIDE #define CAMERA_MODEL_AI_THINKER #include "camera_pins.h" using namespace websockets; WebsocketsServer socket_server; camera_fb_t * fb = NULL; long current_millis; long last_detected_millis = 0; void app_facenet_main(); void app_httpserver_init(); typedef struct { uint8_t *image; box_array_t *net_boxes; dl_matrix3d_t *face_id; } http_img_process_result; static inline mtmn_config_t app_mtmn_config() { mtmn_config_t mtmn_config = {0}; mtmn_config.type = FAST; mtmn_config.min_face = 80; mtmn_config.pyramid = 0.707; mtmn_config.pyramid_times = 4; mtmn_config.p_threshold.score = 0.6; mtmn_config.p_threshold.nms = 0.7; mtmn_config.p_threshold.candidate_number = 20; mtmn_config.r_threshold.score = 0.7; mtmn_config.r_threshold.nms = 0.7; mtmn_config.r_threshold.candidate_number = 10; mtmn_config.o_threshold.score = 0.7; mtmn_config.o_threshold.nms = 0.7; mtmn_config.o_threshold.candidate_number = 1; return mtmn_config; } mtmn_config_t mtmn_config = app_mtmn_config(); face_id_name_list st_face_list; static dl_matrix3du_t *aligned_face = NULL; httpd_handle_t camera_httpd = NULL; typedef enum { START_STREAM, START_DETECT, SHOW_FACES, START_RECOGNITION, START_ENROLL, ENROLL_COMPLETE, DELETE_ALL, } en_fsm_state; en_fsm_state g_state; typedef struct { char enroll_name[ENROLL_NAME_LEN]; } httpd_resp_value; httpd_resp_value st_name; void setup() { Serial.begin(115200); Serial.setDebugOutput(true); Serial.println(); camera_config_t config; config.ledc_channel = LEDC_CHANNEL_0; config.ledc_timer = LEDC_TIMER_0; config.pin_d0 = Y2_GPIO_NUM; config.pin_d1 = Y3_GPIO_NUM; config.pin_d2 = Y4_GPIO_NUM; config.pin_d3 = Y5_GPIO_NUM; config.pin_d4 = Y6_GPIO_NUM; config.pin_d5 = Y7_GPIO_NUM; config.pin_d6 = Y8_GPIO_NUM; config.pin_d7 = Y9_GPIO_NUM; config.pin_xclk = XCLK_GPIO_NUM; config.pin_pclk = PCLK_GPIO_NUM; config.pin_vsync = VSYNC_GPIO_NUM; config.pin_href = HREF_GPIO_NUM; config.pin_sscb_sda = SIOD_GPIO_NUM; config.pin_sscb_scl = SIOC_GPIO_NUM; config.pin_pwdn = PWDN_GPIO_NUM; config.pin_reset = RESET_GPIO_NUM; config.xclk_freq_hz = 20000000; config.pixel_format = PIXFORMAT_JPEG; //init with high specs to pre-allocate larger buffers if (psramFound()) { config.frame_size = FRAMESIZE_UXGA; config.jpeg_quality = 10; config.fb_count = 2; } else { config.frame_size = FRAMESIZE_SVGA; config.jpeg_quality = 12; config.fb_count = 1; } #if defined(CAMERA_MODEL_ESP_EYE) pinMode(13, INPUT_PULLUP); pinMode(14, INPUT_PULLUP); #endif // camera init esp_err_t err = esp_camera_init(&config); if (err != ESP_OK) { Serial.printf("Camera init failed with error 0x%x", err); return; } sensor_t * s = esp_camera_sensor_get(); s->set_framesize(s, FRAMESIZE_QVGA); #if defined(CAMERA_MODEL_M5STACK_WIDE) s->set_vflip(s, 1); s->set_hmirror(s, 1); #endif WiFi.begin(ssid, password); while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); Serial.println("WiFi connected"); app_httpserver_init(); app_facenet_main(); socket_server.listen(82); Serial.print("Camera Ready! Use 'http://"); Serial.print(WiFi.localIP()); Serial.println("' to connect"); } static esp_err_t index_handler(httpd_req_t *req) { httpd_resp_set_type(req, "text/html"); httpd_resp_set_hdr(req, "Content-Encoding", "gzip"); return httpd_resp_send(req, (const char *)index_ov2640_html_gz, index_ov2640_html_gz_len); } httpd_uri_t index_uri = { .uri = "/", .method = HTTP_GET, .handler = index_handler, .user_ctx = NULL }; void app_httpserver_init () { httpd_config_t config = HTTPD_DEFAULT_CONFIG(); if (httpd_start(&camera_httpd, &config) == ESP_OK) Serial.println("httpd_start"); { httpd_register_uri_handler(camera_httpd, &index_uri); } } void app_facenet_main() { face_id_name_init(&st_face_list, FACE_ID_SAVE_NUMBER, ENROLL_CONFIRM_TIMES); aligned_face = dl_matrix3du_alloc(1, FACE_WIDTH, FACE_HEIGHT, 3); read_face_id_from_flash_with_name(&st_face_list); } static inline int do_enrollment(face_id_name_list *face_list, dl_matrix3d_t *new_id) { ESP_LOGD(TAG, "START ENROLLING"); int left_sample_face = enroll_face_id_to_flash_with_name(face_list, new_id, st_name.enroll_name); ESP_LOGD(TAG, "Face ID %s Enrollment: Sample %d", st_name.enroll_name, ENROLL_CONFIRM_TIMES - left_sample_face); return left_sample_face; } void send_face_list(WebsocketsClient &client) { client.send("delete_faces"); // tell browser to delete all faces face_id_node *head = st_face_list.head; char add_face[64]; for (int i = 0; i < st_face_list.count; i++) // loop current faces { sprintf(add_face, "listface:%s", head->id_name); client.send(add_face); //send face to browser head = head->next; } } void delete_all_faces(WebsocketsClient &client) { delete_face_all_in_flash_with_name(&st_face_list); client.send("delete_faces"); } void handle_message(WebsocketsClient &client, WebsocketsMessage msg) { if (msg.data() == "stream") { g_state = START_STREAM; client.send("STREAMING"); } if (msg.data() == "detect") { g_state = START_DETECT; client.send("DETECTING"); } if (msg.data().substring(0, 8) == "capture:") { g_state = START_ENROLL; char person[FACE_ID_SAVE_NUMBER * ENROLL_NAME_LEN] = {0,}; msg.data().substring(8).toCharArray(person, sizeof(person)); memcpy(st_name.enroll_name, person, strlen(person) + 1); client.send("CAPTURING"); } if (msg.data() == "recognise") { g_state = START_RECOGNITION; client.send("RECOGNISING"); } if (msg.data().substring(0, 7) == "remove:") { char person[ENROLL_NAME_LEN * FACE_ID_SAVE_NUMBER]; msg.data().substring(7).toCharArray(person, sizeof(person)); delete_face_id_in_flash_with_name(&st_face_list, person); send_face_list(client); // reset faces in the browser } if (msg.data() == "delete_all") { delete_all_faces(client); } } void loop() { auto client = socket_server.accept(); client.onMessage(handle_message); dl_matrix3du_t *image_matrix = dl_matrix3du_alloc(1, 320, 240, 3); http_img_process_result out_res = {0}; out_res.image = image_matrix->item; send_face_list(client); client.send("STREAMING"); while (client.available()) { client.poll(); fb = esp_camera_fb_get(); if (g_state == START_DETECT || g_state == START_ENROLL || g_state == START_RECOGNITION) { out_res.net_boxes = NULL; out_res.face_id = NULL; fmt2rgb888(fb->buf, fb->len, fb->format, out_res.image); out_res.net_boxes = face_detect(image_matrix, &mtmn_config); if (out_res.net_boxes) { if (align_face(out_res.net_boxes, image_matrix, aligned_face) == ESP_OK) { out_res.face_id = get_face_id(aligned_face); last_detected_millis = millis(); if (g_state == START_DETECT) { client.send("FACE DETECTED"); } if (g_state == START_ENROLL) { int left_sample_face = do_enrollment(&st_face_list, out_res.face_id); char enrolling_message[64]; sprintf(enrolling_message, "SAMPLE NUMBER %d FOR %s", ENROLL_CONFIRM_TIMES - left_sample_face, st_name.enroll_name); client.send(enrolling_message); if (left_sample_face == 0) { ESP_LOGI(TAG, "Enrolled Face ID: %s", st_face_list.tail->id_name); g_state = START_STREAM; char captured_message[64]; sprintf(captured_message, "FACE CAPTURED FOR %s", st_face_list.tail->id_name); client.send(captured_message); send_face_list(client); } } if (g_state == START_RECOGNITION && (st_face_list.count > 0)) { face_id_node *f = recognize_face_with_name(&st_face_list, out_res.face_id); if (f) { char recognised_message[64]; sprintf(recognised_message, "RECOGNISED %s", f->id_name); client.send(recognised_message); } else { client.send("FACE NOT RECOGNISED"); } } dl_matrix3d_free(out_res.face_id); } } else { if (g_state != START_DETECT) { client.send("NO FACE DETECTED"); } } if (g_state == START_DETECT && millis() - last_detected_millis > 500) { // Detecting but no face detected client.send("DETECTING"); } } client.sendBinary((const char *)fb->buf, fb->len); esp_camera_fb_return(fb); fb = NULL; } }
References
Arduino WebSocket library I used: https://github.com/gilmaimon/ArduinoWebsockets
ESP-WHO WeChat Example for the face recognition with names code – https://github.com/espressif/esp-who/tree/master/examples/single_chip/recognition_wechat
Hidden face photo by Honey Yanibel Minaya Cruz on Unsplash
Buy Me A Coffee
If you found something useful above please say thanks by buying me a coffee here...
Hello,
I have the same Problem with the the same Code for my ttgo pir board. It recognizes every face as enrolled face.
After I downgraded to esp-face 0.34, everything is good. My Intension is a face recognation Background Task, wake up by pir and sending Image and message per MQTT. For watching i use a simple webserver.
Peter
Hi, There’s a couple of issues and a possible solution on the ESP-WHO Github page: https://github.com/espressif/esp-who/issues/71
Good day robotzero, first I thank you for your great work to publish and explain to a large extent this project of libre use. I am a manager of technological projects in a Colombian university and we want to know if we can in any way implement or embed this project in Moodle.
We also need a good manual to emulate the use of this project on a Debian 9 virtual machine and later try to include it in Moodle for the use of facial recognition for the presentation of virtual exams.
Thank you very much for your collaboration and support.
Happy day.
Hi there. I don’t think you will be able to use the ESP32 like this easily or at all. You are better to look for something in OpenCV or similar libraries. https://opencv.org/about/
Hello, Congratulations for the code, I added to this code an MQTT client that publishes messages to the Broker whenever it detects new faces, however I noticed 2 problems in the code, if I do not have the web interface of the camera running in browser there is no face detection and a another problem is that if I configure to recognize mode the camera image freezes in the browser in a few minutes, how could I fix these 2 problems?
Hi, I’ll take a look tomorrow but it might be the websockets stuff will only work when you are running in a browser. This sketch is really to load up the faces and then you could use another sketch for normal operations (like this: https://robotzero.one/face-recognition-with-blynk/) I had a problem with it crashing when it’s been detecting a while but I was hoping it was a bug in the Arduino ESP32 library so I was waiting for the new version to be released. It might also be overheating. I have an ESP-EYE that is on the way out because of the heat generated when streaming the camera over WiFi.
Do you see an error in the serial monitor? I’ve been testing and I get crashes but there doesn’t seem to be a pattern. This is the error I see:
CORRUPT HEAP: multi_heap.c:308 detected at 0x3ffe7264
I put a new camera in my ESP-EYE today because the heat had damaged the one it comes with.
Thanks for the answer, I’ll try this solution joining the two codes, in fact I checked some random errors, sometimes the camera rebooted, sometimes the camera stopped communicating with the camera, sometimes only the image in the recognizable mode crashed, hence I updated all files from esp32 direct from the github directory (https://github.com/espressif/arduino-esp32) and started using the AI Thinker ESP 32 CAM card, I made the changes to use the partition created by you and practically a good part of the problems are gone, I just notice the camera restarting the times (every start an MQTT message is sent to my broker) and the image freeze in recognize mode, but other than that, all random errors were corrected
On the websocket library somehow when the line “auto client = socket_server.accept ();” is read in its code the main loop is “locked”, it is only “freed” after some client access the camera server via browser, still behaves strangely, the loop passes to re-read the function to each action taken in the camera web server interface, I tried to implement a reading of offline faces of the camera, however I was prevented by this, this solution that you suggested to me seems to be promising, I will try to implement and communicate you about the results
Hi Robotzero,
Many thanks for sharing, the project is promised . Unfortunately I cant make it work. when arduino compile it complain with this:
esp32CAM:48:15: error: ‘struct mtmn_config_t’ has no member named ‘type’
mtmn_config.type = FAST;
esp32CAM:51:15: error: ‘struct mtmn_config_t’ has no member named ‘pyramid_times’
mtmn_config.pyramid_times = 4;
I did tried with head files from the “ESP who GIT”, still same issue. Very strange since I can see TYPE & pyramid_times in the *.h file. So why did it complain?
#include “c:\Users\tuan\Documents\Arduino\esp-who\components\esp-face\face_detection\include\fd_forward.h”
#include “c:\Users\tuan\Documents\Arduino\esp-who\components\esp-face\face_recognition\include\fr_forward.h”
#include “c:\Users\tuan\Documents\Arduino\esp-who\components\esp-face\face_recognition\include\fr_flash.h”
My board is “esp32 dev module” , camera CAMERA_MODEL_AI_THINKER. My esp32-cam work fine with this example CameraWebServer.ino . can you see why?
Hi,
Which version of the ESP32 boards do you have in the Arduino IDE? I used 1.0.3rc1 in the tutorial. You shouldn’t need to link like that to include files. They might not be the correct versions. The guys at Espressif are making lots of changes to the code for ESP-WHO.
hi
Thanks, it helps with 1.0.3rc1 !. no more error.
https://github.com/espressif/arduino-esp32/releases/tag/1.0.3-rc1
i found the following line at that site
https://github.com/espressif/arduino-esp32/releases/download/1.0.3-rc1/package_esp32_dev_index.json
add this in arduino preferences. Install board 1.0.3rc1 done
c:\Users\tuan\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.3-rc1\
it works fine, except I need to look at my SD card (32GB but only one primary partition 4Gb).
17:52:45.303 -> ………….
17:52:51.306 -> WiFi connected
17:52:51.306 -> httpd_start
17:52:51.306 -> E (11077) fr_flash: Not found
17:52:51.306 -> Camera Ready! Use ‘http://192.168.9.31’ to connect
Thanks
David
a cup of java coffee is on the way, I wish I could buy 2 or 3 for you at once 🙂
hi robotzero
>>read_face_id_from_flash_with_name(&st_face_list);
is it the SD card
I have insert a SD card with 4GB partition (the card is 32GB, I create 1 partition with windows fat32)
I get this when start esp32
17:52:51.306 -> httpd_start
17:52:51.306 -> E (11077) fr_flash: Not found
why fr_flash? there is a 4GB SD in the slot
and when I click capture (after enter a name), nothing were saved as it show in your clip.
david
fr_flash isn’t the SD card. It’s a partition on the flash memory on the board. Maybe try with this tutorial first to set up a new partition type.. https://robotzero.one/esp32-face-door-entry/
thanks, i am now satisfied with your C-code on https://robotzero.one/esp32-camera-module , quite stable, fast
Hi robotzero,
I have been using your code to capture some faces and it has been working successfully.
I am developing a different application that uses the recognised faces. To add more faces, I recompiled your code, flashed it to the ESP and I’m getting an error when a browser connects to the ESP32. The error appears immediately after the browser connects. The error shows that the socket has been disconnected. Confusingly, using Safari as the browser on an iMAC, theWeb Inspector reports the error as “WebSocket connection to ‘ws://192.168.1.68:82/’ failed: Could not decode a text frame as UTF-8.” whereas using Firefox as the browser the equivalent development tool (Web Console) shows the error as “The connection to ws://192.168.1.68:82/ was interrupted while the page was loading.”
The serial monitor output is shown below:
21:05:51.758 -> [D][esp32-hal-psram.c:47] psramInit(): PSRAM enabled
21:05:51.827 ->
21:05:52.586 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 0 – WIFI_READY
21:05:52.586 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 2 – STA_START
21:05:52.796 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 4 – STA_CONNECTED
21:05:52.831 -> [D][WiFiGeneric.cpp:336] _eventCallback(): Event: 7 – STA_GOT_IP
21:05:52.831 -> [D][WiFiGeneric.cpp:379] _eventCallback(): STA IP: 192.168.1.68, MASK: 255.255.255.0, GW: 192.168.1.1
21:05:53.071 -> .
21:05:53.071 -> WiFi connected
21:05:53.108 -> httpd_start
21:05:53.108 -> Camera Ready! Use ‘http://192.168.1.68’ to connect
21:05:53.108 -> Code file: ESP-who-recognition-with-names
21:05:58.803 -> [D][WiFiClient.cpp:482] connected(): Disconnected: RES: 0, ERR: 128
I am using core 1.0.3-rc1 and the hardware works successfully with the example:
ESP->camera->webserver successfully so I believe the hardware is OK. To make sure that the file “camera_index.h” had not become corrupt, I redownloaded the file and repeated the compilation but I still get the same error(s).
I am using the most recent Arduino web sockets library (version 0.4.9) and since the error appears to be with the websockets library I downgraded to the original version you had used in your tutorial – version 0.4.0 but still get the same error(s).
Can you advise how I might track down the error please?
Hi,
I’ve seen errors like this during various project testing but usually after running for a while. I’m not sure if it’s a bug in the WiFiClient library, a bug in my code, trying to process to much data, or just the module overheating and the WiFi quitting. I’ve been waiting for them to release 1.0.3 final to see if it gets better.
This tools gives more information about what happened during a crash if the module crashes with an exception – https://github.com/me-no-dev/EspExceptionDecoder
Hi there, many thanks for your speedy reply. The error occurs when connecting a browser to the ESP and is consistent. The code doesn’t crash so the exception decoder doesn’t help. I have been inserting Serial.print’s to trace exactly where the error occurs. I have narrowed the search down and the error occurs in the function “index_handler” when sending the file “index_ov2640_html_gz”. I have done quite a lot of searching for errors similar to ones reported in the browser and they seem to point to a problem in the WiFiClient library: reference: https://github.com/espressif/arduino-esp32/issues/1921
I do not believe there is a bug in your code. I plan to continue to track the problem down and will update you if I find anything meaningful. I too await formal release of 1.0.3.
Many thanks for your response so far.
Hi,
I’ve seen the errors likes this “face_id_name_list was not declared in this scope”. I installed the library “ArduinoWebsockets”, the camera_index.h and camera_pins.h.
Please help me.
Thanks a lot.
Hi, Probably the wrong ESP32 Hardware Library installed. Which version do you have?
Dear Sir,
I’ve checked ur reply and i updated the ESP32 Board to the 1.0.3 version. The web loaded ok, but when i try to capture a face (from a picture), it didnt save that face after take 5 samples. Actually , it turns from 1, 2, 3, 4 and 7!
Thanks alot Sir.
Hi. Try 1.0.4 the face recognition was fixed in that version.
And how to know that when i used the 8GB sd card, how many people can I capture?
Capturing a person takes how many gb in the storage?
Thanks a lot Sir
Faces are captured on the board itself. It doesn’t use the SD card. One face is 2KB. This isn’t really meant for professional use. More for fun or possibly home security as an extra layer.
i got error every time i delete a face . when i reset/restart the esp The list does not look properly arranged .
the face is deleted from the flash and the list .
here were the code disconnected :
[E][WiFiClient.cpp:365] write(): fail on fd 63, errno: 104, “Connection reset by peer”
i still did not find a solution .
static esp_err_t send_face_list(WebsocketsClient &client)
{
Serial.println(st_face_list.count);
client.send(“delete_faces”); // tell browser to delete all faces
read_face_id_from_flash_with_name(&st_face_list);
face_id_node *head = st_face_list.head;
char add_face[64];
for (int i = 0; i id_name);
Serial.println(add_face);
client.send(add_face); // here the code crash
head = head->next;
}
}
You’ve replaced this line: sprintf(add_face, “listface:%s”, head->id_name);
with your
Serial.println(… line.
when i copy/paste the code it got deleted in the comment , but i have it on the code . i only add Serial.println( .. to trace the code .
I’ve just tested it again and it seems to work with the list when I reboot but… it does crash when I add a new face and then try to recognise it but not straight away…
It crashes around line 328 here:
Serial.println(f->id_name);I don’t know why though.
I made a video: https://vimeo.com/392791481
Must have been asleep.. sometimes there is no ‘f’ because a face is detected but not recognised! I’ve removed that line.
you can change this line : if (f) ……to if (f != NULL ) to solve the problem ” always finds a matching face even when a face hasn’t been captured “
Hello
Hi there
Thanks for the great work. I tried it and it works fine. However, I have a problem that the connected relay only switches if there is an http connection to the esp32-cam modul. If I close the browser, the connected relay no longer works. Can you explain to me where I have to change the program accordingly so that it runs permanently even without an http browser.
Hi. This version runs without a browser being connected – https://robotzero.one/esp32-face-door-entry/ Or.. you should just be able to remove or comment out the code for the WebSocket connection once you have saved your faces. Buy me a coffee and I’ll write the code for you.
My idea is this. The web module should be preserved in this way, but the detection and switching of the relay should also work if there is no active web session.
In addition, it would be great if you could send an http call in addition to switching the relay, which could also be used as an argument, e.g. contains the recognized name (e.g. for control via fhem).
The crowning glory would be if you still had a log (via web front end) in which you can see at what time a face was recognized and attached the picture (storage on an SD card).
Up to what punk is that a coffee?
Hmm.. I’ll take a look tomorrow at this.
It’s not the most elegant code but this version has a button that disconnects the interface and then the recognition runs without websockets – https://robotzero.one/wp-content/uploads/2020/03/FaceDoorEntryESP32CAMPremium.zip I’ll try to work out a way for it to do this on a timeout or disconnection.
Also this version has the http call you wanted. I’m looking at the log and picture on the SD card.
Hi there
Thank you for your efforts. I just tried it and it works as described.
1.) However, the module should start with the detection and not first be brought into this state with “Disconnect”. The module should start detection as soon as the power supply is established.
2.) Can not the “Disconnect” also take place as soon as the browser window is closed?
Would be great if that could be adjusted.
Greetings from Germany
Ralf
Hi, The problem with disconnections that no final packet is sent so the board doesn’t know it’s disconnected. I’m going to try and write around this using the http interface. I have saving the image to SD done now.
I’m thinking of starting a premium section on the site and have basic and premium versions of my projects to make a bit of money. This might be the first one if I can get it working nicely.
Thanks (from Spain)
How to add
static void rgb_print(dl_matrix3du_t *image_matrix, uint32_t color, const char * str){
fb_data_t fb;
fb.width = image_matrix->w;
fb.height = image_matrix->h;
fb.data = image_matrix->item;
fb.bytes_per_pixel = 3;
fb.format = FB_BGR888;
fb_gfx_print(&fb, (fb.width – (strlen(str) * 14)) / 2, 10, color, str);
}
static int rgb_printf(dl_matrix3du_t *image_matrix, uint32_t color, const char *format, …){
char loc_buf[64];
char * temp = loc_buf;
int len;
va_list arg;
va_list copy;
va_start(arg, format);
va_copy(copy, arg);
len = vsnprintf(loc_buf, sizeof(loc_buf), format, arg);
va_end(copy);
if(len >= sizeof(loc_buf)){
temp = (char*)malloc(len+1);
if(temp == NULL) {
return 0;
}
}
vsnprintf(temp, len+1, format, arg);
va_end(arg);
rgb_print(image_matrix, color, temp);
if(len > 64){
free(temp);
}
return len;
}
static void draw_face_boxes(dl_matrix3du_t *image_matrix, box_array_t *boxes, int face_id){
int x, y, w, h, i;
uint32_t color = FACE_COLOR_YELLOW;
if(face_id 0){
color = FACE_COLOR_GREEN;
}
fb_data_t fb;
fb.width = image_matrix->w;
fb.height = image_matrix->h;
fb.data = image_matrix->item;
fb.bytes_per_pixel = 3;
fb.format = FB_BGR888;
for (i = 0; i len; i++){
// rectangle box
x = (int)boxes->box[i].box_p[0];
y = (int)boxes->box[i].box_p[1];
w = (int)boxes->box[i].box_p[2] – x + 1;
h = (int)boxes->box[i].box_p[3] – y + 1;
fb_gfx_drawFastHLine(&fb, x, y, w, color);
fb_gfx_drawFastHLine(&fb, x, y+h-1, w, color);
fb_gfx_drawFastVLine(&fb, x, y, h, color);
fb_gfx_drawFastVLine(&fb, x+w-1, y, h, color);
#if 0
// landmark
int x0, y0, j;
for (j = 0; j landmark[i].landmark_p[j];
y0 = (int)boxes->landmark[i].landmark_p[j+1];
fb_gfx_fillRect(&fb, x0, y0, 3, 3, color);
}
#endif
}
This code into your code
You want to write onto the image frames so you see the green boxes etc in the stream? I didn’t use that code because it slows down the frame rate. It would be tricky to put it back but you can see how to do some of it in this sketch: https://github.com/robotzero1/face-follow-robot/blob/master/esp32-wifi-version.ino
Hello
Thank you for sharing the code with detailed instruction.
It worked well in the first time with new partition as you suggested.
However, after a couple of times, the video didn’t show anymore with that new partition.
There’s no error when compiling and uploading.
When I tried to reuse the existing partition “Huge APP”, there was video shown but the recognition function did not work since there is no “fr” in that partition.
Please can you advise how to resolve this issue.
Many thanks in advance.
Hi, Is there still a problem? The video shouldn’t just stop appearing. Do you see anything in the serial monitor when the video isn’t showing?
Hello
Still the same problem though I have tried to reset it many times. There is no warning/error shown on serial monitor apart from “Camera Ready! Use ‘http://192.168.1.27’ to connect”
Where in this example is the code http pages, CSS styles? Where is it formed, where does it come from? I was unable to find anything related to this.
Check this post for details: https://robotzero.one/esp32-cam-custom-html/
Thank you!
I downloaded the code from this example. It works great! Thanks for this!
Then I started to refine it and as a result it turned out that even if I download the “CameraWebServer”, I get a web page exactly as in this code, but there is no image and status in the green field, there is also no information about the saved faces. This happens if “FaceRecognition(2621440 bytes with OTA)” is enabled. If you use “CameraWebServer” then “Huge APP(3MB no OTA/1MB SPIFFS)” works fine.
I think there is some kind of problem with the new section “FaceRecognition(2621440 bytes with OTA)”, which manifests itself over time. The code for clearing the memory, which is given here: https://robotzero.one/esp32-face-door-entry/ , it doesn’t solve the problem, data for the example from this page is not deleted. Even if I use a different partition scheme, for example Huge APP(3MB no OTA/1MB SPIFFS), And then back to the scheme FaceRecognition(2621440 bytes with OTA) old data remains unaffected.
How to erase all data and start life from an empty disk?
It sounds like you’ve over-written the camera_index.h file for the CameraWebServer example with the code from the camera_index.h for this tutorial. In other words.. the camera_index.h file contains the HTML for current project. It doesn’t get saved to the ESP32 anywhere.
hi !
i ‘ve uploaded the sketch complete but i can not enroll the humun face . I don t know reason why. is there something wrong ?
=> I will buy the coffee for you if you help me to code completely .
i did detail as below :
i used board mananger => esp32 1.0.4 new updated .
rzo_partitions additional
#define CAMERA_MODEL_AI_THINKER
modified the boars.txt
tks
Hi, Do you see anything in the serial monitor?
i can see detail serial monitor as below :
22:06:00.034 -> dl_matrix3dqq_fc_with_bias, value > DL_QTP_MAX
22:06:00.034 -> dl_matrix3dqq_fc_with_bias, value dl_matrix3dqq_fc_with_bias, value > DL_QTP_MAX
=> i can see the video steaming but When a click the capture button => no face detected .
thank for your helps .
Does the normal CameraWebServer example work for you?
Camerawebserver example work without face recognition . Is there something wrong ? Plz help me to understand what happen ? May be missing something ?
So you can detect faces when the Face Detection button is on but when you press the enrol button it doesn’t start the enrolling sequence? You can see what should happen in this video: https://www.youtube.com/watch?v=gZgGniTLCiU&feature=youtu.be
Hi robotzero, I’ve an issue with this project. It works perfect but after some connections (I think between 20 and 30), the web page doesn’t display anymore the video and the users doesn’t show anymore, however, if I upload the CameraWebServer example from the Arduino IDE, the video shows perfectly, so it isn’t an issue of hardware. I think someone asked the same thing, but I couldn’t find the reason for this bug, I will buy you some coffees if you can help me 🙂
Hi, So after 20 to 30 ‘recognitions’ it stops streaming? So if I want to test this I should add myself and step in front of the camera 30 times? I guess you don’t have it connected to serial to see what happens on the ESP32? Maybe in the browser console (F12) there is something. I’m guessing the Websocket fails and doesn’t start up again. I’ll run a test here tomorrow.
I´ve tested a new ESP32 Cam, and it took like 10 iterations to show this symptom. I don’t know if it’s something related to the partition, or erasing users and adding new ones.
I’ve tried some things, and if I select the partition: “Huge APP” it does stream again in the page, but I can´t add faces and the one’s that were saved doesn’t appear.
When you say iterations, Do you mean adding 10 people to the system or recognising the same person 10 times broke it?
Adding, for example 3 users. And then eliminating 1 or 2 and adding other. Also connecting the camera, using it for like 5 minutes and turning it off. Reconnect again and using the recognition.
I tried what you said about the console in the web page, and always say:
HTTP error: status code 404, net ::ERR_UNKNOWN_URL_SCHEME
Is it the code pasted from the page above? Or the code on Github?
I will change the function and see what happens.
From this blog. I didn’t know there was a GitHub link.
Github has a version with a lock as seen here – https://robotzero.one/access-control-with-face-recognition/ but I’m testing with the code above at the moment.
Can you try changing the send_face_list function to this:
https://pastebin.com/AChDXrFY
and look in the serial monitor for the listface: items. After doing some testing I found one of the names saved on the ESP32 was corrupted. Not sure why atm.
Finally, it works!! Also I noticed that my custom PCB wasn´t giving enough current to the CAM, so when I started the web page it reset itself. Just that now with the corrected function the names that are shown in the interface are the same, also I can’t erase them individually,it just works if I click in the “DELETE ALL” button. Any advice?
This is just a temporary fix to try and work out what is causing the occasional corruption of the names. Next time you get a problem, it you connect it to serial you will see listface:xxxxx for the names and one of them will be junk characters. The code also needs something to check if the Wi-Fi and Websocket connections are active and reconnect (or just reset the board) if not.
I´ve been a little busy, but I I have bought you some coffees. Thanks a lot!
I’ve updated the tutorial to reflect the new way of uploading partitions in release 1.0.5. I’ve created a new partition scheme as well, maybe this won’t have the corruption issue.
Hello, I have the same problem as Ivan. The code has worked fine until today. Today the streaming is not working.
With the example code of streaming, my ESP32 is still working, so it’s not a hardware problem.
In the serial monitor appears this:
—
I’m using 1.0.5 version.
I think there is a problem with the usernames getting corrupted somehow on the flash memory. Try the change here: https://pastebin.com/AChDXrFY and see if it starts working again. It’s just a test to see if that’s the problem.
Now it works, although the listface only says anonymous. But the camera works, and when I click recognise it recognises well.
I think there’s a bug somewhere in the ESP library that causes it to corrupt the storage where the face data and name is stored.
hey thank you for your explaining , but i follow every think as you said but iam getting this error below :
C:\Users\Abdulrahman Ahmed\AppData\Local\Temp\arduino_build_865522\sketch\app_httpd.cpp: In function ‘esp_err_t index_handler(httpd_req_t*)’:
app_httpd.cpp:587:51: error: ‘index_ov3660_html_gz’ was not declared in this scope
return httpd_resp_send(req, (const char *)index_ov3660_html_gz, index_ov3660_html_gz_len);
^
app_httpd.cpp:587:73: error: ‘index_ov3660_html_gz_len’ was not declared in this scope
return httpd_resp_send(req, (const char *)index_ov3660_html_gz, index_ov3660_html_gz_len);
^
Multiple libraries were found for “WiFi.h”
Used: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi
Not used: C:\Program Files (x86)\Arduino\libraries\WiFi
Using library ArduinoWebsockets at version 0.5.0 in folder: C:\Users\Abdulrahman Ahmed\Documents\Arduino\libraries\ArduinoWebsockets
Using library WiFi at version 1.0 in folder: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi
Using library HTTPClient at version 1.2 in folder: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\HTTPClient
Using library WiFiClientSecure at version 1.0 in folder: C:\Users\Abdulrahman Ahmed\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFiClientSecure
exit status 1
‘index_ov3660_html_gz’ was not declared in this scope
Did you add the two other files to your sketch folder? “Add to the folder where the Sketch has been saved these two files: camera_index.h and camera_pins.h” from the tutorial.
hi WordBot
should these 2 files( camera_index.h and camera_pins.h ) be add to
sketch folder? i was unable to find this sketch folder,
where is this sketch folder? need some help, please
Hi, Should look something like this: https://pasteboard.co/K1vZrc2u.png
Hello , in your gitgub , you have two partitions files , which one to use , i get fr_flash error in serial monitor.
You say in GitHub we must use partitions.csv , but in video you use the other partitions file ? so which one is it?
I cannot stream the video in the web interface 🙁
Depends which version of the ESP32 hardware library you have. partions.csv works with versions after 1.0.4
Great project, I adjust it to my needs, added buttons for the doorbell and for manual open door. Takes a picture when doorbell pressed and stores it on the SD card. Every event (a button was pressed…) is stored on the SD card as log file for info or debugging. Using static AP address. Doorbell and manual opener works always, with or without connection to smartphone. Added buttons in WebSockets to take pics manually from smartphone, open door by smartphone button. Several Leds are indicating: door is open/closed, door lock activated/deaktivated. Working at the moment to activate a buzzer as warning indicator when door longer open than 10 min. On smartphone screen it shows the total space and used space of the SD card, here I have to fix the apperance. When the project is completed I will post it in youtube. The project is for my tiny house, having no internet connection yet. Keep going
Hi, I have a question can I find somwhere the library fd_forward?
Thanks
Hi, which version of the ESP32 Hardware Library are you using?
Hi thanks for answering so fast.
If you mean the esp32 board version, I have installed the 1.0.6
Can you try with 1.0.4 or 1.0.5. I don’t know if 1.0.6 works.
With 1.0.5 I have the same error and with 1.0.4 I get another error :
xtensa-esp32-elf-g++: error: unrecognized command line option ‘-mfix-esp32-psram-cache-strategy=memw’
exit status 1
Error compiling for board AI Thinker ESP32-CAM.
Are you using version 1.x.x of the Arduino IDE?
Yes I have the version : arduino-1.8.19-windows
Do you these settings in the IDE:
(although this doesn’t show the custom partition) ?
I just tried compiling and I saw a couple of different errors from I think a change in the compiler so I updated a couple of functions (new code in tutorial) and it compiles. I have it working in the browser but I didn’t change the partitions so it doesn’t save the face data.
You might find this one https://robotzero.one/access-control-with-face-recognition/ a bit easier because with 1.0.5 you don’t need to create the custom partition. The difference with this one is it controls a electronic door lock but you can leave that part out.
I wonder if you have a problem with your installation it’s still not working. Does the CameraWebServer example work?
Yes the CameraWebServer works, I will try the access control.
Thankyou
I’m having the same error about the fd_forward.h . My Arduino IDE can’t find this file. Is it an arduino library or one file you created?
I’ve seen that in the CameraWebServer example don’t use the libraries “fd_forward.h”, “fr_forward.h” and “fr_flash.h” that are the libraries that my IDE can’t find
Do you have this path on your computer?
C:\Users\xxxxx\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.4\tools\sdk\include\esp-face