ESP32-CAM Face Recognition for Home Automation

featured.jpg

Using face recognition to open a door or control other home automation devices

This tutorial will explain how to save enrolled images in the on-board flash so they survive the ESP32 powering off and use these saved recognitions to control devices connected to the ESP32. There are three steps.

  • Create a new partition scheme to enable persistent storage
  • Modify the CameraWebServer example sketch to save face data to the new partition
  • Use these saved recognitions to control devices connected to the ESP32

Before following this tutorial make sure your camera works by following this tutorial Ai-Thinker ESP32-CAM in the Arduino IDE

Persistent Storage Partition Scheme

A new partition scheme with persistent storage on the on-board flash is needed. You can download a scheme I created from here: https://robotzero.one/wp-content/uploads/2019/04/rzo_partitions.csv

Add this file to the directory containing the other partition schemes. This is found in one of two places, depending on how you installed the Arduino IDE.

Arduino IDE installed from the Windows Store:

C > Users > *your-user-name* > Documents > ArduinoData > packages > esp32 > hardware > esp32 > 1.0.1 > tools > partitions

Arduino IDE installed from the Arduino website:

C > Users > *your-user-name* > AppData > Local > Arduino15  > packages > esp32 > hardware > esp32 > 1.0.1 > tools > partitions

The new scheme has to be added to your ESP device in the boards manager configuration file – boards.txt. Again this is found in one of two places.

Arduino IDE installed from the Windows Store:

C > Users > *your-user-name* > Documents > ArduinoData > packages > esp32 > hardware > esp32 > 1.0.1

Arduino IDE installed from the Arduino website:

C > Users > *your-user-name* > AppData > Local > Arduino15  > packages > esp32 > hardware > esp32 > 1.0.1

Add the following three lines below the existing partitionScheme options for the esp32wrover board in this boards.txt file.

esp32wrover.menu.PartitionScheme.rzo_partition=Face Recognition (2621440 bytes with OTA)
esp32wrover.menu.PartitionScheme.rzo_partition.build.partitions=rzo_partitions
esp32wrover.menu.PartitionScheme.rzo_partition.upload.maximum_size=2621440

Close and reopen the IDE to confirm the new ‘Face Recognition’ partition scheme is available in the Tools menu.

There’s an article here that explains in much more detail how to set up a new scheme and duplicate a board definition: Partition Schemes in the Arduino IDE

Capture Face Data to Persistent Storage

The CameraWebServer example in the IDE doesn’t save enrolled faces in a way that will survive power loss. To modify it to use the new partition a few changes need to be made to the code.

In the Arduino IDE,  make a copy of your working CameraWebServer Sketch from the previous tutorial by saving it with a new file name such as CameraWebServerPermanent.

You should see three tabs in the Arduino IDE similar to the image below:

Arduino IDE Tabs

In the second tab (app_httpd.cpp) make the following changes.

After #include “fr_forward.h” (around line 24) add:

#include "fr_flash.h";

Change int8_t left_sample_face = enroll_face(&id_list, aligned_face);(&id_list, aligned_face);  (around line 178) to:

int8_t left_sample_face = enroll_face_id_to_flash(&id_list, aligned_face);

After face_id_init(&id_list, FACE_ID_SAVE_NUMBER, ENROLL_CONFIRM_TIMES); (around line 636) add:

read_face_id_from_flash(&id_list);

Flash and run this Sketch in the same way as before. Enrolled face data is now being saved to the new partition on the flash memory.

Face Recognition Trigger Event

I’ve written a Sketch that sets a pin HIGH on the board while a known face is detected. After 5 seconds pass, if there is no recognised face the pin is set LOW. The code in the function rzoCheckForFace() in the Sketch below can be changed to whatever function you require when a face is recognised. I prefer to set pins as low during setup because mentally I think of it as ‘off’ and then set them high when I want to trigger something because I think of it as ‘on’. You can use pin 12 instead or in addition to pin 2

Paste the following code into a new sketch

#include "esp_camera.h"
#include "fd_forward.h"
#include "fr_forward.h"
#include "fr_flash.h"

#define relayPin 2 // pin 12 can also be used
unsigned long currentMillis = 0;
unsigned long openedMillis = 0;
long interval = 5000;           // open lock for ... milliseconds

#define ENROLL_CONFIRM_TIMES 5
#define FACE_ID_SAVE_NUMBER 7

mtmn_config_t init_config()
{
  mtmn_config_t mtmn_config = {0};
  mtmn_config.min_face = 80;
  mtmn_config.pyramid = 0.7;
  mtmn_config.p_threshold.score = 0.6;
  mtmn_config.p_threshold.nms = 0.7;
  mtmn_config.r_threshold.score = 0.7;
  mtmn_config.r_threshold.nms = 0.7;
  mtmn_config.r_threshold.candidate_number = 4;
  mtmn_config.o_threshold.score = 0.7;
  mtmn_config.o_threshold.nms = 0.4;
  mtmn_config.o_threshold.candidate_number = 1;
  return mtmn_config;
}

static face_id_list id_list = {0};
dl_matrix3du_t *image_matrix =  NULL;
camera_fb_t * fb = NULL;
mtmn_config_t mtmn_config = init_config();
dl_matrix3du_t *aligned_face = dl_matrix3du_alloc(1, FACE_WIDTH, FACE_HEIGHT, 3);

void setup() {
  Serial.begin(115200);

  digitalWrite(relayPin, LOW);
  pinMode(relayPin, OUTPUT);

  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = 5;
  config.pin_d1 = 18;
  config.pin_d2 = 19;
  config.pin_d3 = 21;
  config.pin_d4 = 36;
  config.pin_d5 = 39;
  config.pin_d6 = 34;
  config.pin_d7 = 35;
  config.pin_xclk = 0;
  config.pin_pclk = 22;
  config.pin_vsync = 25;
  config.pin_href = 23;
  config.pin_sscb_sda = 26;
  config.pin_sscb_scl = 27;
  config.pin_pwdn = 32;
  config.pin_reset = -1;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  config.frame_size = FRAMESIZE_SVGA;
  config.jpeg_quality = 12;
  config.fb_count = 1;

  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }

  //drop down frame size for higher initial frame rate
  sensor_t * s = esp_camera_sensor_get();
  s->set_framesize(s, FRAMESIZE_QVGA);

  face_id_init(&id_list, FACE_ID_SAVE_NUMBER, ENROLL_CONFIRM_TIMES);
  read_face_id_from_flash(&id_list);// Read current face data from on-board flash
}

void rzoCheckForFace() {
  currentMillis = millis();
  if (run_face_recognition()) { // face recognition function has returned true
    Serial.println("Face recognised");
    digitalWrite(relayPin, HIGH); //close (energise) relay
    openedMillis = millis(); //time relay closed
  }
  if (currentMillis - interval > openedMillis){ // current time - face recognised time > 5 secs
    digitalWrite(relayPin, LOW); //open relay
  }
}

bool run_face_recognition() {
  bool faceRecognised = false; // default
  int64_t start_time = esp_timer_get_time();
  fb = esp_camera_fb_get();
  if (!fb) {
    Serial.println("Camera capture failed");
    return false;
  }

  int64_t fb_get_time = esp_timer_get_time();
  Serial.printf("Get one frame in %u ms.\n", (fb_get_time - start_time) / 1000); // this line can be commented out
  
  image_matrix = dl_matrix3du_alloc(1, fb->width, fb->height, 3);
  uint32_t res = fmt2rgb888(fb->buf, fb->len, fb->format, image_matrix->item);
  if (!res) {
    Serial.println("to rgb888 failed");
    dl_matrix3du_free(image_matrix);
  }

  esp_camera_fb_return(fb);

  box_array_t *net_boxes = face_detect(image_matrix, &mtmn_config);

  if (net_boxes) {
    if (align_face(net_boxes, image_matrix, aligned_face) == ESP_OK) {

      int matched_id = recognize_face(&id_list, aligned_face);
      if (matched_id >= 0) {
        Serial.printf("Match Face ID: %u\n", matched_id);
        faceRecognised = true; // function will now return true
      } else {
        Serial.println("No Match Found");
        matched_id = -1;
      }
    } else {
      Serial.println("Face Not Aligned");
    }

    free(net_boxes->box);
    free(net_boxes->landmark);
    free(net_boxes);
  }

  dl_matrix3du_free(image_matrix);
  return faceRecognised;
}

void loop() {
  rzoCheckForFace();
}

When you flash and run this new Sketch you should see ‘Face recognised’  in the serial monitor when a matched face is found.

Opening a Door

The Sketch above combined with a relay or Mosfet module can be used to switch an electrical device on or off. This can be used to open or unlock a door

The diagram below shows the wiring for a opening a lock. From the Sketch above, when a face is recognised, pin IO2 is set HIGH and the relay closes so current flows from the high voltage power supply to the electric door lock.

ESP32-CAM Door Lock Relay

While setting up and testing a project like this you might prefer to have the serial device connected as below

ESP32-CAM Door Lock Relay with USB

The door lock in this example could be anything you want to temporarily supply power to.

The project will work with many relay and Mosfet modules that have a 3v input such as the items below. The opto isolated Mosfet module (green connectors) needed the resistor bypassed. The smaller red one works with just the signal and ground connected.

These are available from eBay: Basic Mosfet | Opto Isolated Mosfet | Opto isolated 3v Relay

There’s lots of options for controlling a high voltage device from the ESP32 and it is important to understand the dangers to yourself and the ESP32 if you use the wrong solution.  The mechanical relay board is rated for higher voltages but I personally wouldn’t use mains electricity with a device like this.

I have some more tutorials like this coming. Feel free to buy me a coffee to help development 😉 .

References

Event trigger for IDF: https://github.com/espressif/esp-who/blob/master/examples/single_chip/recognition_with_command_line/main/app_facenet.c
In-depth relay information: https://www.youtube.com/watch?v=d9evR-K6FAY

2 Replies to “ESP32-CAM Face Recognition for Home Automation”

  1. Mel says:

    You’ve used IO2 or 12 for the relay control. Aren’t these pins also used for SD card? I’m looking for any module pins that are “free”, seem to be scarce commodity; conflicts with flash LED, SD etc.; idea would be to use an external PIR to save jpeg to SD card, or means to display image remotely. Also would this module work with the OV2640 fisheye camera?

    1. WordBot says:

      Yeah.. All the pins are assigned to something. Those two seem to be useable for other things but probably not at the same time as using the SD reader. I’ve got a tutorial in progress for writing to the micro-SD or uploading. Not sure about the fisheye.. there’s a few of the little OV2640 cameras and I’m not sure the cables are the same.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

scroll to top