Face Tracking Pan and Tilt with an ESP32-CAM

ESP32 Face Tracking Pan Tilt Platform

Using the ESP-WHO library and a pan and tilt platform to track a moving face.

With the Espressif ESP-FACE library it’s easy to detect a face and find its location in the frame. The library provides a function called draw_face_boxes that is normally used to display a box around a detected face.

Face Detected

The X and Y co-ordinates of this box combined with its height and width can be used find the centre of the box and therefore the centre of the face.

For example, if X is at 105px, Y is at 90px, and the box has a width of 50px and height of 70px then the centre can be found by adding half the width or height of the box to the X or Y values like this: x+w/2, y+h/2 so for the figures above, 105+50/2 and 90+70/2 would give the face centre as x:130 and y:125.

Face Tracking Finding Co-ordinates

One of the tricky parts of using a pan and tilt platform to track a face is converting the distance of the face from the centre in pixels to the degrees the platform needs to move. I’ve chosen the simplest method using a basic conversion from pixels to degrees.

One guide I found recommended using the diagonal measurement of the sensor as below:
For QVGA (320×240)
sqrt(sq(320) + sq(240)) = 400
and then dividing the field of view (for my camera 45 degrees) by this to get the pixels per degree of rotation:
400/45 = 8.89

So for every ~9 pixels of movement in the frame, the servo moves 1 degree in that direction.

Face Tracking Calculating Movement

However, with the platforms I’ve used, the degrees of movement of the servos don’t coincide with the change in degrees of the view area because either the pan or tilt is offset from the centre of rotation.

Pan and Tilt Platform
Pan and Tilt Platform
ESP32-CAM Mount
ESP32-CAM Mount

My original plan was to get the reading and move the platform straight to that location but often it would overshoot (possibly a problem with the off-centred sensor or maybe just my maths) and start oscillating back and forth. So I changed the code to only move half the registered distance each time until it reached the new location. I experimented with looping this movement until completed and then return to detecting, but I went for continuous detection and calculation in the end.

I’ve seen other tutorials where the servos are moved in the direction of the face until the face is in the centre of the frame which is another approach. I think this might only work well when the frame rate is higher. The face detection runs about a 3 frames per second.

Another thing I’ve noticed is that variations in the detected face location mean the pan and tilt platform wanders a little when the face is centred. Some code could be added that so the servos are only activated if the face is outside of the centre area.

Face Tracking Video Demonstration

Wiring Diagram

The wiring is the same as the basic pan and tilt tutorial here. I use a USB power bank for the 5v source with one of these USB cable connectors to make it easy to connect and disconnect the power.

ESP32-CAM Pan and Tilt Wiring Diagram

The Code

If you’ve not used the ESP32-CAM before you will need to read through this tutorial first – https://robotzero.one/esp32-cam-arduino-ide/ to get familiar with it.

You also need to install the ArduinoWebsockets library by searching in Tools > Manage Libraries:

Library Manager ArduinoWebsockets

Copy and paste the Sketch below and save it. Copy these two files: camera_index.h and camera_pins.h to the same directory. You should be able to compile and run the same way as other ESP32-CAM projects. This project works with version 1.0.4 of the ESP32 hardware libraries for the Arduino IDE.

I’ve also created a version with the green box around the face. The frame rate is less on this version because it takes time to combine the box with the frame and convert to jpg. You can download it from pastebin here: https://pastebin.com/ECQPxuec

If anyone has suggestions for improving the maths or how to calculate degrees of movement when the sensor pan or tilt movement is off the axis centre please let me know via the comments or contact form.


3D printable pan tilt mount: https://www.thingiverse.com/thing:3579507
Pan and tilt location calculation (complicated): https://stackoverflow.com/questions/44253787/translating-screen-coordinates-x-y-to-camera-pan-and-tilt-angles
Pan and tilt location calculation (simple – the one I used): https://stackoverflow.com/questions/17499409/opencv-calculate-angle-between-camera-and-pixel
The reason simple isn’t accurate: https://www.quora.com/How-can-I-find-the-pixels-per-degree-if-I-know-the-resolution-and-angle-of-view-for-a-pi-cam

80 Replies to “Face Tracking Pan and Tilt with an ESP32-CAM”

  1. Anonymous says:

    The code seems to have some Copy paste error. I mean some parts of the code repeats itself over and over again. Please check the uploaded code and make corrections in the website.

    1. WordBot says:

      Hi, Dunno what happened there but I fixed it.

  2. ilas says:

    Hello, everything works except the movement of servos, specifically the signal to the servos is 50% constant even in the presence of the face is not centered in all conditions. If I try to force the servo I feel that tends to remain in the same position. Where am I doing wrong?

    1. WordBot says:

      Hi. If you change this code
      ledcAnalogWrite(2, 90); // channel, 0-180
      to ledcAnalogWrite(2, 180); // channel, 0-180
      does the servo move to 180 position?

      You can also try some Serial.println(); statements to check the face movement is being detected.
      pan_center = (pan_center + move_to_x) / 2;
      for example

  3. ilas says:

    hello, thanks for the reply.
    Yes, if I change to ledcAnalogWrite (2, 180); // channel, 0-180, the servo moves at about 180 degrees
    If I include Serial.println (pan_center); immediately after the while, on the serial I always read “90” even, if I move my face at any point …
    I’ve just finished the box too …

    1. WordBot says:

      Hi, Does the CameraWebServer example work for you? File > Examples > ESP32 > Camera > CameraWebServer

  4. ilas says:

    Hi, yes,work correctly also CameraWebServer with face recognition and detection.
    I have CAMERA_MODEL_AI_THINKER module.
    The 5V is stable.
    The servo is good, It seems to always detect the face in the middle when I load your program because when it includes Serial.println (pan_center); I always read the value 90 on serial.
    I’m trying to figure out where I’m wrong ..

    1. WordBot says:

      Hi, I’ve found the problem. The draw_face_boxes() function isn’t being called because (I’m guessing) the face_detect() function has changed in the newer versions of the Espressif ESP32 Arduino hardware libraries. Face detection was broken in 1.0.2 so I used 1.0.1 for this script. It works if you choose 1.0.1 in the Boards Manager. I’ll update the script for 1.0.4 at some point.

      EDIT: now works with 1.0.4

  5. ilas says:

    Hi, I installed the 1.0.1 version but the problem persists.
    I’m thinking of everything, which browser do you use?
    The image via web I see only with Firefox while with the old Explorer and Explorer Edge does not pass streaming .. I have not tried with Chrome.
    I can confirm however that with the 1.0.4 version the “CameraWebServer” program works perfectly (my face appears with a yellow outline once it is hooked).
    During the ignition, the servants make a gesture of movement (first one and the other) to return to the central position.
    Thanks for your interest …

    1. WordBot says:

      Hi, I tried in IE11 and it doesn’t work. Probably doesn’t support WebSockets. Edge works for me. Try adding this code to see if you see anything in the serial monitor:

      pan_center = (pan_center + move_to_x) / 2;

  6. ilas says:

    Hi, check if the coffee has arrived 😉
    I wrote the Serial.println (pan_center); inside the loop and actually on the serial monitor I always read “90” (it’s a test I had already done before but with version 1.0.4 now have installed the 1.0.1).
    I would like to be able to solve this problem for Halloween as a treat for my children by putting a small paper ghost in front of the camera for face autotracking …
    I understand that it is hard work to compile for version 1.0.4 ..
    Do you have any other ideas?
    Thanks in advance.

    1. WordBot says:

      Thanks for the coffee! Try putting the println code in the draw_face_boxes() function to see if it’s being called by the loop.

  7. ilas says:

    Hi WordBot, yes, actually the draw_face_boxes () function is not called, or rather it is called but only twice at an interval of about 2 minutes, but then, it is no longer called.
    The video streaming on web it’s ever regular ( about 3fps ).

    1. WordBot says:

      I wonder if you need more light or the face is too close or far away. The draw_face_boxes() function is called when a face is detected.

  8. ilas says:

    it’s something that I thought too, I tried in all light and distance conditions, I don’t know if it’s the same detection method, but I noticed that with the “Camera Web server” example the yellow face detection panel works very well even in low light conditions, with a very close face (about 30 cm) and a face with a maximum distance of about 1.5 m work correctly.
    No fear, sometimes I’ll check on your site if you’ve updated this section.
    Thanks anyway..

  9. ehelicon says:

    Nice initiative! face tracking. but it’s failed in my case, May be due I am using Boards Manager 1.04. Hope the support for 1.04 will release soon. Thanks!

    1. WordBot says:

      Do you see any errors in the serial monitor?

      EDIT: now works with 1.0.4

  10. ehelicon says:

    no any messages in serial monitor, except the http url after boot. Thanks!

  11. romius says:

    Hey. I tried to download the code, http works, but it doesn’t show the picture. I tried to connect the servo, leads to position 90 and far no reaction. It does not transmit errors to the port; the servo value is always 90.

    1. WordBot says:

      Hi, Do you know what version of the ESp32 hardware libraries you have?

  12. romius says:

    I tried on 1.01 and 1.0.4. the camera web server works without problems. I understand that the stream module is removed, and whether the image should be hit on http

  13. romius says:

    Hey. It turned out I had to make the servo move. I took an example of a web server camera and integrated your code and the frame drawing algorithm there. But for now, you need to turn on the recognition mechanism with your hands. Here is a link to the beta version of the code. https://drive.google.com/open?id=1B6oMwtvWSeZTu2Z6gg5-p63AIX_sC_f1
    But with your code, something is not right.
    In general, many thanks.

  14. WordBot says:

    The code now works with 1.0.4

  15. Quarantined says:

    When I try to use this I get the following error:

    WiFi connected
    Camera Ready! Use ‘’ to connect
    matrix3du item alloc failed.
    Guru Meditation Error: Core 1 panic’ed (LoadProhibited). Exception was unhandled.

    and the ESP32 cam reboots.

    Any suggestions?

    1. WordBot says:

      I’m not sure. Does the CameraWebServer example work for you?

  16. Quarantined says:

    I did not have the PSRam enabled. Shrug.

    Cool tutorial.


  17. fabqu says:

    I get some strange behaviour with mtmn_config_t.
    It “has no member pyramid_times” and “type”.
    ‘box_array_t {aka struct tag_box_list}’ has no member named ‘score’

    I made these lines comments and it worked. But servos are not moving…

    May you help?

    1. WordBot says:

      Hi, Do you know what version of the ESP32 Hardware library you have installed?

  18. I am trying to do face tracting pan-tilt with esp32-cam for my robotic head project. I saw that you have already done that successfully. Many thanks for your effort and sharing but I could not find the code (sketch) here. Am I missing something?..

    1. WordBot says:

      Code is in the page but maybe you have a script blocker in your browser and it’s blocking the source at pastebin? I need to move this to github at some point.

  19. Dincer Hepguler says:

    Thank you for your fast response… I only see the pastebin link for the face tracking sketch for the green-box version. I know that some browsers block pastebin but I can open it by adding a “p” such as pastebinp to the link… I already got that one but still can not see the actual sketch… 🙁

  20. Dincer Hepguler says:

    https://pastebinp.com/ECQPxuec is the link for the green-box version, am I right?… I do not see any other link for actual sketch…

    1. WordBot says:

      Hi – here’s the other sketch – https://pastebin.com/cLY0MAYx it’s embedded in the page above, I don’t know why it doesn’t show in your browser.

  21. Dincer Hepguler says:

    Thank you for the new link… I just got it now… Your green-box version also works but it is so slow that sometimes loses the face tracking and sometimes overshoots… I will now try the original version of the sketch… regards…

  22. leslie eldridge says:

    Hi. I am new to this and completely lost. on compile at this line…
    return httpd_resp_send(req, (const char *)index_ov2640_html_gz, index_ov2640_html_gz_len);
    I get this error:
    ‘index_ov2640_html_gz’ was not declared in this scope

    1. WordBot says:

      Hi. You missed this step…
      Copy these two files: camera_index.h and camera_pins.h to the same directory

  23. Justin Peng says:

    Hi, I am having the same problem as Leslie Eldridge too. I get this error:
    ‘index_ov2640_html_gz’ was not declared in this scope. I have copied the files to the same directory as you said, but the same error occured.

    1. WordBot says:

      Hi, It should work. camera_index.h contains the definition of index_ov2640_html_gz and the file is included here: #include “camera_index.h”

  24. Justin Peng says:

    Hi, I’ve tried copying and pasting the files again. I went to the CameraWebserver Example but it worked fine even though it had the same exact code. Any ideas?

    1. WordBot says:

      You could try copying the content of the camera_index.h file into the Arduino Sketch below the includes.

  25. Justin Peng says:

    Oh, It worked! Thanks, but they’re still a problem. When it detected my face, serial would send me: dl_matrix3dqq_fc_with_bias, value > DL_QTP_MAX
    dl_matrix3dqq_fc_with_bias, value < DL_QTP_MIN
    I believe it is reversed(?). If so, how do I fix this

    1. WordBot says:

      I sometimes see this message when I’m working on a project. I don’t know what it means but I think it’s when it hasn’t got a clear face capture in a frame.

  26. Mario C says:

    Hi there, sorry for reviving up an old thread, but I keep getting an error when I compile the sketch.
    /Users/mm/Documents/Arduino/libraries/ArduinoWebsockets/src/tiny_websockets/internals/ws_common.hpp:4:10: fatal error: string: No such file or directory
    Using library ArduinoWebsockets at version 0.5.0 in folder: /Users/mariocortes/Documents/Arduino/libraries/ArduinoWebsockets
    exit status 1
    Error compiling for board Arduino Uno.

    Im a def new to this and after browsing other forums I couldn’t resolve my issue and i’m hoping I can get some guidance. Thanks in advance

    1. WordBot says:

      Hi, It looks like you have the wrong board selected. ‘Error compiling for board Arduino Uno.’.
      Have you installed the ESP32 hardware libraries?

  27. Miles H Lewis says:

    Hi everything works fine but I can’t detect faces. even on the cameraWebServer example the camera works fine except for for the face detection. any ideas? thanks

    1. WordBot says:

      Hi, Do you have good lighting? Try moving closer and further from the screen. Here’s another video demo: https://www.youtube.com/watch?v=gZgGniTLCiU&ab_channel=robotzero.one

  28. Norton says:

    Hi, the wifi seems to be rebooting every time movement is detected by the camera. Would you happen to know why this is happening? Thanks.

    1. WordBot says:

      Hi, Does it do it if you disconnect the pins to the servos? It could be a power issue.

      1. Norton says:

        Yes, it does it whether servos connected or not.
        Code uploads fine, servos move to default position, cam connects to wi-fi, displays the live video in Chrome, I focus on myself, it’s fine, I move, it reboots.

        1. WordBot says:

          Is it the version with the green boxes or without? Do you know which version of the ESP32 hardware libraries you have?

          You could try commenting out the two ledcAnalogWrite lines at the end of the draw_face_boxes function to see if it’s that function that’s crashing.

          1. Norton says:

            Tested the esp32 example webserver code as well and tried your code using the green boxes and without.
            In all cases, the output from the serial monitor in Arduino IDE shows an error but still sets up the webserver and displays the IP. The error is: flash read err, 1000 ets_main.c 371

            1. WordBot says:

              I don’t know what that error is. I found this https://github.com/espressif/esp-idf/issues/113 but they are mostly talking about the IDF rather than Arduino. If it works with other Sketches but still crashes with mine it’s just a case of commenting out lines until you find which one crashes it. You could try different ESP32 hardware libraries in the Arduino IDE.

      2. norton says:

        Yes. After I upload the code without servos attached, after I press the reset button on the esp32cam, the Arduino IDE serial monitor shows the IP address and then I try to face track the camera, the image freezes on the web page and the serial monitor acts as though I pressed the reset button on the esp32cam.

      3. norton says:


  29. Erik says:


    what I don’t understand is, that in your design, the ESP32-cam is laid down to the right, so turned 90 degrees.
    If you lay it down like that,the face recognition does not work, and yet in your code i can not find anything to turn the cam, even in the web-server page you can not turn the cam. If you make a stand with the cam turned up right, the image can not work due to the screen resolution doesn’t correspond anymore.
    Can you elaborate on that ?
    Or did you make a different sketch?

    1. WordBot says:

      Hi, Some of the ESP32s have an OV2640 camera module at 90 degrees so you would have to change the mount to match. It’s a PITA because some of my tutorials have ESP32s with cameras at 90 degrees and others not.

  30. Arman Ahmad says:

    Hello sir, I didn’t understand the “copy camera_index.h and camera_pins.h” part, I will be very happy if you can elaborate that part. Thanks in advance Sir

    1. WordBot says:

      There’s two files linked that you need to download and copy to the sketch folder.

  31. Jen says:

    Hi sir, can you tell what is the variable of x and y? how to get that variable? and what kind of algorithm used on this? Thank you sir. I don’t get an answer I try to look for it but your explanation only makes sense. Please help me, sir

    1. WordBot says:

      Hi x and y are from the box array that the face detection library creates. For example:
      x = ((int)boxes->box[i].box_p[0])

      1. Jenn says:

        Okay sir, thank you. Do you know what kind of algorithm use in face recognition?

        1. WordBot says:

          I don’t remember now which algorithm. They had some documentation on Github for this but they’ve changed the system (at least for the IDF) and I think the docs have gone. New docs (but new system) https://github.com/espressif/esp-dl

          Maybe in a comment on here I’ve mentioned the algorithm but I can’t find it.

  32. dan says:

    The code and all are good for me. But im having a problem where my esp32 need to be vertically placed in order to have my face upright and the servo will keep tilting down when im moving to the left. As your setup the esp32 shown where horizantally placed. Btw im using AI Thinker ESP32 Cam for the board. Thank you again.

    1. WordBot says:

      Hi, Yeah this is because the manufacturers just use any random camera they can get and some of them are at 90 degrees. I don’t think you can easily change the angle in software but maybe something could be changed in this code. I don’t really have time to look at the moment.

      float move_to_x = pan_center + ((-160 + face_center_pan) / 8.89) ;
      float move_to_y = tilt_center + ((-120 + face_center_tilt) / 8.89) ;

      pan_center = (pan_center + move_to_x) / 2;
      ledcAnalogWrite(2, pan_center); // channel, 0-180

      tilt_center = (tilt_center + move_to_y) / 2;
      int reversed_tilt_center = map(tilt_center, 0, 180, 180, 0);
      ledcAnalogWrite(4, reversed_tilt_center); // channel, 0-180

      1. dan says:

        Thank you for the reply. I will try to change this code. Thank you again for the amazing project.

  33. Baczkun says:

    Hi sir, I need help, I’m new to this, I cannot get it to work, I always get this error messages when compiling the code:

    sketch\app_httpd.cpp.o:(.bss.camera_httpd+0x0): multiple definition of `camera_httpd’
    sketch\PanTilt.ino.cpp.o:(.bss.camera_httpd+0x0): first defined here
    collect2.exe: error: ld returned 1 exit status
    exit status 1
    Error compiling for board ESP32 Wrover Module.

    I used esp32 library 1.0.4, ArduinoWebSockets 0.4.5

    1. WordBot says:

      Hi, Sounds like your sketch has two definitions for camera_httpd. Look at the code for app_httpd and PanTilt.ino

  34. Francisco says:

    I need help:
    I us Windows and I have ArduinoWebsockets V0.5.3 is installed

    I have download and compile the ino file, but I get an error message that certain fd_forward.h file are missing.
    I have installed manuel fd_forward.h. Now I have this errors:

    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino: In function ‘mtmn_config_t app_mtmn_config()’:
    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino:33:15: error: ‘struct mtmn_config_t’ has no member named ‘type’
    mtmn_config.type = FAST;
    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino:33:22: error: ‘FAST’ was not declared in this scope
    mtmn_config.type = FAST;
    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino:36:15: error: ‘struct mtmn_config_t’ has no member named ‘pyramid_times’
    mtmn_config.pyramid_times = 4;
    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino: In function ‘void loop()’:
    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino:212:55: error: ‘face_detect’ was not declared in this scope
    net_boxes = face_detect(image_matrix, &mtmn_config);
    C:\Users\49176\Documents\Arduino\sketch_aug29a\sketch_aug29a.ino:216:23: error: ‘box_array_t {aka struct tag_box_list}’ has no member named ‘score’

    What can I do?

    1. WordBot says:

      Which version of the ESP32 hardware library do you have installed? Most of my tutorials only work with 1.0.4 and 1.0.5

      1. Francisco says:

        You mean esp32 by Espressif Systems? This is on 2.0.4

        1. WordBot says:

          Yep. This project probably won’t work because they changed a lot of stuff with the face detection/recognition.

          1. Francisco says:

            Ok I have installed 1.0.5 and this ist better, only this errror:

            fatal error: dl_lib.h: No such file or directory

            Who can I found dl_lib.h?

            1. WordBot says:

              Can you try with 1.0.4? I thought everything worked with 1.0.5 but maybe this one has some code that doesn’t work with that version.

              1. Francisco says:

                unfortunately does not work either.
                – Board: ESP32 Dev Module
                – esp32 von Espressif Systems 1.0.5
                – ArduinoWebsockets 0.4.5
                fatal error: stdbool.h: No such file or directory

                I copied the file manuel to the folder below link:

                But does not work, always get the same error message

  35. Francisco says:

    I tried everything I can, this is now the current error message. Can anyone do something with that?

    c:/users/49176/appdata/local/arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/gcc8_4_0-esp-2021r2-patch3/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:\Users\49176\AppData\Local\Temp\arduino-sketch-5FC2C82E00CBA406C619255403BE83DF\sketch\sketch_aug29a.ino.cpp.o:(.literal._Z4loopv+0x20): undefined reference to `face_detect’
    c:/users/49176/appdata/local/arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/gcc8_4_0-esp-2021r2-patch3/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:\Users\49176\AppData\Local\Temp\arduino-sketch-5FC2C82E00CBA406C619255403BE83DF\sketch\sketch_aug29a.ino.cpp.o: in function `loop()’:
    C:\Users\49176\Documents\Arduino\sketch_aug29a/sketch_aug29a.ino:212: undefined reference to `face_detect’
    collect2.exe: error: ld returned 1 exit status

    exit status 1

    Compilation error: exit status 1

    1. WordBot says:

      Did you try with 1.0.4 . It’s odd it can’t find the face_detect function.

      1. Francisco says:

        I have now uninstalled and reinstalled everything.
        – Board: AI Thinker
        – esp32 by Espressif Systems 1.0.4
        – ArduinoWebsockets 0.4.5

        and I was finally able to compiling 🙂
        Now I have the problem, as soon as a face is discovered, the camera image is frozen. (I haven’t plugged the servos, only look the stream on PC)

        1. WordBot says:

          Do you see anything in the serial monitor?

  36. Meeeem1337 says:

    hello, I have such a strange problem, the board restarts when I point the camera at my face, I do not know why the board behaves like this, can you please help?

    1. WordBot says:

      Hi, Sounds like a power issue. The board is running a process when it does the face detection and this uses more power than the board can supply and it crashes. It might be that it detects the face, tries to move the motors and then crashes.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

scroll to top