An auto-balancing robot that can track your face and follow you around a room
Elegoo kindly sent me a Tumbller self-balancing robot for my project. It’s easy to put together and comes with clear instructions. Fully assembled it looks like this.
The kit comes with all the parts you need to build a remote control or autonomous self-balancing robot.
The robot has a custom motherboard with a socket for the Arduino Nano and a Bluetooth module soldered on.
The robot uses various sensors to keep balance and measure its environment.
I wanted to add face tracking to the robot so it could follow a person around the house. I added an ESP32-CAM camera module, mounted with 3d printed parts and connected via cables to serial on the Arduino Nano on the Robot.
On the ESP32 I use the ESP-FACE libraries to first detect a face and then measure the distance and location of the face in the frame.
This data is then sent to the Arduino Nano which processes this data and controls the robots direction. I have previous face detection projects for the ESP32-CAM and I used the https://robotzero.one/face-tracking-esp32-cam/ project as the basis.
The height of a detected face is measured using the ESP32 face detection library and this is used to calculate the approximate distance to the face using the following formula:
int eq_top = 3.6 * 200 * 240; // f(mm) x real height(mm) x image height(px) int eq_bottom = smoothed_face_height * 2.7; //object height(px) x sensor height(mm) int face_distance = eq_top / eq_bottom;
eq_top is the focal length (f) of the ov2640 I’m using (3.6) multiplied by the real height (estimate) of a face (20mm) multiplied by the image height (240px) of the camera frame.
eq_bottom is the smoothed (averaged) detected face height – calculated by averaging the last 5 face height readings from the camera – multiplied by the physical size of the sensor (2.7mm)
eq_top is then divided by eq_bottom to get the distance of the person’s face from the camera. In testing over the serial monitor this worked quite well to get the distance
The data captured by the ESP32 is sent using a second serial connection on pins 2 and 14 to the Arduino Nano on the robot. The code for this looks like this:
Serial2.print('<'); // start marker Serial2.print(face_center_pan); Serial2.print(','); // comma separator Serial2.print(face_center_tilt); Serial2.print(','); // comma separator Serial2.print(face_distance); Serial2.println('>'); // end marker
face_center_pan is calculated the same way as the https://robotzero.one/face-tracking-esp32-cam/ tutorial. face_center_tilt isn’t used for now in the project.
In the code repository on Github there are two Sketches for the ESP32. The esp32-wifi-version.ino sketch has extra code which enables viewing of the camera feed in a browser, including the green box around the face. This extra code reduces the frame rate and therefore the fluidity of the face capture. The robot is more responsive when the esp32-fast-version.ino sketch is used.
Arduino Nano Code
On the Arduino Nano the serial data is captured as a character array by the recvWithStartEndMarker() function. The received characters are then converted into variables and mapped to values by this code:
int result = sscanf(receivedChars, "%i,%i,%i", &pan, &tilt, &distance); // face location on camera, mapped to turn new_setting_turn_speed = map(pan, 0, 320, 40, -40); // face distance, mapped to speed new_setting_car_speed = map(distance, 10, 1200, -20, 20);
The further the face is from the centre of the frame, the faster the robot will turn and the greater the distance measured from the face, the faster the robot will move.
One problem with the robot is the distance from the face to the camera also includes a vertical distance. Increasing the height of the camera makes the robot less stable and therefore less able to capture faces.
The face recognition library and camera work really well but only in a limited distance range. It’s possible to increase the distance by reducing the minimum detected face size in the settings but this makes the detection slower and can make the robot less responsive.
This project can also be found on YouTube here: https://www.youtube.com/watch?v=cdpqpqMnBwI
If you found something useful above please say thanks by buying me a coffee here...
Calculating Distance – https://photo.stackexchange.com/questions/12434/how-do-i-calculate-the-distance-of-an-object-in
Arduino Smoothing – https://www.arduino.cc/en/tutorial/smoothing
Comma Separated Variables Over Serial – https://forum.arduino.cc/index.php?topic=541887.0
Arduino Easing Library (potential future motor control improvement) – https://github.com/luisllamasbinaburo/Arduino-Easing