Robot with an ESP32, Movement Sensor and three.js

Toy-Robot-ESP32-MPU9250-Threejs.png

Adding a sensor backpack to a toy robot and using it to control a 3D model displayed in a browser.

This project uses a 3D printed ‘backpack’ containing an ESP32, a 9 axis movement sensor, battery and touch switch attached to a toy robot.

ESP32 MPU9250 Robot Backpack

The ESP32 reads data from the movement sensor as the robot is rotated and tilted. This data is received by a web page served by the ESP32 and using the three.js JavaScript library a previously captured 3D model of the robot follows the movement.

Oh… and there’s 3D generated lightning!

Robot and Lightning

For this project I used an Iron Man toy from eBay, an ESP32 board with microSD card reader, an MPU9250 9-axis board and a TTP223 capacitive touch button from AliExpress. Some cables and a spare microSD card. You also need a little 500mAh 43x25x8.5 mm LiPo if you want to power it from a battery. You can either print the robot backpack using a 3D printer or just use thick cardboard and some elastic bands.

You might prefer to use your own action toy or other item as the basis for the 3D model. To do this you need to turn your physical object into a 3D file to be displayed in a browser.

Creating a 3D Model from Photos

Creating the 3D model from photographs, often called photogrammetry involves taking many photos of your object from many different angles and using software to recreate the object in 3D.

There are many ways of doing this and for my project I used a combination of Regard3D, Meshmixer, Creators3D online viewer and Microsoft’s 3D Builder. The workflow is a little convoluted but worked for my application.

To start I took about 50 images of the robot from three different heights.

Robot Images for 3D Reconstruction

In Regard3D I used the following settings to create the 3D data:

Matches
Detector(s): AKAZE Threshold: 0.0001 Dist ratio: 0.9 Camera model: Pinhole radial 3 Matching algorithm: FLANN

Triangulation
New Incremental, MaxPair initilization, intrinsic camera parameters refined

Densification
CMVS/PMVS UseVis: no Level: 1 Cell size: 2 Threshold: 0.7 wsize: 7 Min image num: 3 Max cluster size: 100

Surface
Type: Posisson Parameters: Depth: 9 Samples per Node: 8.6 Point weight: 4 Trim threshold: 5 Colorization: Textures Color Params: GeomVisTest: yes Global seam level: yes Local seam level: yes Outlier removal: None

Regard 3D Screen Capture

The result isn’t perfect but usable to start. There are many tutorials for Regard3D and possibly other settings will work better for your case.

At this point I was exporting the surface as a obj file to clean up in MeshLab but I couldn’t find any way of re-exporting it from Meshlab which included the textures.

I turned to Meshmixer and used Select > Unwrap Brush to clean all the extraneous material you can see in the Regard 3D screen above. I saved the cleaned up file in 3mf format because this format contains all the 3D data and textures in one file.

At this point your 3D model is probably the wrong way up and too small to be used in the second part of the project. I found Microsoft’s 3D Builder worked really well for re-orientating and enlarging the model using a combination of the rotation and movement controls, especially with the Object > Settle command to make the model sit flat.

3D Builder Move and Rotate Controls

Making the model similar to the dimensions below worked well for three.js

3D Builder Resize

So the model rotates around the centre, set the location as below. You can see the model will sunk half way into the floor.

3D Builder Centering Model

The model is now ready to be export as a GLB file and used in three.js but the filesize is large and will take ages to download and display from the SD card on the ESP32 board. The easiest solution I found to reduce the file size is the Creators 3D online viewer and converter found here: https://www.creators3d.com/online-viewer. Drag and drop the GLB file and choose the following options to export a new GLB file:

Creators 3D Optimizations

Assembling the Backpack

Once you have your 3D file ready you can assemble the backpack with the ESP32 and motion sensor. I used Tinkercad to create the parts and then just glued them together. You can see in the Tinkercad screen below that I used an STL export of the robot to make a robot shaped hole in the backpack so it fits over the robot.

Backpack design in Tinkercad

The 3D prints look like this:

3D Prints before assembly

Assembled, the backpack looks like this:

Backpack 3D Print Rear
Back
Backpack 3D Print Inside
Inside

The Arduino Code

The Sketch consists of two parts. The first part reads from the movement sensor and converts it into calibrated data. The second part is a webserver hosting a web page. The webpage contains JavaScript including the three.js library that receives the calibrated data over websockets and updates the position of the 3D model.

There’s several Arduino libraries for the MPU-9250. I used the refactored version of kriswiner library found here: https://github.com/hideakitai/MPU9250. This can be installed using the Arduino Library Manager:

MPU9250 by hideakitai in the Arduino Library Manager

To increase the sensitivity level of the readings. I changed line 20 in Documents\Arduino\libraries\MPU9250\mPU9250.h to:

template <typename WireType, AFS AFSSEL = AFS::A2G, GFS GFSSEL = GFS::G250DPS, MFS MFSSEL = MFS::M16BITS>

The ESPAsyncWebserver library also needs to be installed by downloading the ‘Download ZIP’ link and in the IDE installing it with Sketch > Include Library > Add .ZIP Library.

Connect the ESP32 and MPU9250 as below, copy the files in the SD folder from the Github repository here – https://github.com/robotzero1/esp32-mpu9250-three.js onto the SD card, insert it and connect via USB. Before assembling the backpack, the MPU9250 should be calibrated.

Wiring Diagram for the MPU9250 Backpack

The first time you upload the script make sure you have the Serial Monitor open in the Arduino IDE because a calibration script will start. During the first part of calibration the MPU9250 sensor needs to be flat. During the second part, move it in a figure of 8 movement using the full extent of your arm.

MPU9250 Flat
Calibration step 1
MPU9250 Figure 8
Calibration step 2

After the calibration has completed you can assemble the backpack.

The project is now complete. If you open the IP address shown in the Serial Monitor or on your router’s list of connections in the browser you will see the 3D model of the robot. The on-screen robot should follow the movements of the physical robot and pressing the button on the backpack should result in lightning between the robot’s hands.

Project Demonstration

Here’s a quick demo of the 3D robot following the movement of the physical robot:

I’ve uploaded another demo but without the ESP32 and sensor so you can see the effect without building the project. https://robotzero.one/examples/threejs/robot/orbit-control.html Click on the robot to move it.

The three.js Library and ESPAsyncWebserver

If you want to recreate the code from scratch and not use the files in the SD card folder you can install three.js in a few ways . For me the easiest way was to download the zip file from Github, extract it and copy the build and examples folders to my app folder and refer to them like this in my HTML file:

import * as THREE from './build/three.module.js';
import { GLTFLoader } from './examples/jsm/loaders/GLTFLoader.js';

For a simple application with only a few dependencies the ESP32 works fine but I discovered that the lightning code requested too many JS files for ESP32 webserver to cope with so I used Parcel to bundle all the JS files into one file and this only requires one request:

  webserver.on("/parcel.e31bb0bc.js", HTTP_GET, [](AsyncWebServerRequest * request) {
    request->send(SD_MMC, "/parcel.e31bb0bc.js", "application/javascript");
  });

I found Parcel easier to set up and use than Webpack. The Github repository contains the files that I used to build the bundled script file here: https://github.com/robotzero1/esp32-mpu9250-three.js/tree/master/parcel

MPU-9250 Libraries for the ESP32

There’s quite a few MPU-9250 libraries for the Arduino IDE. Some of them work with the ESP32.

Probably the most comprehensive library which includes sensor fusion algorithms – https://github.com/kriswiner/MPU9250
Port of the kriswiner’s library for the GY-21 (the board I have) to include the BMP280 sensor – https://github.com/TheChapu/GY-91
Refactored version of kriswiner library (used in this project) – https://github.com/hideakitai/MPU9250
An ESP32 library that allows control over the 9250’s DMP features – https://github.com/rupin/SparkFun_MPU-9250-DMP_Arduino_Library .
An alternative library – https://github.com/bolderflight/MPU9250/
Easiest library I found – https://github.com/asukiaaa/MPU9250_asukiaaa just works with an ESP32 but doesn’t have ‘fusion’. Need to run the GetMadOffset sketch to see magnet data.

For my purposes it came down to choice between the refactored kriswiner library and using the correction algorithms in the library code or using the ported Sparkfun library where the correction algorithms are done on the chip.

Other Sensor Options

BNO080 replacement for the not well received 005 devices – https://es.aliexpress.com/item/32915488261.html (and the library – https://github.com/jps2000/BNO080)

Links

Great article about 9 D0F sensors – https://github.com/kriswiner/MPU6050/wiki/Affordable-9-DoF-Sensor-Fusion

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

scroll to top