A guide on how to add or edit the flash memory partitions in the Arduino IDE
Sometimes when you are creating a complicated Sketch especially when using WiFi or Bluetooth libraries you can run out of space in the flash memory on your device. You see an error like this
Sketch too big. Sketch uses 2233498 bytes (113%) of program storage space. Maximum is 1966080 bytes.
Most of the time this can be fixed by simply choosing a different in-built partition scheme from the Tools menu such as Huge App:
However if you need the OTA (Over the Air) partition or need to store data that survives power off and also need more space than is available in the IDE schemes then you can tweak the partitions to change the space available in different parts of the flash storage. Partitions can be thought of as slices of a cake or pizza. You can make them larger or smaller but the overall space is the same.
Create Partitions
The partition schemes are stored as CSV files. They can be found in one of two places, depending on how you installed the Arduino IDE.
Arduino IDE installed from the Windows Store:
C > Users > *your-user-name* > Documents > ArduinoData > packages > esp32 > hardware > esp32 > 1.0.1 > tools > partitions
Arduino IDE installed from the Arduino website:
C > Users > *your-user-name* > AppData > Local > Arduino15 > packages > esp32 > hardware > esp32 > 1.0.1 > tools > partitions
They look like this when opened in Notepad
You can see the min_spiffs partition.csv scheme has two app partitions (app0 and app1) of 0x1E0000. Using a Hex to Decimal online converter the 0x1E0000 hex number converts to 1966080 bytes hence the error message above Sketch uses 2233498 bytes (113%) of program storage space. Maximum is 1966080 bytes
The huge_app partition scheme has only one app partition of 0x300000 which is 3145728 bytes. This is plenty for the application but there is nowhere to store the permanent data as app0 is overwritten every time the board is flashed.
ESP32 Hardware Library v1.0.5 New Partition System
The ESP32 Arduino Hardware Library release 1.0.5 brings with a new much easier way of uploading a custom Partition Scheme. The new partition file can be added to the same directory as the Sketch and are uploaded with the Sketch. Choose ‘Huge App’ from the Tools > Partition Scheme menu to make sure the Sketch will upload. The actual partition used will be the one uploaded.
Persistent Storage
I needed to add a new partition for saving computed faces from a face recognition application. The computed faces are stored in a part of the on-board flash that has to be set up in certain way. A new partition type is needed. I created a new partition scheme file like this:
In this file I changed the size of the app0 partition to be 0x280000 which is 2621440 bytes. I also created a second partition called fr with 0xEF000 (978944 bytes) of space . This second partition is used by my application to store permanent data. I used the name, type and subtype from this code for fr_flash.h here
#define FR_FLASH_TYPE 32
#define FR_FLASH_SUBTYPE 32
#define FR_FLASH_PARTITION_NAME “fr”
You can download my partition file from here: https://robotzero.one/wp-content/uploads/2019/04/rzo_partitions.csv
The offset of each partition should be the value of the previous offset plus the size of the previous partition. With SSDs and flash memory there are recommended sizes for partitioning disks. I had an error about offsets when I tried to create this using my own values and in the end I used HEX values from other examples to create my partitions. If someone knows the recommendations please let me know.
Adding the New Partition Scheme to the IDE
From v1.0.5 of the Arduino Hardware Library the instructions below are no longer needed. Just add a new partition named partitions.csv to the same directory as your Sketch. It won’t appear in the IDE tools menu but if you select ‘Huge App’ the Sketch will upload and use the new partitions.
Pre v1.0.5 instructions…
Once the new partition scheme file has been created you need to add a reference to it in the boards manager so it can be selected in the Arduino IDE. The boards.txt file is found in one of two places
Arduino IDE installed from the Windows Store:
C > Users > *your-user-name* > Documents > ArduinoData > packages > esp32 > hardware > esp32 > 1.0.1
Arduino IDE installed from the Arduino website:
C > Users > *your-user-name* > AppData > Local > Arduino15 > packages > esp32 > hardware > esp32 > 1.0.1
The boards.txt file can be edited in Notepad. Each definition is separated by a line like this: #########
I copied ESP Dev Board definition and changed the name and added three new lines for the new partition file. You could just add these lines to an existing definition if you prefer:
rzo.menu.PartitionScheme.rzo_partition=Face Recognition (2621440 bytes with OTA) rzo.menu.PartitionScheme.rzo_partition.build.partitions=rzo_partitions rzo.menu.PartitionScheme.rzo_partition.upload.maximum_size=2621440
You can copy and paste the definition below to create a new board if you have used my rzo partition definition file above.
##############################################################
rzo.name=Robot Zero One
rzo.upload.tool=esptool_py
rzo.upload.maximum_size=1310720
rzo.upload.maximum_data_size=2621440
rzo.upload.wait_for_upload_port=true
rzo.serial.disableDTR=true
rzo.serial.disableRTS=true
rzo.build.mcu=esp32
rzo.build.core=esp32
rzo.build.variant=esp32
rzo.build.board=ESP32_DEV
rzo.build.f_cpu=240000000L
rzo.build.flash_size=4MB
rzo.build.flash_freq=40m
rzo.build.flash_mode=dio
rzo.build.boot=dio
rzo.build.partitions=default
rzo.build.defines=
rzo.menu.PSRAM.disabled=Disabled
rzo.menu.PSRAM.disabled.build.defines=
rzo.menu.PSRAM.enabled=Enabled
rzo.menu.PSRAM.enabled.build.defines=-DBOARD_HAS_PSRAM -mfix-esp32-psram-cache-issue
rzo.menu.PartitionScheme.default=Default
rzo.menu.PartitionScheme.default.build.partitions=default
rzo.menu.PartitionScheme.minimal=Minimal (2MB FLASH)
rzo.menu.PartitionScheme.minimal.build.partitions=minimal
rzo.menu.PartitionScheme.no_ota=No OTA (Large APP)
rzo.menu.PartitionScheme.no_ota.build.partitions=no_ota
rzo.menu.PartitionScheme.no_ota.upload.maximum_size=2097152
rzo.menu.PartitionScheme.huge_app=Huge APP (3MB No OTA)
rzo.menu.PartitionScheme.huge_app.build.partitions=huge_app
rzo.menu.PartitionScheme.huge_app.upload.maximum_size=3145728
rzo.menu.PartitionScheme.min_spiffs=Minimal SPIFFS (Large APPS with OTA)
rzo.menu.PartitionScheme.min_spiffs.build.partitions=min_spiffs
rzo.menu.PartitionScheme.min_spiffs.upload.maximum_size=1966080
rzo.menu.PartitionScheme.fatflash=16M Fat
rzo.menu.PartitionScheme.fatflash.build.partitions=ffat
rzo.menu.PartitionScheme.rzo_partition=Face Recognition (2621440 bytes with OTA)
rzo.menu.PartitionScheme.rzo_partition.build.partitions=rzo_partition
rzo.menu.PartitionScheme.rzo_partition.upload.maximum_size=2621440
rzo.menu.CPUFreq.240=240MHz (WiFi/BT)
rzo.menu.CPUFreq.240.build.f_cpu=240000000L
rzo.menu.CPUFreq.160=160MHz (WiFi/BT)
rzo.menu.CPUFreq.160.build.f_cpu=160000000L
rzo.menu.CPUFreq.80=80MHz (WiFi/BT)
rzo.menu.CPUFreq.80.build.f_cpu=80000000L
rzo.menu.CPUFreq.40=40MHz (40MHz XTAL)
rzo.menu.CPUFreq.40.build.f_cpu=40000000L
rzo.menu.CPUFreq.26=26MHz (26MHz XTAL)
rzo.menu.CPUFreq.26.build.f_cpu=26000000L
rzo.menu.CPUFreq.20=20MHz (40MHz XTAL)
rzo.menu.CPUFreq.20.build.f_cpu=20000000L
rzo.menu.CPUFreq.13=13MHz (26MHz XTAL)
rzo.menu.CPUFreq.13.build.f_cpu=13000000L
rzo.menu.CPUFreq.10=10MHz (40MHz XTAL)
rzo.menu.CPUFreq.10.build.f_cpu=10000000L
rzo.menu.FlashMode.qio=QIO
rzo.menu.FlashMode.qio.build.flash_mode=dio
rzo.menu.FlashMode.qio.build.boot=qio
rzo.menu.FlashMode.dio=DIO
rzo.menu.FlashMode.dio.build.flash_mode=dio
rzo.menu.FlashMode.dio.build.boot=dio
rzo.menu.FlashMode.qout=QOUT
rzo.menu.FlashMode.qout.build.flash_mode=dout
rzo.menu.FlashMode.qout.build.boot=qout
rzo.menu.FlashMode.dout=DOUT
rzo.menu.FlashMode.dout.build.flash_mode=dout
rzo.menu.FlashMode.dout.build.boot=dout
rzo.menu.FlashFreq.80=80MHz
rzo.menu.FlashFreq.80.build.flash_freq=80m
rzo.menu.FlashFreq.40=40MHz
rzo.menu.FlashFreq.40.build.flash_freq=40m
rzo.menu.FlashSize.4M=4MB (32Mb)
rzo.menu.FlashSize.4M.build.flash_size=4MB
rzo.menu.FlashSize.2M=2MB (16Mb)
rzo.menu.FlashSize.2M.build.flash_size=2MB
rzo.menu.FlashSize.2M.build.partitions=minimal
rzo.menu.FlashSize.16M=16MB (128Mb)
rzo.menu.FlashSize.16M.build.flash_size=16MB
rzo.menu.FlashSize.16M.build.partitions=ffat
rzo.menu.UploadSpeed.921600=921600
rzo.menu.UploadSpeed.921600.upload.speed=921600
rzo.menu.UploadSpeed.115200=115200
rzo.menu.UploadSpeed.115200.upload.speed=115200
rzo.menu.UploadSpeed.256000.windows=256000
rzo.menu.UploadSpeed.256000.upload.speed=256000
rzo.menu.UploadSpeed.230400.windows.upload.speed=256000
rzo.menu.UploadSpeed.230400=230400
rzo.menu.UploadSpeed.230400.upload.speed=230400
rzo.menu.UploadSpeed.460800.linux=460800
rzo.menu.UploadSpeed.460800.macosx=460800
rzo.menu.UploadSpeed.460800.upload.speed=460800
rzo.menu.UploadSpeed.512000.windows=512000
rzo.menu.UploadSpeed.512000.upload.speed=512000
rzo.menu.DebugLevel.none=None
rzo.menu.DebugLevel.none.build.code_debug=0
rzo.menu.DebugLevel.error=Error
rzo.menu.DebugLevel.error.build.code_debug=1
rzo.menu.DebugLevel.warn=Warn
rzo.menu.DebugLevel.warn.build.code_debug=2
rzo.menu.DebugLevel.info=Info
rzo.menu.DebugLevel.info.build.code_debug=3
rzo.menu.DebugLevel.debug=Debug
rzo.menu.DebugLevel.debug.build.code_debug=4
rzo.menu.DebugLevel.verbose=Verbose
rzo.menu.DebugLevel.verbose.build.code_debug=5
In the IDE the new scheme will be available for the new board or any board you add the new definition to:
Using the ESP32 Partition Manager
Since writing this article I found a couple of tools that might make working with the partitions easier: https://github.com/francis94c/esp-partition-gui and for the Arduino IDE: https://github.com/francis94c/ESP32Partitions
Buy Me A Coffee
If you found something useful above please say thanks by buying me a coffee here...
References
Why programmers use Hex: https://www.i-programmer.info/babbages-bag/478-hexadecimal.html?start=1
Hex to Binary Converter: https://www.rapidtables.com/convert/number/hex-to-binary.html
Hex Calculator: https://ncalculators.com/digital-computation/hex-addition-calculator.htm
Partitioned Pizza Photo: Vita Marija Murenaite on Unsplash
Hi!
i will try to use your board definition for a ESP32 CAM board but I have one question.
How it is possible that OTa is enabled if you don’t have two partitions for OTA? and no space also?
Thank you!
You’re right (I think!) the partition should just be called
Face Recognition (2621440 bytes)
because there isn’t the OTA partition for uploading new sketches.
Hi,
I am new in ESP32-CAM, I have follow your project and successfully save permanent face information to flash. But may I know if there is some way I can ease and start over again ? Thank you very much.
Great! This tutorial has a little Sketch that deletes all the faces https://robotzero.one/esp32-face-door-entry/ under Deleting Faces From the Memory
Oh, sorry that I overlook the code….
However, I follow the delete face program, the output as below but when I upload again Camera WebServer. The image still not enrolled. I have try using the new partition (Faceetection) or Huge App but both have same result….Any idea ? Thank you.
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1100
load:0x40078000,len:10088
load:0x40080400,len:6380
entry 0x400806a4
E (1417) fr_flash: No ID Infomation
Faces Read
E (1418) fr_flash: No ID Infomation
All Deleted
It looks like there were no faces stored from the output above.
Are you using the version 1.0.1 of the boards library? https://robotzero.one/wp-content/uploads/2019/01/boards-manager-esp32.jpg
Yes, it is version 1.0.1. I try few more times to install delete face and cam web server prog. but still unable to enroll again… Maybe is there some way to clearup the whole partition instead ? Thx
Which tutorial are you using to enroll faces? You don’t need to clear the partitions each time but if you want to use permanent faces you should only use the custom partition table.
Hi,
I uploaded the Camera WebServer example app to test with image enrollment, and tested also your tutorial of CameraWebServer Permanent after deleted the saved image tutorial uploaded. I excepted the first facedetected will enroll but both result show intrusion detected…. Therefore i believed the image was not successfully removed. Thank you.
I used the esptool with “erase_flash”, but still got “No Match Found” in serial monitor / “Intrusion detected” in the web UI for the both Camera Web Server or Camera Web Server Permanent….stuck here and don’t know why….
Hi, You shouldn’t need to erase anything. What happens if you follow the instructions here: https://robotzero.one/esp32-face-door-entry/ to create a new partition type and then make a copy of the Camera Web Server example and make the changes. Can you enrol faces? IF that part works then you should be able to then upload the sketch in the second part (without deleting anything) and use those enrolled faces to trigger actions.
Sorry to disturb you again I just have resolve the problem now. Your tutorial work perfectly. Thank you very much,
Hi, i am unable to understand: How to edit the board.txt. I copied and pasted the text as you said….But unable to get partition schema in Arduino ide.
Can, you share the edited board.txt file
Please,help
Hi, Did you restart the IDE? It should work if you follow the steps.
hello , i have the same problem as yours . how did you solve it ?
hi , if I increase the app0 size what else I have to change in the same partition table you create ?
I tried increasing the app0 , when I plug the esp the sketch is upload but the esp continuously reset !
It’s like a pizza, if you make one bit bigger another has to be smaller. They also have to have have the correct starting place. It’s something I don’t know fully.
Hi sarah, I have the same problem when I tried increasing the app0. Did you solve the problem?
Take a look at this.. https://github.com/francis94c/ESP32Partitions it might help
Appreciate the breakdown here. This walked through the concepts nicely.
One small correction you might want to make here:
>From v1.0.5 of the Arduino Hardware Library the instructions below are no longer needed. Just add a new partition named partition.csv to the same directory as your Sketch.
It is actually looking for a “partitions.csv” file (plural) as mentioned here:
https://github.com/espressif/arduino-esp32/issues/2258
From a quick test, it won’t show up in the menu, but the partitions will be created properly.
Thanks for pointing out the error. I’ve fixed it now and added a little more text.
Hi, I have some problems with the error “E (3848) fr_flash: Not found”, I tried to add partition file csv in the directory of the file arduino, also adding the partitions and the other things said in this page but I have the same error. How can I solve it? Also I cannot find the “fr_flash.h” library. Help me please. Thanks
Hi, which tutorial are you following? Normally you need v1.0.4 or 1.0.5 of the ESP32 Hardware Library