Initializing NVS on a large partition

Oromis
Posts: 21
Joined: Mon Sep 25, 2017 1:44 pm

Initializing NVS on a large partition

Postby Oromis » Fri Aug 17, 2018 12:42 pm

Hi,

our IoT product usually sends a steady stream of data to our server. If we lose connectivity to the server, we buffer the messages in NVS. To be able to bridge longer offline periods without any data loss, we want to use a large partition for NVS, which is why we upgraded our flash memory to 16 MB.

Unfortunately, I noticed that when I increase the NVS partition size, the chip won't start any more due to an out-of-memory error. Looking into the nvs_pagemanager.cpp file, it seams that the NVS system tries to allocate RAM for each page, which exhausts the heap. It works for a 1MB NVS partition, but fails for 2MB or larger partitions.

The backtrace (for a 6MB NVS partition):

Code: Select all

0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x40089273: invoke_abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x4008929f: abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x4012f998: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_op.cc:54
0x4012f95d: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_opv.cc:32
0x40125710: nvs::PageManager::load(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_pagemanager.cpp:24 (discriminator 4)
0x401244b6: nvs::Storage::init(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_storage.cpp:41
0x40123d23: nvs_flash_init_custom at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454
0x40123d8d: nvs_flash_init_partition at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454
Is there any way to create an NVS on a large flash partition? To be clear, I do not want to store large chunks of data (each one is less than 200 bytes) and I like NVS very much for its efficiency and simplicity.

If there is no way, then I probably need create a SPIFFS partition...

Thank you for your help, you guys rock!

chegewara
Posts: 2207
Joined: Wed Jun 14, 2017 9:00 pm

Re: Initializing NVS on a large partition

Postby chegewara » Fri Aug 17, 2018 2:59 pm

I dont remember exact numbers, maybe you will find it somewhere on this forum, but nvs partition is using some heap. The bigger nvs partition is the more heap is reserved.

Ritesh
Posts: 1365
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Initializing NVS on a large partition

Postby Ritesh » Fri Aug 17, 2018 5:06 pm

Oromis wrote:Hi,

our IoT product usually sends a steady stream of data to our server. If we lose connectivity to the server, we buffer the messages in NVS. To be able to bridge longer offline periods without any data loss, we want to use a large partition for NVS, which is why we upgraded our flash memory to 16 MB.

Unfortunately, I noticed that when I increase the NVS partition size, the chip won't start any more due to an out-of-memory error. Looking into the nvs_pagemanager.cpp file, it seams that the NVS system tries to allocate RAM for each page, which exhausts the heap. It works for a 1MB NVS partition, but fails for 2MB or larger partitions.

The backtrace (for a 6MB NVS partition):

Code: Select all

0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x40089273: invoke_abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x4008929f: abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x4012f998: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_op.cc:54
0x4012f95d: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_opv.cc:32
0x40125710: nvs::PageManager::load(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_pagemanager.cpp:24 (discriminator 4)
0x401244b6: nvs::Storage::init(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_storage.cpp:41
0x40123d23: nvs_flash_init_custom at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454
0x40123d8d: nvs_flash_init_partition at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454
Is there any way to create an NVS on a large flash partition? To be clear, I do not want to store large chunks of data (each one is less than 200 bytes) and I like NVS very much for its efficiency and simplicity.

If there is no way, then I probably need create a SPIFFS partition...

Thank you for your help, you guys rock!
Hi,

I think it's better to use SPIFFS file system for high data amount storage purpose instead of using NVS.
Regards,
Ritesh Prajapati

cashchew
Posts: 5
Joined: Wed Sep 18, 2019 4:14 pm

Re: Initializing NVS on a large partition

Postby cashchew » Wed Sep 18, 2019 4:29 pm

Large nvs partition does indeed use a lot of heap. Is there a way to force the heap allocation to use external SPI RAM? SPIFFS is very slow when it comes to read speed if the partition is >2M, and getting a file written successfully is becoming less and less reliable when the capacity usage is 50% and above. Fatfs is easily get corrupted with power outage while a file is open for write, and there are times failed to be remount on next reset.

Ritesh
Posts: 1365
Joined: Tue Sep 06, 2016 9:37 am
Location: India
Contact:

Re: Initializing NVS on a large partition

Postby Ritesh » Sat Sep 21, 2019 6:36 pm

cashchew wrote:
Wed Sep 18, 2019 4:29 pm
Large nvs partition does indeed use a lot of heap. Is there a way to force the heap allocation to use external SPI RAM? SPIFFS is very slow when it comes to read speed if the partition is >2M, and getting a file written successfully is becoming less and less reliable when the capacity usage is 50% and above. Fatfs is easily get corrupted with power outage while a file is open for write, and there are times failed to be remount on next reset.
Hello,

You can modify linker script to allocate heap memory using PSRAM.
Regards,
Ritesh Prajapati

Who is online

Users browsing this forum: No registered users and 135 guests