Wear levelling FATFS performance really slow even for reading?

jcsbanks
Posts: 305
Joined: Tue Mar 28, 2017 8:03 pm

Wear levelling FATFS performance really slow even for reading?

Postby jcsbanks » Wed Apr 24, 2019 2:13 pm

Why is it so slow even for reading a file? It is < 250KB/s.

So if I do an operation on 500KB, that involves processing the data multiple times in multiple ways in PSRAM, it is instant from the user point of view. But loading two 500KB files takes over 4 seconds, about the same speed as writing, which also sometimes produces interrupt watchdog timeouts like OTA updates do with large partitions when the CAN driver is active since the CAN driver's interrupt functions do not seem to be in IRAM. It is actually faster to get stuff from a websocket over Wifi than it is to get it from the SPI flash using VFS/FATFS. It is really quick if a file is embedded but I need encrypted flash and the flexibility to write occasionally.

I'm using 4096 byte sectors and performance mode. I'm starting to think I need to read and write partitions separately, but that is a real shame to fragment a system that otherwise gives a lot of flexibility. I'm having to start threading file opens whilst other things happen as they are so obviously slow. For saving, I'm saving to a RAM disk to speed things up (like saving half a second when writing a few bytes). It is manageable, just, but it is the one area I'm really fighting, performance is otherwise stellar and once stuff is in PSRAM, the thing flies along.

Is this performance expected? Code snippet for reading a file into a RAM buffer included. It is pretty standard stuff unless I'm doing something wrong with fread parameter that is 1, but that was my understanding of how to use it. It is the fread that takes all the time, CPU speed is 240MHz, flash is 16MB WROVER at 80MHz QIO. CPU usage from other tasks is a few percent total.

Code: Select all

			//open file
			FILE* f = NULL;
			f = fopen(filepath, "rb");
			if (!f) {
				my_print_web("*Unable to open file %s for reading!", filepath);
				return false;			
			}
			//get filesize
			fseek(f, 0, SEEK_END);
			int filesize = ftell(f);
			fseek(f, 0, SEEK_SET);
			*pstart = (uint8_t*)malloc(filesize);
			if (!*pstart){
				my_print_web("*Unable to malloc file %s size %d!", filepath, filesize);
				retval = false;
			}
			//read file
			else{
				if (fread(*pstart, filesize, 1, f) != 1){
					my_print_web("*Error reading file %s size %d!", filepath, filesize);
					free(*pstart);
					*pstart = NULL;
					retval = false;
				}else{
					*pend = *pstart + filesize;
				}
			}	
			fclose(f);
			

flipatoms
Posts: 3
Joined: Mon Sep 23, 2019 8:45 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby flipatoms » Mon Sep 23, 2019 8:48 pm

I have a similar problem. I'm saving log files first to a memory buffer then to flash with fatfs. 3-4 KB takes 470 ms :(

jcsbanks
Posts: 305
Joined: Tue Mar 28, 2017 8:03 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby jcsbanks » Tue Sep 24, 2019 7:59 pm

I abandoned FATFS for reading large files and only write small difference files to it. Large files are stored in flash at OTA update time and read using mmap.

doglike
Posts: 63
Joined: Fri Aug 18, 2017 4:21 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby doglike » Thu Feb 27, 2020 11:58 am

flipatoms wrote:
Mon Sep 23, 2019 8:48 pm
I have a similar problem. I'm saving log files first to a memory buffer then to flash with fatfs. 3-4 KB takes 470 ms :(
Same issue...
Writing a log file with ~60kB takes ~9sec.
During this time I have to block that core completely for the write-task, otherwise it takes even longer.

This is disappointing...there is no filesystem implementation, that work satisfactory. FFat, Spiffs (gets slower and slower with increasing fill of flash).... :( :( :(

vonnieda
Posts: 145
Joined: Tue Nov 07, 2017 3:42 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby vonnieda » Thu Feb 27, 2020 3:06 pm

Check out LittleFS: https://github.com/ARMmbed/littlefs

It's much faster, and pretty easy to implement using IDF's VFS layer.

Jason

doglike
Posts: 63
Joined: Fri Aug 18, 2017 4:21 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby doglike » Thu Feb 27, 2020 9:19 pm

Thanks! Is there any example? I am trying to implement it since some hours with no success :(

vonnieda
Posts: 145
Joined: Tue Nov 07, 2017 3:42 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby vonnieda » Thu Feb 27, 2020 9:22 pm

doglike wrote:
Thu Feb 27, 2020 9:19 pm
Thanks! Is there any example? I am trying to implement it since some hours with no success :(
I can't share my implementation unfortunately, but have a look at https://github.com/lllucius/esp32_littleflash. I learned very much from that when making my own implementation.

Jason

doglike
Posts: 63
Joined: Fri Aug 18, 2017 4:21 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby doglike » Tue Mar 10, 2020 7:43 am

Thanks vonnieda! Studied the library, but didn't have time yet to implement it in my project.
Could you maybe share your experiences with littlefs regarding writing speed?
I my usecase I am writing 60kB logfiles into the flash in ~9sec (FFAT).
With SPIFFS the job was done in ~800ms, what was ok for my purpose. But SPIFFS has other issue as written before...

How much speed improvement is possible with littlefs?

vonnieda
Posts: 145
Joined: Tue Nov 07, 2017 3:42 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby vonnieda » Tue Mar 10, 2020 2:33 pm

doglike wrote:
Tue Mar 10, 2020 7:43 am
Thanks vonnieda! Studied the library, but didn't have time yet to implement it in my project.
Could you maybe share your experiences with littlefs regarding writing speed?
I my usecase I am writing 60kB logfiles into the flash in ~9sec (FFAT).
With SPIFFS the job was done in ~800ms, what was ok for my purpose. But SPIFFS has other issue as written before...

How much speed improvement is possible with littlefs?
Here's results from a little file IO test I wrote a while back. Results are in milliseconds:
create 100 small files: 42152
100 directory reads: 1712
write 1MB file: 26592
read 1MB file: 4973
remove 1MB file: 153
remove 100 small files: 17441
total: 93023

doglike
Posts: 63
Joined: Fri Aug 18, 2017 4:21 pm

Re: Wear levelling FATFS performance really slow even for reading?

Postby doglike » Thu Mar 12, 2020 8:09 am

OK fine. I could get it running yesterday and it looks good :)
I am using this implementation, because of a very nice esp_spiffs_xxx like API: https://github.com/joltwallet/esp_littlefs
First tests:
- writing 60kB file in 600ms :) (SPIFFS=800ms, FFAT=9sec)

Thanks for that, vonnieda


@ESP team: Is there any native littlefs implementation planned in later IDF releases ?

Who is online

Users browsing this forum: ESP_Roland, ESP_rrtandler and 117 guests