ESP-IDF SPIFFS can't find files on ESP32?

nockieboy
Posts: 13
Joined: Fri Sep 21, 2018 1:21 pm

ESP-IDF SPIFFS can't find files on ESP32?

Postby nockieboy » Tue Sep 25, 2018 2:35 pm

I'm building a test http server application using the ESP-IDF on my ESP32 and learning C whilst I'm doing it. I've sorted out the http server part, now I'm trying to move on to getting SPIFFS working so I can serve html, css and js files from the ESP32.

SPIFFS is working, as best I can tell - I've used the example SPIFFS code to add the functionality to my project and it has formatted the file system on the ESP32 and initialises just fine, according to the serial console.

I've now tried to upload some files (just .htm, .css and .js files) which I've done as follows:
  • Relevant files were placed in a folder (.data)
    I've used mkspiffs to build an image of the file folder:

    Code: Select all

    mkspiffs -c data -b 4096 -p 256 -s 0x100000 spiffs.bin
This has produced a spiffs.bin file, which I have then uploaded to the ESP32 at offset 0xF0000 (according to the custom partition.csv, which is identical to the one in the SPIFFS example project). I have verified the contents of spiffs.bin with a hex editor and it contains the three files it is supposed to.

Code: Select all

python esptool.py --chip esp32 --port COM14 --baud 115200 write_flash -z 0xF0000 spiffs.bin
This seems to work - the script completes without any errors.

The ESP32 is reset and the following code executes - it should say that the files exist, but it says it can't find them!

Code: Select all

void initialise_SPIFFS()
{
    ESP_LOGI(TAG, "Initializing SPIFFS");
    esp_vfs_spiffs_conf_t conf = {
      .base_path = "/spiffs",
      .partition_label = NULL,
      .max_files = 5,
      .format_if_mount_failed = true
    };

    // Use settings defined above to initialize and mount SPIFFS filesystem.
    // Note: esp_vfs_spiffs_register is an all-in-one convenience function.
    esp_err_t ret = esp_vfs_spiffs_register(&conf);

    if (ret != ESP_OK) {
        if (ret == ESP_FAIL) {
            ESP_LOGE(TAG, "Failed to mount or format filesystem");
        } else if (ret == ESP_ERR_NOT_FOUND) {
            ESP_LOGE(TAG, "Failed to find SPIFFS partition");
        } else {
            ESP_LOGE(TAG, "Failed to initialize SPIFFS (%s)", esp_err_to_name(ret));
        }
        return;
    }

    ret = esp_spiffs_info(NULL, &total, &used);
    if (ret != ESP_OK) {
        ESP_LOGE(TAG, "Failed to get SPIFFS partition information (%s)", esp_err_to_name(ret));
    } else {
        // SPIFFS set up a-ok
        SPIFFS_OK = true;
        // Check if files exist
        struct stat st;
        if (stat("/spiffs/dashboard.css", &st) == 0) {
            ESP_LOGI(TAG, "Found dashboard.css!");
        } else {
            ESP_LOGI(TAG, "Can't find dashboard.css!");
        }
        if (stat("/spiffs/index.htm", &st) == 0) {
            ESP_LOGI(TAG, "Found index.htm!");
        } else {
            ESP_LOGI(TAG, "Can't find index.htm!");
        }
        if (stat("/spiffs/scripts.js", &st) == 0) {
            ESP_LOGI(TAG, "Found scripts.js!");
        } else {
            ESP_LOGI(TAG, "Can't find scripts.js!");
        }
    }

}
What am I doing wrong? :roll:

mikemoy
Posts: 599
Joined: Fri Jan 12, 2018 9:10 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby mikemoy » Tue Sep 25, 2018 10:40 pm

Did you ever find a solution to this ?

nockieboy
Posts: 13
Joined: Fri Sep 21, 2018 1:21 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby nockieboy » Wed Sep 26, 2018 6:52 am

No, not yet. :cry:

mikemoy
Posts: 599
Joined: Fri Jan 12, 2018 9:10 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby mikemoy » Wed Sep 26, 2018 11:09 am

I got this working last night. There is allot the docs assume you already know :roll:

I am still honing in values but this should get you started as well.

#1) create a "partitions.csv" file in your project folder. This is what I have in there.

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, spiffs, 0x210000 , 1M, 
#2) in menuconfig select Serial flasher config ---> Flash size, and set it for 4MB
(NOTE: I was uploading a large site so you may not need to do this step)

#3) in menuconfig select Partition Table ---> Partition Table (Custom partition table CSV) ---> and choose "Custom partition table CSV" and press select, this should bring you back to Partition Table (Custom partition table CSV). Now scroll one down to "Custom partition CSV file" and press enter, then enter the filename "partitions.csv" and choose ok.

#4) make your project again

#5) Now we need to get the files into it. I believe you already know this step so I used this (change C:/z to your folder where your files are at)

Code: Select all

./mkspiffs.exe -c C:/z -b 4096 -p 256 -s 0x100000 spiffs.bin
Then to upload it:

Code: Select all

python $IDF_PATH/components/esptool_py/esptool/esptool.py --chip esp32 --port COM8 --baud 921600 write_flash -z 0x210000 spiffs.bin
NOTE: the 0x210000 in this line should be the same as in yoru .vsc file.

#6) now flash your board and it should be good now.

FYI, I wanted to list the directory and found this example which does a great job at it.
https://github.com/loboris/ESP32_spiffs ... stSpiffs.c

Now I am off to see how i can stream these file in the HTTP server, and whole other nut to crack.

nockieboy
Posts: 13
Joined: Fri Sep 21, 2018 1:21 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby nockieboy » Wed Sep 26, 2018 11:57 am

mikemoy wrote:I got this working last night. There is allot the docs assume you already know :roll:
I'm finding that a lot with those docs... :?
mikemoy wrote: I am still honing in values but this should get you started as well.

#1) create a "partitions.csv" file in your project folder. This is what I have in there.

Code: Select all

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 1M,
storage,  data, spiffs, 0x210000 , 1M, 
Yes, I've been using the partitions.csv file from the ESP-IDF SPIFFS examples in the github repo - seems there's a bit of information
missing from it - the size parameter for the storage partition! That was the issue I was having and why my ESP32 wasn't seeing any
files, because the partitions file wasn't set up properly. Still, you live and learn... :roll:
mikemoy wrote: FYI, I wanted to list the directory and found this example which does a great job at it.
https://github.com/loboris/ESP32_spiffs ... stSpiffs.c

Now I am off to see how i can stream these file in the HTTP server, and whole other nut to crack.
Lovely, thanks for the help and the link - will check that out! I'm just glad the ESP32 is detecting the files now. As with you, off to sort out serving files to the browser now! :D

mikemoy
Posts: 599
Joined: Fri Jan 12, 2018 9:10 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby mikemoy » Wed Sep 26, 2018 12:10 pm

Well, i spoke too soon. Though it lists the files and directories just fine. I can open a file without error, but when i try fread, i get an error.
Also, I found out that you have to (enable) long filename support:
→ Component config → FAT Filesystem support → Long filename support
Then you can select Max long filename length (default is 255)
From this post here. https://esp32.com/viewtopic.php?f=2&t=2651
Sadly this did not help yet.

List of Directory [/spiffs/]
-----------------------------------
T Size Date/Time Name
-----------------------------------
f 121155 03/01/1970 06:36 bootstrap.min.css
f 37045 22/02/1972 19:16 bootstrap.min.js
f 318 18/10/1972 17:45 favicon.ico
f 6486 21/10/1972 18:34 index.html
f 86927 02/12/1972 11:50 jquery-3.3.1.min.js
f 378 14/06/1974 12:43 on-off-switch-onload.js
f 1774 18/06/1974 07:44 on-off-switch.css
f 8128 30/06/1974 11:01 on-off-switch.js
f 320 21/08/1974 19:08 test.txt
-----------------------------------
262531 in 9 file(s)
-----------------------------------
SPIFFS: free 672 KB of 934 KB
-----------------------------------

nockieboy
Posts: 13
Joined: Fri Sep 21, 2018 1:21 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby nockieboy » Wed Sep 26, 2018 1:28 pm

Haven't gotten round to working on serving up files yet, but don't know if this forum post would be of any help?

viewtopic.php?t=4830#p20838

mikemoy
Posts: 599
Joined: Fri Jan 12, 2018 9:10 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby mikemoy » Wed Sep 26, 2018 1:34 pm

Thanks, I have seen that. For some reason even though its printing the directory, I cannot open files yet for reading.

nockieboy
Posts: 13
Joined: Fri Sep 21, 2018 1:21 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby nockieboy » Wed Sep 26, 2018 3:16 pm

You can't open the files, or can't send them to the client?

I've got it all up and running now. If you can't open files, check to make sure the buffer you'd set up was big enough (it was only 1K?
I've modified that to set the buffer to size of the file being read).

Here's my code:

Code: Select all

char *readFile(char *fname)
{
    int size, res;
    char *buf;
    struct stat st;

    if (stat(fname, &st) == 0) {
        size = st.st_size;
    } else {
        ESP_LOGE(TAG, " Error - cannot find file!");
        return NULL;
    }

    ESP_LOGI(TAG, " Opening file %s, size: %d", fname, size);

    FILE *fd = fopen(fname, "rb");
    if (fd == NULL) {
        ESP_LOGE(TAG, " Error opening file");
        return NULL;
    }
    
    if (size == 0) {
        ESP_LOGE(TAG, " Error - file size zero");
        return NULL;
    }

    buf = calloc(size, 1); // File buffer
    if (buf == NULL) {
        ESP_LOGE(TAG, " Error allocating read buffer");
        free(buf);
        fclose(fd);
        return NULL;
    }

    res = size;
    res = fread(buf, 1, size, fd);
    if (res <= 0) {
        ESP_LOGE(TAG, " Error reading from file");
    }
    else {
        ESP_LOGI(TAG, " %d bytes read", res);
        buf[res] = '\0';
    }

    return((char*)buf);

    res = fclose(fd);
    if (res) {
        ESP_LOGE(TAG, " Error closing file");
    }
} 
And for the request handler:

Code: Select all

esp_err_t index_get_handler(httpd_req_t *req)
{
    /* Send response with custom headers and body set as the
     * string passed in user context*/
    // Get the file to return
    const char* resp_str = (const char*) readFile("/spiffs/index.htm");
    // Set the response headers
    httpd_resp_set_hdr(req, "status", "200");
    httpd_resp_set_hdr(req, "content-type", "text/html; charset=UTF-8");
    httpd_resp_set_hdr(req, "server", "ESP32-10103");
    // Return the response
    httpd_resp_send(req, resp_str, strlen(resp_str));
    return ESP_OK;
}
EDIT: That said, I'm getting heap errors if I try to reload the page, so my (very) basic knowledge of C is causing issues - probably because the file buffer isn't getting emptied for each file and/or I'm doing something stupid with pointers or something?

mikemoy
Posts: 599
Joined: Fri Jan 12, 2018 9:10 pm

Re: ESP-IDF SPIFFS can't find files on ESP32?

Postby mikemoy » Wed Sep 26, 2018 3:40 pm

You can't open the files
Nope, i get an error. I found that you can embed files into your project. I though this to be a better solution than to upload files, as that would get old if you had to program many devices for production.
I started following down this rabbit hole here but now ran into a issue with adding source directories :roll:

https://docs.espressif.com/projects/esp ... irectories

I want to break away from Arduino, but the docs for IDF are frustrating at times.

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 131 guests