Download file from presigned url

[MaxLink]

I'm trying to download a text-file from a pre-sigend AWS S3 url using the esp http client but I'm running into some issues. With the code-sample appended at the end of the post, I can download the Google logo (the url used), the sequence of events will be something like:

Code: Select all

HTTP_EVENT_ON_CONNECTED
HTTP_EVENT_HEADER_SENT
HTTP_EVENT_ON_HEADER
HTTP_EVENT_ON_HEADER
more HTTP_EVENT_ON_HEADER ...
HTTP_EVENT_ON_DATA
more HTTP_EVENT_ON_DATA ...
HTTP_EVENT_ON_FINISH
HTTP_EVENT_DISCONNECTED

While, if I used a pre-signed url for a file on S3 I only get:

Code: Select all

HTTP_EVENT_ON_CONNECTED
HTTP_EVENT_ERROR
HTTP_EVENT_DISCONNECTED

I verified both links in Python, and this row works fine for both urls:

Code: Select all

requests.get(url, verify=False)

I'm using ESP-IDF 4.4 and I've set the following in the config:

• Enable ESP-TLS Serve

• Allow potentially insecure options

• Skip server certificate verification by default

Anyone who has seen anything similar, has tried downloading from a pre-signed S3 link or has any idea how I can debug this?

BR
Max

Code: Select all

esp_err_t _https_event_handle(esp_http_client_event_t *evt) {
    switch(evt->event_id) {
        case HTTP_EVENT_ERROR:
            cout << "HTTP_EVENT_ERROR" << endl;
            break;
        case HTTP_EVENT_ON_CONNECTED:
            cout << "HTTP_EVENT_ON_CONNECTED" << endl;
            break;
        case HTTP_EVENT_HEADER_SENT:
            cout << "HTTP_EVENT_HEADER_SENT" << endl;
            break;
        case HTTP_EVENT_ON_HEADER:
            cout << "HTTP_EVENT_ON_HEADER" << endl;
            break;
        case HTTP_EVENT_ON_DATA:
            cout << "HTTP_EVENT_ON_DATA" << endl;
            break;
        case HTTP_EVENT_ON_FINISH:
            cout << "HTTP_EVENT_ON_FINISH" << endl;
            break;
        case HTTP_EVENT_DISCONNECTED:
            cout << "HTTP_EVENT_DISCONNECTED" << endl;
            break;
    }
    return ESP_OK;
}

void get_data()
{
    esp_http_client_config_t config = {
        .url = "https://www.google.se/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png",
        .cert_pem = NULL,
        .method = HTTP_METHOD_GET,
        .timeout_ms = 10000,
        .event_handler = _https_event_handle,
        .transport_type = HTTP_TRANSPORT_OVER_SSL,
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);
    esp_err_t err = esp_http_client_perform(client);
    esp_http_client_cleanup(client);
}

[MicroController]

any idea how I can debug this?

Well, how about starting by logging what error is reported?

[MaxLink]

That would make a lot of sense, yes. Sadly I've done this, just a while ago so I forgot to include it in the post, sorry for that. The error from esp_http_client_perform is just a generic -1.The data in esp_http_client_event_t has length 0. So I am assuming that means no information, but I haven't found any example for how you could get additional error info from the event. And the struct seem to only have id, data, user-data and header(only for header event). Or is there a way of getting something more out of it?

[MicroController]

Look at esp_http_client_get_status_code(...) and esp_http_client_get_errno(...).

[Dharanesh]

how do you read that file after download!!!!!!!!!!!!

currently am using ESP32c6 with ESP IDF v5.2.1

Code: [Select all] [Expand/Collapse]

1. I (66530) avr: HTTP_EVENT_ON_DATA, len=512
2. I (66540) avr: HTTP_EVENT_ON_DATA, len=512
3. I (66540) avr: HTTP_EVENT_ON_DATA, len=512
4. I (66550) avr: HTTP_EVENT_ON_DATA, len=512
5. I (66550) avr: HTTP_EVENT_ON_DATA, len=512
6. I (66560) avr: HTTP_EVENT_ON_DATA, len=512
7. I (66560) avr: HTTP_EVENT_ON_DATA, len=33
8. I (66570) avr: HTTP_EVENT_ON_FINISH
9. I (66570) avr: HTTP_EVENT_HEADER_SENT

GeSHi © Codebox Plus Extension

i think i download the file but i can't read that file

from s3 bucket i upload the direct bin file of the AVR firmware and i also check the storage.bin (copy that file while build )

here is the way i tried

Code: [Select all] [Expand/Collapse]

1.     uint8_t buffer[BUF_SIZE];
2.     esp_http_client_config_t config = {
3.         .url = "",
4. #ifdef CONFIG_EXAMPLE_USE_CERT_BUNDLE
5.         .crt_bundle_attach = esp_crt_bundle_attach,
6. #else
7.     // .cert_pem = (char *)server_cert_pem_start,
8. #endif /* CONFIG_EXAMPLE_USE_CERT_BUNDLE */
9.         // .event_handler = _https_event_handle,
10.         .keep_alive_enable = true,
11. #ifdef CONFIG_EXAMPLE_FIRMWARE_UPGRADE_BIND_IF
12.         .if_name = &ifr,
13. #endif
14.     };
15.  
16.     char url_buffer[MAX_URL_LEN];
17.     // snprintf(url_buffer, MAX_URL_LEN, "%s%s%s%s%s%s",OTA_URL, ENVIRONMENT,VARIANT,GLASS_MM,AVR_MCU, avr_ota_version);
18.     snprintf(url_buffer, MAX_URL_LEN, "%s%s%s%s%s%s",OTA_URL, ENVIRONMENT,VARIANT,GLASS_MM,AVR_MCU, "/storage.bin");
19.     config.url = (const char *)malloc(strlen(url_buffer) + 1);
20.     if (config.url != NULL) {
21.         strcpy((char *)config.url, url_buffer);
22.         ESP_LOGI("OTA", "%s", config.url);
23.     }
24.     config.method = HTTP_METHOD_GET;
25.     config.timeout_ms = 10000;
26.     config.event_handler = _https_event_handle;
27.     config.transport_type = HTTP_TRANSPORT_OVER_SSL;
28.  
29.  
30.     ESP_LOGI("hiii", "block 1");
31.     esp_http_client_handle_t client = esp_http_client_init(&config);
32.     esp_http_client_perform(client);
33.     esp_err_t err = esp_http_client_open(client, 0);
34.     if (err != ESP_OK) {
35.         ESP_LOGE("avr", "Failed to open HTTP connection (%s)", esp_err_to_name(err));
36.         esp_http_client_cleanup(client);
37.     }else{
38.         ESP_LOGI("avr", "HTTP connection opened");
39.     }
40.     ESP_LOGI("hiii", "block 2");
41.    
42.     char spiff_file[1024];
43.     snprintf(spiff_file, sizeof(spiff_file), "/firm%s", avr_ota_version);
44.  
45.     esp_err_t ret;
46.     size_t total_bytes, used_bytes;
47.     ret = esp_spiffs_info(spiff_file, &total_bytes, &used_bytes);
48.     if(ret == ESP_OK){
49.         printf("File '%s' exists\n", spiff_file);
50.     }else{
51.         printf("File '%s' does not exist\n", spiff_file);
52.     }
53.  
54.     FILE* file = fopen(spiff_file, "wb");
55.     if (file == NULL) {
56.         ESP_LOGE("avr", "Failed to open file for writing");
57.         esp_http_client_cleanup(client);
58.     }
59.  
60.     esp_http_client_fetch_headers(client);
61.     int content_length = esp_http_client_get_content_length(client);
62.     if(content_length <= 0){
63.         printf("Invalid content length\n");
64.         fclose(file);
65.         esp_http_client_cleanup(client);
66.         // return;
67.     }

GeSHi © Codebox Plus Extension

AND here is the LOG

Code: [Select all] [Expand/Collapse]

1. I (66560) avr: HTTP_EVENT_ON_DATA, len=512
2. I (66560) avr: HTTP_EVENT_ON_DATA, len=33
3. I (66570) avr: HTTP_EVENT_ON_FINISH
4. I (66570) avr: HTTP_EVENT_HEADER_SENT
5. I (66570) avr: HTTP connection opened
6. I (66580) hiii: block 2
7. File '/firm/io-6s2f-atc6.bin' does not exist
8. I (66640) avr: HTTP_EVENT_ON_HEADER, key=x-amz-id-2, value=n4WpQ5GZ+2vw9bsdVUQFWff8a3ZSf6HmVDQNRBMfB6QzqE4qDcrvrsMtzIwV91AwYpAJ7/Y3SnE=
9. I (66640) avr: HTTP_EVENT_ON_HEADER, key=x-amz-request-id, value=FR80R2X9W5H45AVA
10. I (66650) avr: HTTP_EVENT_ON_HEADER, key=Date, value=Thu, 28 Mar 2024 05:38:14 GMT
11. I (66660) avr: HTTP_EVENT_ON_HEADER, key=Last-Modified, value=Wed, 27 Mar 2024 11:59:02 GMT
12. I (66670) avr: HTTP_EVENT_ON_HEADER, key=ETag, value="9c3b7ab01d1b3499844cdcf7904bb98c"
13. I (66680) avr: HTTP_EVENT_ON_HEADER, key=x-amz-server-side-encryption, value=AES256
14. I (66690) avr: HTTP_EVENT_ON_HEADER, key=x-amz-meta-sha256, value=6b3eaf7957c9a8c8722307b378ed6f4fc982b08de749b1db4f88e0990a20a47e
15. I (66700) avr: HTTP_EVENT_ON_HEADER, key=x-amz-meta-s3b-last-modified, value=20240327T115509Z
16. I (66710) avr: HTTP_EVENT_ON_HEADER, key=Accept-Ranges, value=bytes
17. I (66710) avr: HTTP_EVENT_ON_HEADER, key=Content-Type, value=application/octet-stream
18. I (66720) avr: HTTP_EVENT_ON_HEADER, key=Server, value=Ama
19. I (66730) avr: HTTP_EVENT_ON_HEADER, key=Content-Length, value=983040
20. I (66740) hiii: block 3
21. I (66740) hiii: block 4
22. I (66740) hiii: block 5
23. I (66740) avr: Reading file /firm/io-6s2f-atc6.bin
24. I (66750) hiii: block 5.1
25. File '/firm/io-6s2f-atc6.bin' does not exist

GeSHi © Codebox Plus Extension