Skip to content

refactor: handle errors occuring during stream download#2094

Open
jlahovnik wants to merge 4 commits intodevelopfrom
interrupted-download-handling
Open

refactor: handle errors occuring during stream download#2094
jlahovnik wants to merge 4 commits intodevelopfrom
interrupted-download-handling

Conversation

@jlahovnik
Copy link
Copy Markdown
Collaborator

@jlahovnik jlahovnik commented Mar 17, 2026 •
edited
Loading

For provider planetary_computer it was observed that sometimes the stream download gets interrupted. To cope with this situation, the following error handling was implemented:

  • If a ChunkedEncodingError occurs and range requests are available for the url-> retry with Range header starting from where the connection broke.
  • If a ChunkedEncodingError occurs and range requests are not availble, retry the request. This is not ideal because some data will be sent twice but it invoids to completely in interrupt the download and return an invalid zip file. This case mainly occured for preview files.
  • ConnectionError: retry request twice

@jlahovnik jlahovnik self-assigned this Mar 17, 2026
@jlahovnik jlahovnik marked this pull request as draft March 17, 2026 17:25
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 17, 2026

Test Results

    4 files  ±0      4 suites  ±0   3m 17s ⏱️ +4s
  685 tests ±0    684 ✅ ±0  1 💤 ±0  0 ❌ ±0 
2 782 runs  ±0  2 776 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit 23a3624. ± Comparison against base commit 798a34a.

@eodag-bot
Copy link
Copy Markdown
Collaborator

eodag-bot commented Mar 17, 2026

badge

Code Coverage (Ubuntu)

Details 
Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         251      11  95.62%   104-115, 377, 644
config.py                                      305      24  92.13%   70-72, 75, 78, 81, 85, 89, 93-95, 583-585, 707-709, 728, 736, 766-771, 773
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/collection.py                              150       8  94.67%   176, 212, 215, 322, 361, 364, 382, 385
api/core.py                                    787      57  92.76%   266, 552, 600, 643, 683, 703, 744-749, 774, 858-877, 891, 897, 1050, 1055, 1154, 1193-1194, 1290-1291, 1317, 1348-1349, 1375, 1388, 1449-1450, 1467, 1481-1482, 1556-1561, 1573-1576, 1688, 2075, 2197, 2285-2286
api/provider.py                                381      35  90.81%   186, 190-191, 316-337, 405, 521, 525-526, 531-534, 545, 621-631, 821-822, 870, 877, 908-911, 947-948, 956-957
api/search_result.py                           181      19  89.50%   111, 123, 133, 154, 203, 220, 320, 375-378, 448, 453-454, 488, 502, 525-526, 532
api/product/__init__.py                         18       2  88.89%   57, 59
api/product/_assets.py                          52       4  92.31%   97, 191, 194-198
api/product/_product.py                        248      20  91.94%   189, 305-306, 324-325, 434, 463, 470, 577, 597, 620-623, 632-635, 689, 755, 767
api/product/metadata_mapping.py                791      53  93.30%   125-127, 220-225, 249, 307-308, 396, 417, 469-470, 507, 528-531, 554, 566-567, 608, 631, 661-666, 731-736, 748, 756, 988, 1163, 1172-1176, 1193-1198, 1331, 1354, 1363, 1385, 1390, 1442, 1514, 1535, 1561, 1575, 1600, 1646, 1715, 1790
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     29       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       4  81.82%   48, 55, 68-69
plugins/manager.py                             173      16  90.75%   102-107, 179, 201, 219-220, 232, 271-272, 372-375, 387-388
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101       8  92.08%   178-180, 228-229, 255-257
plugins/apis/usgs.py                           182      25  86.26%   157, 263, 297, 339-341, 346, 374-375, 380, 410-417, 428-433, 455-461
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 288-289, 339-343
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       54      39  27.78%   50-55, 68-115
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                63      25  60.32%   62-65, 72-75, 81, 85, 89, 100-116, 131-157
plugins/crunch/filter_property.py               30       5  83.33%   56-61, 64-65
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        401      76  81.05%   270, 304, 353-356, 386-387, 395-399, 479-482, 522-524, 528, 559-560, 566-570, 601, 666-674, 738-833, 845-850, 888, 914, 959-961, 1013
plugins/download/base.py                       280      35  87.50%   133, 162, 309-310, 368-369, 411, 415-426, 440, 517-521, 551, 586-587, 612-621, 680, 701, 723, 731, 765
plugins/download/http.py                       593      94  84.15%   234, 276-279, 341-344, 347, 354-359, 390-392, 409, 424, 484, 519, 533, 547, 557-561, 577-582, 593, 612, 649-652, 673, 683, 690, 855, 887, 917-926, 962, 987-988, 1007-1012, 1021, 1036-1038, 1042, 1045, 1060-1061, 1080-1095, 1163, 1175, 1226, 1232-1244, 1247-1249, 1289-1290, 1302, 1312, 1362-1363, 1393, 1413, 1454-1456, 1510-1511
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         196      18  90.82%   109, 113, 137-143, 200-203, 296, 317, 442, 492, 525-528, 537
plugins/search/build_search_result.py          504      83  83.53%   259-260, 296, 300, 320, 550-561, 576-578, 701, 725, 727, 794, 802-806, 827, 837, 863, 908, 932, 960, 978-993, 1043, 1068, 1071, 1075, 1084, 1090, 1128-1149, 1190, 1217-1218, 1227-1236, 1300, 1315, 1321, 1340-1349, 1470-1471, 1515, 1524-1526, 1583, 1631-1641
plugins/search/cop_marine.py                   268      56  79.10%   57, 65-67, 77-78, 83, 88-89, 105, 107, 110, 176-177, 220, 236, 242, 246, 250, 261, 272-273, 281, 313-316, 322, 343, 347, 351, 355, 359-363, 369-372, 375-392, 409-412, 465-469, 474, 486
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 247-288
plugins/search/qssearch.py                     823      94  88.58%   415-416, 533-534, 557-558, 570-574, 789-795, 853, 949, 956, 1027, 1048, 1051-1052, 1070, 1079-1080, 1107, 1179, 1188, 1193-1210, 1219, 1234, 1243-1246, 1256, 1278, 1367, 1390, 1463-1464, 1470, 1560, 1667-1671, 1737, 1740, 1744-1745, 1766-1769, 1781, 1803-1815, 1823, 1858-1860, 1883-1889, 1896, 1950, 1973, 1978-1979, 1994, 2000, 2010, 2100, 2104, 2115, 2139, 2152, 2160-2170, 2208-2212
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            84      18  78.57%   99-127, 166-169, 182, 224
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              167      43  74.25%   58, 62, 71-75, 86-98, 126-128, 135-140, 216, 219, 257, 267-283, 288, 290, 312, 317, 325, 335
types/bbox.py                                   39      19  51.28%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                          10       0  100.00%
types/queryables.py                            112       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/stac_extensions.py                       114       1  99.12%   282
types/stac_metadata.py                         116      16  86.21%   92, 113-114, 147, 177-191, 200-207
utils/__init__.py                              608      43  92.93%   71, 206, 237-238, 247-273, 276, 291, 371-375, 450-454, 534, 574-575, 604, 982-985, 1036, 1055-1056, 1085, 1103-1104, 1216, 1304, 1468, 1557, 1572, 1581, 1593, 1783
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 113       4  96.46%   171, 329-331
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10115    1346  86.69%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py      +29     +17  -2.20%
TOTAL                         +29     +17  -0.13%

Results for commit: 23a3624

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@eodag-bot
Copy link
Copy Markdown
Collaborator

eodag-bot commented Mar 17, 2026

badge

Code Coverage (Windows)

Details 
Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         251      11  95.62%   104-115, 377, 644
config.py                                      305      24  92.13%   70-72, 75, 78, 81, 85, 89, 93-95, 583-585, 707-709, 728, 736, 766-771, 773
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/collection.py                              150       8  94.67%   176, 212, 215, 322, 361, 364, 382, 385
api/core.py                                    787      57  92.76%   266, 552, 600, 643, 683, 703, 744-749, 774, 858-877, 891, 897, 1050, 1055, 1154, 1193-1194, 1290-1291, 1317, 1348-1349, 1375, 1388, 1449-1450, 1467, 1481-1482, 1556-1561, 1573-1576, 1688, 2075, 2197, 2285-2286
api/provider.py                                381      35  90.81%   186, 190-191, 316-337, 405, 521, 525-526, 531-534, 545, 621-631, 821-822, 870, 877, 908-911, 947-948, 956-957
api/search_result.py                           181      19  89.50%   111, 123, 133, 154, 203, 220, 320, 375-378, 448, 453-454, 488, 502, 525-526, 532
api/product/__init__.py                         18       2  88.89%   57, 59
api/product/_assets.py                          52       4  92.31%   97, 191, 194-198
api/product/_product.py                        248      20  91.94%   189, 305-306, 324-325, 434, 463, 470, 577, 597, 620-623, 632-635, 689, 755, 767
api/product/metadata_mapping.py                791      53  93.30%   125-127, 220-225, 249, 307-308, 396, 417, 469-470, 507, 528-531, 554, 566-567, 608, 631, 661-666, 731-736, 748, 756, 988, 1163, 1172-1176, 1193-1198, 1331, 1354, 1363, 1385, 1390, 1442, 1514, 1535, 1561, 1575, 1600, 1646, 1715, 1790
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     29       0  100.00%
api/product/drivers/generic.py                  11       0  100.00%
api/product/drivers/sentinel1.py                33       0  100.00%
api/product/drivers/sentinel2.py                33       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       4  81.82%   48, 55, 68-69
plugins/manager.py                             173      16  90.75%   102-107, 179, 201, 219-220, 232, 271-272, 372-375, 387-388
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101       8  92.08%   178-180, 228-229, 255-257
plugins/apis/usgs.py                           182      25  86.26%   157, 263, 297, 339-341, 346, 374-375, 380, 410-417, 428-433, 455-461
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 288-289, 339-343
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       54      39  27.78%   50-55, 68-115
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                63      25  60.32%   62-65, 72-75, 81, 85, 89, 100-116, 131-157
plugins/crunch/filter_property.py               30       5  83.33%   56-61, 64-65
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        401      76  81.05%   270, 304, 353-356, 386-387, 395-399, 479-482, 522-524, 528, 559-560, 566-570, 601, 666-674, 738-833, 845-850, 888, 914, 959-961, 1013
plugins/download/base.py                       280      37  86.79%   133, 162, 229-231, 309-310, 368-369, 411, 415-426, 440, 517-521, 551, 586-587, 612-621, 680, 701, 723, 731, 765
plugins/download/http.py                       593      94  84.15%   234, 276-279, 341-344, 347, 354-359, 390-392, 409, 424, 484, 519, 533, 547, 557-561, 577-582, 593, 612, 649-652, 673, 683, 690, 855, 887, 917-926, 962, 987-988, 1007-1012, 1021, 1036-1038, 1042, 1045, 1060-1061, 1080-1095, 1163, 1175, 1226, 1232-1244, 1247-1249, 1289-1290, 1302, 1312, 1362-1363, 1393, 1413, 1454-1456, 1510-1511
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         196      18  90.82%   109, 113, 137-143, 200-203, 296, 317, 442, 492, 525-528, 537
plugins/search/build_search_result.py          504      83  83.53%   259-260, 296, 300, 320, 550-561, 576-578, 701, 725, 727, 794, 802-806, 827, 837, 863, 908, 932, 960, 978-993, 1043, 1068, 1071, 1075, 1084, 1090, 1128-1149, 1190, 1217-1218, 1227-1236, 1300, 1315, 1321, 1340-1349, 1470-1471, 1515, 1524-1526, 1583, 1631-1641
plugins/search/cop_marine.py                   268      56  79.10%   57, 65-67, 77-78, 83, 88-89, 105, 107, 110, 176-177, 220, 236, 242, 246, 250, 261, 272-273, 281, 313-316, 322, 343, 347, 351, 355, 359-363, 369-372, 375-392, 409-412, 465-469, 474, 486
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 247-288
plugins/search/qssearch.py                     823      94  88.58%   415-416, 533-534, 557-558, 570-574, 789-795, 853, 949, 956, 1027, 1048, 1051-1052, 1070, 1079-1080, 1107, 1179, 1188, 1193-1210, 1219, 1234, 1243-1246, 1256, 1278, 1367, 1390, 1463-1464, 1470, 1560, 1667-1671, 1737, 1740, 1744-1745, 1766-1769, 1781, 1803-1815, 1823, 1858-1860, 1883-1889, 1896, 1950, 1973, 1978-1979, 1994, 2000, 2010, 2100, 2104, 2115, 2139, 2152, 2160-2170, 2208-2212
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            84      18  78.57%   99-127, 166-169, 182, 224
resources/__init__.py                            0       0  100.00%
resources/shp/__init__.py                        0       0  100.00%
types/__init__.py                              167      43  74.25%   58, 62, 71-75, 86-98, 126-128, 135-140, 216, 219, 257, 267-283, 288, 290, 312, 317, 325, 335
types/bbox.py                                   39      19  51.28%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                          10       0  100.00%
types/queryables.py                            112       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/stac_extensions.py                       114       1  99.12%   282
types/stac_metadata.py                         116      16  86.21%   92, 113-114, 147, 177-191, 200-207
utils/__init__.py                              608      43  92.93%   71, 206, 237-238, 247-273, 276, 291, 371-375, 450-454, 534, 574-575, 604, 982-985, 1036, 1055-1056, 1085, 1103-1104, 1216, 1304, 1468, 1557, 1572, 1581, 1593, 1783
utils/cache.py                                  22       0  100.00%
utils/dates.py                                 113       4  96.46%   171, 329-331
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       0  100.00%
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10115    1348  86.67%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py      +29     +17  -2.20%
TOTAL                         +29     +17  -0.13%

Results for commit: 23a3624

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@jlahovnik jlahovnik marked this pull request as ready for review March 18, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@jlahovnik jlahovnik

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jlahovnik @eodag-bot