Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(plugins): raise error on failed empty stream_zip #1475

Merged
merged 3 commits into from
Jan 21, 2025

Conversation

jlahovnik
Copy link
Collaborator

@jlahovnik jlahovnik commented Jan 15, 2025

There were situations where in server mode the response at the download was empty but the response code was 200.
This problem occurred when a zip was created and streamed and the download of the first file failed. In that case, the download was stopped, so the result was empty, but the stream_zip function was hiding the raised error from the user. This problem is now fixed, an error will be returned if the download fails.

@jlahovnik jlahovnik self-assigned this Jan 15, 2025
Copy link
Contributor

github-actions bot commented Jan 15, 2025

Test Results

    4 files  ± 0      4 suites  ±0   6m 24s ⏱️ -5s
  591 tests + 4    588 ✅ + 4   3 💤 ±0  0 ❌ ±0 
2 364 runs  +16  2 266 ✅ +16  98 💤 ±0  0 ❌ ±0 

Results for commit 32180eb. ± Comparison against base commit 4233b05.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Jan 15, 2025

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-737, 839-890, 894
config.py                                      430      27  93.72%   80-82, 91, 99, 103-105, 176, 187, 692-694, 807-810, 853-854, 863-864, 969, 1032-1037, 1039
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    783      74  90.55%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   82, 91, 98, 112
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        188      20  89.36%   70-72, 238-239, 314, 343, 400, 414-417, 430, 454-457, 500-506
api/product/metadata_mapping.py                695      76  89.06%   119-121, 218, 250-251, 297-298, 308-320, 322, 333, 339-351, 398-399, 436, 457-460, 483, 491-492, 578-579, 603-604, 610-613, 628-629, 778, 824, 977, 986-990, 1007-1012, 1139, 1153-1173, 1193, 1198, 1327, 1341, 1366, 1412, 1464, 1487-1488, 1504-1508, 1524, 1532
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      14  91.86%   106-111, 161, 199, 221, 225, 249, 389-392, 404-405
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   171-173, 220-221, 247-249, 302-303
plugins/apis/usgs.py                           180      26  85.56%   156, 258, 292, 327-329, 334, 360-361, 366, 396-403, 414-419, 441-447, 478
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   45, 58
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   43-45, 49-52
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                100      16  84.00%   141, 170, 172, 207-220, 276-280
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        490     162  66.94%   269, 282, 349-352, 366-370, 412-414, 418, 450-451, 457-461, 491, 523, 527, 534, 564-572, 576, 608-616, 627-629, 660-734, 752-810, 821-826, 838-851, 876, 891-893, 896, 906-914, 922-935, 945-967, 974-986, 1024, 1050, 1095-1097, 1317
plugins/download/base.py                       253      42  83.40%   127, 155, 235-238, 291-292, 340-344, 350-351, 393, 396-410, 422, 426, 490-494, 524-525, 550-558, 560-564, 607, 629, 651, 659
plugins/download/creodias_s3.py                 25       9  64.00%   55-69
plugins/download/http.py                       529      95  82.04%   227, 323-326, 329, 336-341, 359-374, 391, 403, 451, 458-464, 482, 496, 510, 518-520, 536-541, 552, 570, 606-609, 638, 642, 662, 739, 758-769, 777-782, 792-809, 827, 857-866, 902, 927-928, 947-952, 958, 961, 978, 981, 996-997, 1032, 1094, 1109, 1168-1169, 1175, 1185, 1221, 1257, 1277, 1312-1314
plugins/download/s3rest.py                     116      24  79.31%   119, 153, 160, 195, 222-229, 232-234, 238, 249-255, 263-264, 267-271, 294, 315-318
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         145      11  92.41%   101, 105, 129-135, 275, 295, 428
plugins/search/build_search_result.py          357      49  86.27%   237, 265-266, 302, 305, 379-382, 471-488, 516, 565, 567, 592-599, 631-633, 637, 659, 694, 745, 774-775, 790-805, 848, 873, 876, 879, 887, 999-1000, 1006, 1017, 1075, 1126
plugins/search/cop_marine.py                   244      47  80.74%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 228, 234, 238, 242, 255, 266-267, 275, 303, 307, 322, 326, 330, 334, 338-342, 348-351, 354-368, 385, 434-438, 443, 455
plugins/search/creodias_s3.py                   57       3  94.74%   59, 77, 116
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     733      89  87.86%   449, 507, 521, 525-531, 559-563, 676-688, 733-736, 807-808, 856, 875, 882, 894, 951, 972, 975-976, 985-986, 995-996, 1005-1006, 1033, 1108-1113, 1117-1126, 1160, 1182, 1259, 1349, 1434-1435, 1445, 1522-1526, 1588, 1591, 1597-1598, 1619, 1647-1659, 1666, 1698-1700, 1710-1716, 1746, 1769, 1774-1775, 1790, 1806, 1891-1894, 1899-1902, 1911, 1941-1945, 1951
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   260      66  74.62%   258, 266, 284-301, 316-352, 446, 483-522, 705, 712-765
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 192      24  87.50%   86, 109-111, 285-290, 318, 514-516, 533-538, 567, 569, 573-574, 578-579
rest/stac.py                                   319      63  80.25%   308, 330, 382-385, 412-439, 470-472, 495, 527-528, 610-650, 672-688, 780-784, 791, 845-846, 907, 997-999
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180       6  96.67%   225-229, 282, 285, 353, 385
rest/types/queryables.py                        57       5  91.23%   93-98, 162
rest/types/stac_search.py                      125       7  94.40%   128, 174, 189-191, 199, 203
rest/utils/__init__.py                          94      12  87.23%   101-102, 121-123, 175, 185-199
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              133      42  68.42%   63, 76-80, 91-103, 131-133, 140-145, 186, 205, 227, 237-253, 258, 260, 282, 287, 295, 305
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 81      16  80.25%   129-132, 136-143, 155-161, 174-176
utils/__init__.py                              533      48  90.99%   81, 197-198, 207-234, 237, 252, 332-336, 409-413, 432-434, 447-461, 540, 555, 595-596, 625, 1000-1003, 1011-1012, 1050-1051, 1098-1099, 1233
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9995    1635  83.64%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +1      -9  +1.74%
TOTAL                          +1      -9  +0.09%

Results for commit: 32180eb

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Jan 15, 2025

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-737, 839-890, 894
config.py                                      430      28  93.49%   80-82, 91, 99, 103-105, 176, 187, 692-694, 807-810, 853-854, 863-864, 969, 1000, 1032-1037, 1039
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    783      74  90.55%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   82, 91, 98, 112
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        188      20  89.36%   70-72, 238-239, 314, 343, 400, 414-417, 430, 454-457, 500-506
api/product/metadata_mapping.py                695      77  88.92%   119-121, 218, 250-251, 297-298, 308-320, 322, 333, 339-351, 398-399, 436, 457-460, 483, 491-492, 578-579, 603-604, 610-613, 628-629, 778, 824, 977, 986-990, 1007-1012, 1139, 1153-1173, 1193, 1198, 1327, 1341, 1366, 1412, 1464, 1487-1488, 1491, 1504-1508, 1524, 1532
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      14  91.86%   106-111, 161, 199, 221, 225, 249, 389-392, 404-405
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   171-173, 220-221, 247-249, 302-303
plugins/apis/usgs.py                           180      26  85.56%   156, 258, 292, 327-329, 334, 360-361, 366, 396-403, 414-419, 441-447, 478
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   45, 58
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   43-45, 49-52
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                100      16  84.00%   141, 170, 172, 207-220, 276-280
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        490     162  66.94%   269, 282, 349-352, 366-370, 412-414, 418, 450-451, 457-461, 491, 523, 527, 534, 564-572, 576, 608-616, 627-629, 660-734, 752-810, 821-826, 838-851, 876, 891-893, 896, 906-914, 922-935, 945-967, 974-986, 1024, 1050, 1095-1097, 1317
plugins/download/base.py                       253      44  82.61%   127, 155, 222-224, 235-238, 291-292, 340-344, 350-351, 393, 396-410, 422, 426, 490-494, 524-525, 550-558, 560-564, 607, 629, 651, 659
plugins/download/creodias_s3.py                 25       9  64.00%   55-69
plugins/download/http.py                       529      95  82.04%   227, 323-326, 329, 336-341, 359-374, 391, 403, 451, 458-464, 482, 496, 510, 518-520, 536-541, 552, 570, 606-609, 638, 642, 662, 739, 758-769, 777-782, 792-809, 827, 857-866, 902, 927-928, 947-952, 958, 961, 978, 981, 996-997, 1032, 1094, 1109, 1168-1169, 1175, 1185, 1221, 1257, 1277, 1312-1314
plugins/download/s3rest.py                     116      24  79.31%   119, 153, 160, 195, 222-229, 232-234, 238, 249-255, 263-264, 267-271, 294, 315-318
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         145      11  92.41%   101, 105, 129-135, 275, 295, 428
plugins/search/build_search_result.py          357      50  85.99%   237, 265-266, 302, 305, 379-382, 471-488, 516, 565, 567, 592-599, 631-633, 637, 659, 694, 745, 759, 774-775, 790-805, 848, 873, 876, 879, 887, 999-1000, 1006, 1017, 1075, 1126
plugins/search/cop_marine.py                   244      47  80.74%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 228, 234, 238, 242, 255, 266-267, 275, 303, 307, 322, 326, 330, 334, 338-342, 348-351, 354-368, 385, 434-438, 443, 455
plugins/search/creodias_s3.py                   57       3  94.74%   59, 77, 116
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     733     125  82.95%   449, 507, 521, 525-531, 559-563, 676-688, 733-736, 807-808, 856, 875, 882, 894, 951, 972, 975-976, 985-986, 995-996, 1005-1006, 1033, 1108-1113, 1117-1126, 1160, 1182, 1259, 1349, 1434-1435, 1445, 1522-1526, 1588, 1591, 1597-1598, 1619, 1647-1659, 1666, 1698-1700, 1710-1716, 1746, 1769, 1774-1775, 1790, 1806, 1874-1984
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   260     153  41.15%   157, 159, 161, 164-165, 179-189, 198-199, 205, 208, 249-303, 316-352, 383-420, 435-451, 467-476, 483-522, 539, 581-666, 705, 712-765
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 192     192  0.00%    18-592
rest/stac.py                                   319      68  78.68%   242, 308, 330, 382-385, 412-439, 470-472, 495, 527-528, 610-650, 672-688, 715, 780-784, 791, 845-846, 852, 907, 945, 978, 997-999
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180      20  88.89%   225-229, 262-264, 282, 285, 291, 295, 353, 371-386
rest/types/queryables.py                        57      13  77.19%   50-51, 58-59, 66-67, 93-98, 107-108, 162
rest/types/stac_search.py                      125      11  91.20%   126-128, 174, 189-191, 199, 203, 251, 254
rest/utils/__init__.py                          94      30  68.09%   72-78, 98, 101-102, 121-123, 136, 143, 168-176, 183-204
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              133      44  66.92%   63, 67, 76-80, 91-103, 131-133, 140-145, 186, 200, 205, 227, 237-253, 258, 260, 282, 287, 295, 305
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 81      16  80.25%   129-132, 136-143, 155-161, 174-176
utils/__init__.py                              533      48  90.99%   81, 197-198, 207-234, 237, 252, 332-336, 409-413, 432-434, 447-461, 540, 555, 595-596, 625, 1000-1003, 1011-1012, 1050-1051, 1098-1099, 1233
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9995    2070  79.29%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +1      -9  +1.74%
TOTAL                          +1      -9  +0.09%

Results for commit: 32180eb

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@jlahovnik jlahovnik requested a review from sbrunato January 15, 2025 14:41
@alambare alambare requested review from anesson-cs and removed request for sbrunato January 15, 2025 15:54
@alambare alambare added bug Something isn't working dedl DEDL related labels Jan 15, 2025
@jlahovnik jlahovnik marked this pull request as draft January 17, 2025 10:05
@jlahovnik jlahovnik marked this pull request as ready for review January 17, 2025 13:46
Copy link
Collaborator

@anesson-cs anesson-cs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change the docstrings of the test.

@anesson-cs anesson-cs marked this pull request as draft January 20, 2025 08:45
@jlahovnik jlahovnik marked this pull request as ready for review January 20, 2025 09:00
@jlahovnik
Copy link
Collaborator Author

Please change the docstrings of the test.

changed

@sbrunato sbrunato changed the title fix: avoid empty response with code 200 with streamed zip fix(plugins): raise error on failed empty stream_zip Jan 21, 2025
@sbrunato sbrunato merged commit f82f016 into develop Jan 21, 2025
10 checks passed
@sbrunato sbrunato deleted the fix-code-200-empty-response branch January 21, 2025 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dedl DEDL related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants