fix: handle exceptions occuring on failed index write

[alambare]

• log an error in case of a product type failure
• (whoosh) clean up writer for failed document add when exceptions were absorbed (cf: Fix #375: Clean up writer for failed document add when exceptions wer… whoosh-community/whoosh#543)

[github-actions]

Test Results

    4 files  ±0      4 suites  ±0   6m 12s ⏱️ ±0s
  582 tests ±0    579 ✅ ±0   3 💤 ±0  0 ❌ ±0 
2 328 runs  ±0  2 230 ✅ ±0  98 💤 ±0  0 ❌ ±0 

Results for commit e98789a. ± Comparison against base commit 132ff00.

♻️ This comment has been updated with latest results.

[github-actions]

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-735, 837-888, 892
config.py                                      428      27  93.69%   83-85, 94, 102, 106-108, 179, 190, 690-692, 805-808, 851-852, 861-862, 967, 1030-1035, 1037
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    782      74  90.54%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                700      76  89.14%   131-133, 230, 262-263, 309-310, 320-332, 334, 345, 351-363, 410-411, 448, 469-472, 495, 503-504, 590-591, 615-616, 622-625, 640-641, 790, 836, 989, 998-1002, 1019-1024, 1151, 1165-1185, 1205, 1210, 1339, 1353, 1424, 1476, 1499-1500, 1516-1520, 1536, 1544, 1563
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   170-172, 219-220, 246-248, 301-302
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       258      49  81.01%   136, 164, 300-301, 355-359, 365-366, 408, 411-425, 437, 441, 505-509, 539-540, 548-565, 572-580, 582-586, 629, 651, 673, 681
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       545     106  80.55%   237, 333-336, 339, 346-351, 369-384, 401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 939, 964-965, 984-989, 995, 998, 1015, 1032-1033, 1063-1064, 1071, 1133, 1148, 1207-1208, 1214, 1224, 1260, 1296, 1316, 1351-1353
plugins/download/s3rest.py                     116      24  79.31%   118, 152, 159, 194, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         146      11  92.47%   100, 104, 128-134, 274, 294, 433
plugins/search/build_search_result.py          341      49  85.63%   249, 277-278, 314, 317, 386-389, 478-495, 523, 558, 560, 585-592, 622-624, 628, 650, 685, 736, 765-766, 781-796, 839, 864, 867, 870, 878, 997-998, 1004, 1015, 1073, 1124
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   59, 77, 111
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     732      89  87.84%   450, 508, 522, 526-532, 560-564, 677-689, 734-737, 808-809, 857, 876, 883, 895, 952, 973, 976-977, 986-987, 996-997, 1006-1007, 1034, 1105-1110, 1114-1123, 1157, 1179, 1256, 1346, 1431-1432, 1442, 1519-1523, 1585, 1588, 1594-1595, 1616, 1644-1656, 1663, 1695-1697, 1707-1713, 1743, 1766, 1771-1772, 1787, 1803, 1888-1891, 1896-1899, 1908, 1938-1942, 1948
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   257      63  75.49%   258, 266, 284-301, 316-352, 446, 483-522, 705, 712-760
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 192      24  87.50%   94, 117-119, 293-298, 326, 522-524, 541-546, 575, 577, 581-582, 586-587
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180       6  96.67%   225-229, 282, 285, 353, 385
rest/types/queryables.py                        57       5  91.23%   93-98, 162
rest/types/stac_search.py                      125       7  94.40%   138, 184, 199-201, 209, 213
rest/utils/__init__.py                          94      12  87.23%   111-112, 131-133, 185, 195-209
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              128      42  67.19%   66, 79-83, 94-106, 134-136, 143-148, 189, 208, 230, 240-256, 261, 263, 285, 290, 298, 308
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 82      16  80.49%   131-134, 138-145, 157-163, 176-178
utils/__init__.py                              520      47  90.96%   85, 194-195, 204-231, 234, 249, 329-333, 406-410, 429-431, 444-458, 537, 552, 592-593, 957-960, 968-969, 1007-1008, 1055-1056, 1190
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9959    1658  83.35%

Diff against develop

Filename           Stmts    Miss  Cover
---------------  -------  ------  -------
api/core.py           +3      +2  -0.22%
types/whoosh.py      +67     +16  -19.51%
TOTAL                +70     +18  -0.07%

Results for commit: e98789a

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

[github-actions]

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         323      64  80.19%   62-63, 87, 674-735, 837-888, 892
config.py                                      428      28  93.46%   83-85, 94, 102, 106-108, 179, 190, 690-692, 805-808, 851-852, 861-862, 967, 998, 1030-1035, 1037
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    782      74  90.54%   317-318, 368, 658, 702-705, 743, 787, 821, 866-871, 897, 988, 1056, 1194, 1279-1291, 1327, 1329, 1357, 1361-1372, 1385-1391, 1474-1477, 1510-1530, 1582, 1599-1603, 1615-1618, 1954, 1978-1984, 2235, 2239-2243, 2252-2254, 2298-2299, 2328-2329
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                700      77  89.00%   131-133, 230, 262-263, 309-310, 320-332, 334, 345, 351-363, 410-411, 448, 469-472, 495, 503-504, 590-591, 615-616, 622-625, 640-641, 790, 836, 989, 998-1002, 1019-1024, 1151, 1165-1185, 1205, 1210, 1339, 1353, 1424, 1476, 1499-1500, 1503, 1516-1520, 1536, 1544, 1563
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   170-172, 219-220, 246-248, 301-302
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       258      51  80.23%   136, 164, 231-233, 300-301, 355-359, 365-366, 408, 411-425, 437, 441, 505-509, 539-540, 548-565, 572-580, 582-586, 629, 651, 673, 681
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       545     107  80.37%   237, 333-336, 339, 346-351, 369-384, 401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 939, 964-965, 984-989, 995, 998, 1015, 1032-1033, 1046, 1063-1064, 1071, 1133, 1148, 1207-1208, 1214, 1224, 1260, 1296, 1316, 1351-1353
plugins/download/s3rest.py                     116      24  79.31%   118, 152, 159, 194, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         146      11  92.47%   100, 104, 128-134, 274, 294, 433
plugins/search/build_search_result.py          341      50  85.34%   249, 277-278, 314, 317, 386-389, 478-495, 523, 558, 560, 585-592, 622-624, 628, 650, 685, 736, 750, 765-766, 781-796, 839, 864, 867, 870, 878, 997-998, 1004, 1015, 1073, 1124
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   59, 77, 111
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     732     125  82.92%   450, 508, 522, 526-532, 560-564, 677-689, 734-737, 808-809, 857, 876, 883, 895, 952, 973, 976-977, 986-987, 996-997, 1006-1007, 1034, 1105-1110, 1114-1123, 1157, 1179, 1256, 1346, 1431-1432, 1442, 1519-1523, 1585, 1588, 1594-1595, 1616, 1644-1656, 1663, 1695-1697, 1707-1713, 1743, 1766, 1771-1772, 1787, 1803, 1871-1981
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   257     150  41.63%   157, 159, 161, 164-165, 179-189, 198-199, 205, 208, 249-303, 316-352, 383-420, 435-451, 467-476, 483-522, 539, 581-666, 705, 712-760
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 192     192  0.00%    18-600
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180      20  88.89%   225-229, 262-264, 282, 285, 291, 295, 353, 371-386
rest/types/queryables.py                        57      13  77.19%   50-51, 58-59, 66-67, 93-98, 107-108, 162
rest/types/stac_search.py                      125      11  91.20%   136-138, 184, 199-201, 209, 213, 261, 264
rest/utils/__init__.py                          94      30  68.09%   82-88, 108, 111-112, 131-133, 146, 153, 178-186, 193-214
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              128      44  65.62%   66, 70, 79-83, 94-106, 134-136, 143-148, 189, 203, 208, 230, 240-256, 261, 263, 285, 290, 298, 308
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            108       1  99.07%   63
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 82      16  80.49%   131-134, 138-145, 157-163, 176-178
utils/__init__.py                              520      47  90.96%   85, 194-195, 204-231, 234, 249, 329-333, 406-410, 429-431, 444-458, 537, 552, 592-593, 957-960, 968-969, 1007-1008, 1055-1056, 1190
utils/exceptions.py                             46       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9959    2094  78.97%

Diff against develop

Filename           Stmts    Miss  Cover
---------------  -------  ------  -------
api/core.py           +3      +2  -0.22%
types/whoosh.py      +67     +16  -19.51%
TOTAL                +70     +18  -0.04%

Results for commit: e98789a

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results