feature: More resilient evaluations

[MaxiBoether]

This PR addresses several issues:

1. When the evaluator crashes with an exception, we actually also send this to the supervisor

2. We validate whether we receive the correct number of results at the supervisor

3. We fix the validation of the correct number of evaluations at the evaluator

4. Some minor additions of retrying to connect in case the connection gets lost

Our observation of missing evaluations should now be caught at the first point (we actually retry on exception at the evaluator) but the others serve as a guarding mechanism.

[robinholzi]

Thanks, didn't see anything indicating this cannot work.

[github-actions]

^{( % to main)
( % to main)}

[codecov]

Codecov Report

Attention: Patch coverage is 78.65169% with 19 lines in your changes missing coverage. Please review.

Project coverage is 85.57%. Comparing base (a15b3c6) to head (404e0de).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
modyn/supervisor/internal/grpc_handler.py	58.06%	13 Missing ⚠️
.../internal/pipeline_executor/evaluation_executor.py	92.30%	3 Missing ⚠️
...n/supervisor/internal/triggers/datadrifttrigger.py	0.00%	2 Missing ⚠️
...evaluator/internal/grpc/evaluator_grpc_servicer.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #635      +/-   ##
==========================================
- Coverage   85.61%   85.57%   -0.04%     
==========================================
  Files         258      258              
  Lines       11332    11371      +39     
==========================================
+ Hits         9702     9731      +29     
- Misses       1630     1640      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.