You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the evaluation is only performed during the pipeline execution. However we later might want to evaluate the models produced in this pipeline on more datasets, and we shouldn't need to re-execute the entire pipeline as training takes a long time.
After having executed a pipeline, the models are stored in the model store. We should be able to use modyn client to directly launch an evaluation request against a specific model (with request including model id) and a specific evaluation dataset (with request including a description on the evaluation dataset).
The text was updated successfully, but these errors were encountered:
We now have most of the points ready? Just the client integration is missing. I'd say this isn't needed as one can simply create a pydantic model and use it instead through the cli entrypoint.
Currently the evaluation is only performed during the pipeline execution. However we later might want to evaluate the models produced in this pipeline on more datasets, and we shouldn't need to re-execute the entire pipeline as training takes a long time.
After having executed a pipeline, the models are stored in the model store. We should be able to use modyn client to directly launch an evaluation request against a specific model (with request including model id) and a specific evaluation dataset (with request including a description on the evaluation dataset).
The text was updated successfully, but these errors were encountered: