Final eval mtbench_branch #67
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sallyom
force-pushed
the
final-eval-mtbench
branch
from
2ee354e to
bf8e33c
Compare
Closed
sallyom
force-pushed
the
final-eval-mtbench
branch
4 times, most recently
from
e63abf7 to
b5b22cc
Compare
|
@MichaelClifford hold off on reviewing this for a bit - I'm importing defs from ilab rather than cp/pasting them here - will push that change after i test that it works |
sallyom
force-pushed
the
final-eval-mtbench
branch
2 times, most recently
from
f6b6067 to
b6ef1f9
Compare
|
@MichaelClifford this is ready for review, test running now |
MichaelClifford
approved these changes
Oct 8, 2024
sallyom
force-pushed
the
final-eval-mtbench
branch
from
b6ef1f9 to
b213b21
Compare
|
Will merge after the current run completes |
MichaelClifford
approved these changes
Oct 8, 2024
Signed-off-by: sallyom <[email protected]>
Signed-off-by: sallyom <[email protected]>
sallyom
force-pushed
the
final-eval-mtbench
branch
from
b213b21 to
dccf427
Compare
leseb
reviewed
Oct 9, 2024
| def git_clone_op( | ||
| taxonomy: dsl.Output[dsl.Dataset], | ||
| repo_branch: str, | ||
| repo_pr: Optional[int], | ||
| repo_url: Optional[str], | ||
| ): | ||
| return | ||
| return dsl.ContainerSpec( |
There was a problem hiding this comment.
Why did we need this change?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds final eval mt_bench_branch
Also, adds a helpers package - for
launch_vllm,stop_vllm-used inmt_bench_branch,mmlu_branch, andmt_bench, so need a central place for these.For mmlu_branch & mt_bench_branch, a taxonomy is required - this PR adds the taxonomy to the fake fixtures of SDG, to allow for mocking SDG with running final evaluation.