Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add swiss legal evals as new community tasks #389

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

Add swiss legal evals as new community tasks #389

Changes from 6 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e2a27a7
Add swiss legal evals as new community tasks
JoelNiklaus Nov 11, 2024
aa409c8
Removed nltk and numpy dependencies.
JoelNiklaus Nov 12, 2024
a8ee2a5
Added short dataset descriptions.
JoelNiklaus Nov 12, 2024
8f68844
Merge branch 'main' into add_swiss_legal_evals
clefourrier Nov 13, 2024
c7f7038
Removed open judge models and added COMET and METEOR.
JoelNiklaus Nov 13, 2024
0ca5af6
Merge branch 'main' into add_swiss_legal_evals
clefourrier Nov 19, 2024
1d51a01
Merge branch 'main' into add_swiss_legal_evals
NathanHB Nov 19, 2024
5d41ce0
Ran pre-commit hooks.
JoelNiklaus Nov 20, 2024
8194125
Changed prompt template.
JoelNiklaus Nov 20, 2024
c58ae44
Added legal translation specific judge prompt.
JoelNiklaus Nov 21, 2024
ff3705f
Improved judge prompt.
JoelNiklaus Nov 21, 2024
091ec11
Changed metric selection.
JoelNiklaus Nov 21, 2024
5a47956
Made generation_size dependent on the config.
JoelNiklaus Nov 22, 2024
6bf7fa2
Fixed error in config.
JoelNiklaus Nov 22, 2024
6cf1c2a
Fixed error in config.
JoelNiklaus Nov 22, 2024
b548801
Added support for multiple devices.
JoelNiklaus Nov 22, 2024
ee2a83c
Fixed some bugs for evaluation on GPUs.
JoelNiklaus Nov 25, 2024
36b7e94
Added batch inference for heavy metrics and multiplied each score by …
JoelNiklaus Nov 26, 2024
5ba218f
Added few shot examples and did some refactoring.
JoelNiklaus Nov 26, 2024
8490841
Merge branch 'main' into add_swiss_legal_evals
JoelNiklaus Nov 26, 2024
576b847
Switched to an own judge class.
JoelNiklaus Nov 26, 2024
41bb59a
Fixed issue with judge metric not showing up in results.
JoelNiklaus Nov 26, 2024
d82cd91
Fixed issue with evaluation on GPUs.
JoelNiklaus Nov 27, 2024
1b13d9f
Speed up metric computation on GPUs.
JoelNiklaus Nov 27, 2024
df0f3f0
Added more logging.
JoelNiklaus Nov 27, 2024
980c257
Switched to sample level scores for faster evaluation.
JoelNiklaus Nov 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading