Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ngram match vs weighted ngram match #52

Open
Fritz-D opened this issue Jun 21, 2024 · 1 comment
Open

ngram match vs weighted ngram match #52

Fritz-D opened this issue Jun 21, 2024 · 1 comment

Comments

@Fritz-D
Copy link

Fritz-D commented Jun 21, 2024

Expected behaviour for these two metrics would be that the weighted ngram match score with a weight of 1 for all ngrams would be equivalent to the unweighted ngram match score. This is not the case:
There are following issues:

  1. Brevity Penalty: Your 'references' object in the weighted ngram match score includes a list of references and a list of weights. As you pass this wholecloth into the closest closest_ref_length function, which then treats the list of refs as one ref and the list of weights as another, the brevity penalty degenerates.
  2. Denominator: In the unweighted score calculation, you use 'counts', i.e. the length of the hypothesis to calculate the denominator, in the weighted calculation, you use reference_counts, i.e. the length of the reference. This leads to a difference in scores.
  3. Counts: In the unweighted score calculation, you use a max counts object to compare the hypothesis to the set of references. In the weighted score calculation, you compare the hypothesis with individual reference then average later.

Consider merging the two calculations into one fixed version, wherein the default value for weights is simply 1 across the board.

@Fritz-D
Copy link
Author

Fritz-D commented Jun 21, 2024

While you're at it you might as well use the bitwise Counter and operator (&) to calculate clipped counts rather than doing it manually

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant