Skip to content

Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models

Notifications You must be signed in to change notification settings

grammarly/pillars-of-gec

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 

Repository files navigation

pillars-of-gec

This repository provides code, state-of-the art predictions and links to the pretrained Grammatical Error Correction models for "Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models" paper which was accepted for publication at BEA-2024 (19th Workshop on Innovative Use of NLP for Building Educational Applications; co-located with NAACL 2024).

Structure

Scripts directory contain required code to reproduce some of the baselines and build ensembles.
Data directory contain single systems and ensembles outputs on 3 main GEC benchmarks.
Table bellow contain single system scores and links to trained models available for download.

Pretrained models and results

Model name CoNNL-2014 (test) BEA-2019 (dev) BEA-2019 (test)
Precision Recall F05 Precision Recall F05 Precision Recall F05
CTC-copy [repo] 72.6 47.0 65.5 58.2 38.0 52.7 71.7 59.9 69.0
GECToR-2024 [link] 75.0 44.7 66.0 64.6 37.2 56.3 77.7 59.0 73.1
EditScorer [repo] 78.5 39.4 65.5 67.3 36.1 57.4 81.0 56.1 74.4
T5-11B [link] 70.9 56.5 67.5 60.9 51.1 58.6 73.2 71.2 72.8
UL2-20B [link] 73.8 50.4 67.5 60.5 48.6 57.7 75.2 70.0 74.1
Chat-LLaMa-2-7B-FT [link] 75.5 46.8 67.2 58.3 46.0 55.3 72.3 67.4 71.2
Chat-LLaMa-2-13B-FT [link] 77.2 45.6 67.9 59.8 46.1 56.4 74.6 67.8 73.1
Majority-voting ensemble (best 7) 83.7 45.7 71.8 71.7 42.2 62.9 87.3 64.1 81.4
MAJORITY-VOTING ✚[ majority-voting(best 7), GRECO-rank-w(best 7), GPT-4-rank-a(clust 3) ] 83.9 47.5 72.8 70.6 43.5 62.8 86.1 65.6 81.1

Evaluation

There are 3 evaluation sets that we are using for GEC:

  1. CoNLL-2014 (nucle14-2a, m2 file is available; m2scorer is official scorer)
  2. BEA19-dev (bea-dev, m2 file is available; errant is official scorer)
  3. BEA19-test (bea-test, m2 file is NOT available; score can be got only through codelab sumbission)

Examples of evaluation

Evalsest directory: data/evaluation_sets.

  1. Example of evaluation with Errant
ERRANT_SCORER=path_to_errant_scorer_directory
INPUT_FILE=data/evaluation_sets/bea-dev.txt
M2_FILE=data/evaluation_sets/bea-dev.m2
PRED_FILE=YOUR_PRED_FILE.txt
TMP_FILE=YOUR_TMP_FILE.m2


python $ERRANT_SCORER/parallel_to_m2.py -orig $INPUT_FILE -cor $PRED_FILE -out $TMP_FILE
python $ERRANT_SCORER/compare_m2.py -hyp $TMP_FILE -ref $M2_FILE >> {{result}}
  1. Example of evaluation with m2scorer
M2_SCORER=path_to_m2scorer
M2_FILE=data/evaluation_sets/nucle14-2a.m2
PRED_FILE=YOUR_PRED_FILE.txt
$M2_SCORER $PRED_FILE $M2_FILE >> {{reslut}}

Citation

[to be updated once proceedings are published]

@misc{omelianchuk2024pillars,
      title={Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models}, 
      author={Kostiantyn Omelianchuk and Andrii Liubonko and Oleksandr Skurzhanskyi and Artem Chernodub and Oleksandr Korniienko and Igor Samokhin},
      year={2024},
      eprint={2404.14914},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

About

Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published