Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Process-supervised RM Trainer #2127

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft

Process-supervised RM Trainer #2127

wants to merge 46 commits into from

Commits on Sep 26, 2024

  1. initial skeleton

    @gaetanlop
    gaetanlop committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    357a8c6 View commit details
    Browse the repository at this point in the history

  2. tokenize fn

    @gaetanlop
    gaetanlop committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    841f7a1 View commit details
    Browse the repository at this point in the history

  3. adding bos and eos to tokenization fn

    @gaetanlop
    gaetanlop committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    641e899 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. prmtrainer

    @gaetanlop
    gaetanlop committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    106bc0e View commit details
    Browse the repository at this point in the history

  2. fixing small typo in tokenize

    @gaetanlop
    gaetanlop committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0163dcc View commit details
    Browse the repository at this point in the history

  3. typo in input_ids and labels construction

    @gaetanlop
    gaetanlop committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c2720d7 View commit details
    Browse the repository at this point in the history

  4. numpy dimension

    @gaetanlop
    gaetanlop committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    5034083 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. introduce the stepwise reward trainer

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    8818b6a View commit details
    Browse the repository at this point in the history

  2. update markdown files

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    b777d1c View commit details
    Browse the repository at this point in the history

  3. let user decide post step separator in config

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    afa9e0a View commit details
    Browse the repository at this point in the history

  4. doc post_step_separator

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    2dd752d View commit details
    Browse the repository at this point in the history

  5. do not add post step_tokens to last step of the reasoning process

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    613d838 View commit details
    Browse the repository at this point in the history

  6. renaming prm to stepwisereward

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    b96ef4d View commit details
    Browse the repository at this point in the history

  7. formatting

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    161f5de View commit details
    Browse the repository at this point in the history

  8. fix tokenize kwargs

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    93e6652 View commit details
    Browse the repository at this point in the history

  9. adapt test to the new post_token args

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    3ec4ebe View commit details
    Browse the repository at this point in the history

  10. adding example script

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    1461a61 View commit details
    Browse the repository at this point in the history

  11. fix small typo

    @gaetanlop
    gaetanlop committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    8c4ac31 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. add create_model_card and renaming

    @gaetanlop
    gaetanlop committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8b3fa52 View commit details
    Browse the repository at this point in the history

  2. fixing booleans

    @gaetanlop
    gaetanlop committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8e4e159 View commit details
    Browse the repository at this point in the history

  3. Adding the new stepwise_preference instead of placeholders for datasets

    @gaetanlop
    gaetanlop committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c60bc40 View commit details
    Browse the repository at this point in the history

  4. formatting

    @gaetanlop
    gaetanlop committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    614fb4e View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into prmtrainer

    @qgallouedec
    qgallouedec authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c582464 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Merge branch 'main' into prmtrainer

    @kashif
    kashif authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    424af34 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Update docs/source/_toctree.yml 

    Co-authored-by: Quentin Gallouédec <[email protected]>
    @gaetanlop @qgallouedec
    gaetanlop and qgallouedec authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    b00e32b View commit details
    Browse the repository at this point in the history

  2. Update examples/scripts/stepwise_reward_modeling.py 

    Co-authored-by: Quentin Gallouédec <[email protected]>
    @gaetanlop @qgallouedec
    gaetanlop and qgallouedec authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    d5f780a View commit details
    Browse the repository at this point in the history

  3. Update trl/trainer/stepwise_reward_trainer.py 

    Co-authored-by: Quentin Gallouédec <[email protected]>
    @gaetanlop @qgallouedec
    gaetanlop and qgallouedec authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    f02056a View commit details
    Browse the repository at this point in the history

  4. Update trl/trainer/stepwise_reward_trainer.py 

    Co-authored-by: Quentin Gallouédec <[email protected]>
    @gaetanlop @qgallouedec
    gaetanlop and qgallouedec authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    3ac323f View commit details
    Browse the repository at this point in the history

  5. update push to hub 

    Co-authored-by: Quentin Gallouédec <[email protected]>
    @gaetanlop @qgallouedec
    gaetanlop and qgallouedec authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    436dfd7 View commit details
    Browse the repository at this point in the history

  6. step_separator can't be None 

    Co-authored-by: Quentin Gallouédec <[email protected]>
    @gaetanlop @qgallouedec
    gaetanlop and qgallouedec authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    f4e6d4e View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into prmtrainer

    @gaetanlop
    gaetanlop authored Oct 12, 2024
    Configuration menu
    Copy the full SHA
    6947aef View commit details
    Browse the repository at this point in the history

  8. fix suggested typos

    @gaetanlop
    gaetanlop committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    e0c0648 View commit details
    Browse the repository at this point in the history

  9. add citation

    @gaetanlop
    gaetanlop committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    35de0ee View commit details
    Browse the repository at this point in the history

  10. reformat doc

    @gaetanlop
    gaetanlop committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    c3eb08e View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. reordering init

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    898f621 View commit details
    Browse the repository at this point in the history

  2. push to hub prm800k

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    3a488e0 View commit details
    Browse the repository at this point in the history

  3. changing dataset in example

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    a03aed8 View commit details
    Browse the repository at this point in the history

  4. change dataset format to align with the sky is blue example

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    e77eee2 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into prmtrainer

    @gaetanlop
    gaetanlop authored Oct 13, 2024
    Configuration menu
    Copy the full SHA
    6c62c69 View commit details
    Browse the repository at this point in the history

  6. fix tokenization column names

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    e8e93f1 View commit details
    Browse the repository at this point in the history

  7. fix num labels in openai example

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    2059c51 View commit details
    Browse the repository at this point in the history

  8. add support for conversational dataset

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    701241b View commit details
    Browse the repository at this point in the history

  9. remove training whitespace

    @gaetanlop
    gaetanlop committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    6bb467b View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Merge branch 'main' into prmtrainer

    @gaetanlop
    gaetanlop authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    6b2bd97 View commit details
    Browse the repository at this point in the history

  2. replace tokenizer with processing class

    @gaetanlop
    gaetanlop committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    2030a83 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'prmtrainer' of https://github.com/gaetanlop/trl into pr… 

    …mtrainer
    @gaetanlop
    gaetanlop committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    66baada View commit details
    Browse the repository at this point in the history