PPOTrainer for models

[JarcauCristian]

Does llmstudio train using the PPOTrainer from the trl package, or it can be done afterwards somehow?

[pascal-pfeiffer]

H2O LLM Studio uses a modified version of the trl trainer in the background when using the RLHF problem type

h2o-llmstudio/llm_studio/src/trl/trainer.py

Lines 1 to 18 in bf667d0

	# This file borrows large pieces from the trl library, which is licensed under
	# the Apache 2.0 license.
	# https://github.com/lvwerra/trl/blob/main/trl/trainer/ppo_trainer.py
	# Copyright 2022 The HuggingFace Team. All rights reserved.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing permissions and
	# limitations under the License.

[JarcauCristian]

Alright! Thank you very much!