微调的时候如何接续训练（从上一个检查点继续）

[Lihwnlp]

请问如何从上一个检查点继续训练呢？我的上一个检查点是checkpoint-585，但是我加了--resume_from_checkpoint ${output_model}/checkpoint-585这个参数后，还是会从头开始训练（加或者不加都是从头开始训练），😵，
下边是我的脚本：
deepspeed --include localhost:0,1,2,3 finetune_clm_lora.py
--model_name_or_path /HOME/
--train_files /HOME/
--validation_files /HOME/
--output_dir /HOME/
--per_device_train_batch_size 16
--per_device_eval_batch_size 16
--do_train
--do_eval
--use_fast_tokenizer false
--output_dir ${output_model}
--evaluation_strategy steps
--max_eval_samples 800
--learning_rate 2.0e-4
--gradient_accumulation_steps 8
--num_train_epochs 5
--warmup_steps 0
--load_in_bits 4
--lora_r 8
--lora_alpha 32
--target_modules q_proj,k_proj,v_proj,o_proj,down_proj,gate_proj,up_proj
--logging_dir ${output_model}/logs
--logging_strategy steps
--logging_steps 500
--save_strategy epoch
--preprocessing_num_workers 16
--save_steps 500
--eval_steps 500
--save_total_limit 2000
--seed 42
--disable_tqdm false
--ddp_find_unused_parameters false
--block_size 2048
--report_to tensorboard
--overwrite_output_dir
--deepspeed ds_config_zero2.json
--ignore_data_skip true
--fp16
--gradient_checkpointing
--fp16_full_eval
--ddp_timeout 18000000
--resume_from_checkpoint ${output_model}/checkpoint-585

下边是加了resume_from_checkpoint 的log：
[INFO|trainer.py:1969] 2024-07-22 09:33:14,086 >> ***** Running training *****
[INFO|trainer.py:1970] 2024-07-22 09:33:14,086 >> Num examples = 100,000
[INFO|trainer.py:1971] 2024-07-22 09:33:14,086 >> Num Epochs = 3
[INFO|trainer.py:1972] 2024-07-22 09:33:14,086 >> Instantaneous batch size per device = 16
[INFO|trainer.py:1975] 2024-07-22 09:33:14,086 >> Total train batch size (w. parallel, distributed & accumulation) = 512
[INFO|trainer.py:1976] 2024-07-22 09:33:14,086 >> Gradient Accumulation steps = 8
[INFO|trainer.py:1977] 2024-07-22 09:33:14,086 >> Total optimization steps = 585
[INFO|trainer.py:1978] 2024-07-22 09:33:14,090 >> Number of trainable parameters = 20,971,520
0%| | 0/585 [00:00<?, ?it/s]

[Kevin0412]

我也遇到了类似的情况，加了resume_from_checkpoint后运行，我看到打印出来有一行写
Checkpoint /home/a/Desktop/Llama3-Chinese-8B-Lora-train2/checkpoint-40/adapter_model.bin not found
然后确实在检查点文件夹中pytorch_model.bin或adapter_model.bin这个文件，然后就重新开始了。
` if training_args.do_train:
checkpoint = None
if training_args.resume_from_checkpoint is not None:
resume_from_checkpoint = training_args.resume_from_checkpoint
checkpoint_name = os.path.join(resume_from_checkpoint, "pytorch_model.bin")
if not os.path.exists(checkpoint_name):
checkpoint_name = os.path.join(
resume_from_checkpoint, "adapter_model.bin"
) # only LoRA model - LoRA config above has to fit
resume_from_checkpoint = (
False # So the trainer won't try loading its state
)
# The two files above have a different name depending on how they were saved, but are actually the same.
if os.path.exists(checkpoint_name):
print(f"Restarting from {checkpoint_name}")
adapters_weights = torch.load(checkpoint_name)
set_peft_model_state_dict(model, adapters_weights)
else:
print(f"Checkpoint {checkpoint_name} not found")
# checkpoint = Fa
elif last_checkpoint is not None:
checkpoint = last_checkpoint

    if torch.__version__ >= "2" and sys.platform != "win32":
        model = torch.compile(model)
    
    train_result = trainer.train(resume_from_checkpoint=checkpoint)`

finetune_clm_lora.py[627:653]
感觉像是一个bug

[Kevin0412]

试了一下，参考了finetune_clm.py关于重新训练的写法，似乎是成功了，你试试看。
···python
# Training
if training_args.do_train:
checkpoint = None
'''if training_args.resume_from_checkpoint is not None:
resume_from_checkpoint = training_args.resume_from_checkpoint
checkpoint_name = os.path.join(resume_from_checkpoint, "pytorch_model.bin")
if not os.path.exists(checkpoint_name):
checkpoint_name = os.path.join(
resume_from_checkpoint, "adapter_model.bin"
) # only LoRA model - LoRA config above has to fit
resume_from_checkpoint = (
False # So the trainer won't try loading its state
)
# The two files above have a different name depending on how they were saved, but are actually the same.
if os.path.exists(checkpoint_name):
print(f"Restarting from {checkpoint_name}")
adapters_weights = torch.load(checkpoint_name)
set_peft_model_state_dict(model, adapters_weights)
else:
print(f"Checkpoint {checkpoint_name} not found")
# checkpoint = Fa'''
if training_args.resume_from_checkpoint is not None:
checkpoint = training_args.resume_from_checkpoint
elif last_checkpoint is not None:
checkpoint = last_checkpoint