Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

微调训练过程中学习率不会动态下降 #6347

Closed
1 task done
QuanhuiGuan opened this issue Dec 16, 2024 · 1 comment
Closed
1 task done

微调训练过程中学习率不会动态下降 #6347

QuanhuiGuan opened this issue Dec 16, 2024 · 1 comment
Labels
solved This problem has been already solved

Comments

@QuanhuiGuan
Copy link

QuanhuiGuan commented Dec 16, 2024

Reminder

  • I have read the README and searched the existing issues.

System Info

python 3.9
llamafactory 0.9.1

Reproduction

我的yaml文件(如下),lora微调训练的过程学习率不会动态更新,有大佬可以指点一下吗?lr_scheduler_type: cosine 这个设置也打开了,但是还是不会动态调整学习率,也是bf16,希望大佬们指点。

model_name_or_path: /mnt/SSD_12TB/model_gallery/Qwen2.5-32B-Instruct

stage: sft
do_train: true
finetuning_type: lora
lora_target: all
deepspeed: /mnt/SSD_12TB/ethan/data/LLaMA-Factory/examples/deepspeed/ds_z3_config_Ethan.json

dataset: guanqi_train
template: qwen
overwrite_cache: true
preprocessing_num_workers: 16

output_dir: /mnt/SSD_12TB/ethan/data/LLaMA-Factory/model_save/Qwen2_32B_guangqi_1216_dova_1200
logging_steps: 1
plot_loss: true
overwrite_output_dir: true

use_dora: true
lora_rank: 8
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 4.0e-4
num_train_epochs: 5.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
max_length: 4096
save_strategy: epoch

val_size: 0.05
per_device_eval_batch_size: 4
eval_strategy: steps
eval_steps: 5

image

Expected behavior

No response

Others

No response

@github-actions github-actions bot added the pending This problem is yet to be addressed label Dec 16, 2024
@hiyouga
Copy link
Owner

hiyouga commented Dec 17, 2024

check the deepspeed config

@hiyouga hiyouga closed this as completed Dec 17, 2024
@hiyouga hiyouga added solved This problem has been already solved and removed pending This problem is yet to be addressed labels Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
solved This problem has been already solved
Projects
None yet
Development

No branches or pull requests

2 participants