Hyper Parameter Optimisation of Llama-2

[msmmpts]

How do I plug in Hyper Parameter search as a part of yaml configuration file?

Sample Yaml File :

https://github.com/ludwig-ai/ludwig/blob/master/examples/llama2_7b_finetuning_4bit/llama2_7b_4bit.yaml

[tgaddair]

Hey @msmmpts, here's an example config to perform hyperopt over the lora params:

model_type: llm
base_model: meta-llama/Llama-2-7b-hf

quantization:
  bits: 4

adapter:
  type: lora

input_features:
  - name: instruction
    type: text

output_features:
  - name: output
    type: text

trainer:
  type: finetune
  learning_rate: 0.0003
  batch_size: 2
  gradient_accumulation_steps: 8
  epochs: 3
  learning_rate_scheduler:
    warmup_fraction: 0.01

backend:
  type: local

hyperopt:
  goal: minimize
  metric: loss
  split: train
  parameters:
    adapter.r:
      space: grid_search
      categories: [8, 16, 32]
  executor:
    gpu_resources_per_trial: 1

To run:

ludwig hyperopt --dataset dataset.csv --config config.yaml

You can find more details on Hyperopt here: https://ludwig.ai/latest/configuration/hyperparameter_optimization/

And the user guide here: https://ludwig.ai/latest/user_guide/hyperopt/#running-hyperparameter-optimization