Setup examples folder

README.md

@@ -4,10 +4,12 @@ Mixture of LoRA Experts: leverage the power of fine-tuned LoRA experts by employ
 MoLE works by learning the alpha scaling values for LoRA adapters, which are frozen. These learned alpha values are used to
 gate the LoRA experts in a dense fashion. This method has several advantages:
 
+See the [examples](examples) folder for some examples of how to get started with MoLE.
+
 ## Advantages and features
 - Dense gating of experts allows mixing
 - Because the MoLE layer is the only trainable layer, fine-tuning has few trainable parameters
 - Easy-to-use API: `add_mole_to_model`
 
 ## Installation
-Pending a pip release, `git clone` this repository and run `pip install -e .`.
+Pending a pip release, `git clone` this repository and run `pip install -e .`.

train.ipynb → examples/basic.ipynb

@@ -7,7 +7,6 @@
  "metadata": {},
  "outputs": [],
  "source": [
- "import torch\n",
  "from transformers import AutoModelForCausalLM\n",
  "from peft import PeftModel"
  ]
@@ -100,7 +99,10 @@
  "metadata": {},
  "outputs": [],
  "source": [
- "mole.add_mole_to_model(model=model, mole_config=mole.MoLEClassifierConfig(2),adapters=['adapter_1', 'adapter_2'], peft_config=model.peft_config)"
+ "mole.add_mole_to_model(model = model,\n",
+ " mole_config=mole.MoLEClassifierConfig(2),\n",
+ " adapters=['adapter_1', 'adapter_2'],\n",
+ " peft_config=model.peft_config)"
  ]
  },
  {