Skip to content

Commit f667885

Browse files
committed
First pass at Mac Metal Getting Started
- Added Mac Metal Getting started - Added filler for linux Nvidia/AMD Signed-off-by: JJ Asghar <[email protected]>
1 parent 2a8972e commit f667885

File tree

4 files changed

+1138
-5
lines changed

4 files changed

+1138
-5
lines changed

docs/.index.md.swp

-12 KB
Binary file not shown.

docs/getting-started/linux_amd.md

+382-2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,384 @@
1-
# Getting Started Linux AMD
1+
# Getting Started Mac Metal
22

3+
## tl;dr
4+
5+
!!! note
6+
These steps will pull down a premade `qna.yaml` so you can do a local build. skip the `wget`, `mv`, and `ilab taxonomy diff` if you don't want to do this.
7+
8+
```bash
9+
python3.11 -m venv venv-instructlab-0.18-3.11
10+
source venv-instructlab-0.18-3.11/bin/activate
11+
pip install 'instructlab[mps]'
12+
which ilab
13+
ilab config init
14+
cd ~/Library/Application\ Support/instructlab/
15+
mkdir -p taxonomy/knowledge/astronomy/constellations/Phoenix/
16+
wget https://raw.githubusercontent.com/instructlab/taxonomy/26b3fe21ccbb95adc06fe8ce76c7c18559e8dd05/knowledge/science/astronomy/constellations/phoenix/qna.yaml
17+
mv qna.yaml taxonomy/knowledge/astronomy/constellations/Phoenix/
18+
ilab taxonomy diff
19+
ilab data generate
20+
ilab model train
21+
ilab model convert --model-dir checkpoints/instructlab-granite-7b-lab-mlx-q
22+
ilab model serve --model-path instructlab-granite-7b-lab-trained/instructlab-granite-7b-lab-Q4_K_M.gguf
23+
```
24+
25+
## Steps
26+
27+
### Install `ilab`
28+
29+
1. Create a new directory called `instructlab` to store the files the `ilab` CLI needs when running and `cd` into the directory by running the following command:
30+
31+
```shell
32+
mkdir instructlab
33+
cd instructlab
34+
```
35+
36+
!!! note
37+
The following steps in this document use [Python venv](https://docs.python.org/3/library/venv.html) for virtual environments. However, if you use another tool such as [pyenv](https://github.com/pyenv/pyenv) or [Conda Miniforge](https://github.com/conda-forge/miniforge) for managing Python environments on your machine continue to use that tool instead. Otherwise, you may have issues with packages that are installed but not found in `venv`.
38+
39+
2. There are a few ways you can locally install the `ilab` CLI. Select your preferred installation method from the following instructions. You can then install `ilab` and activate your `venv` environment.
40+
41+
!!! note
42+
`pip install` may take some time, depending on your internet connection. In case installation fails with error ``unsupported instruction `vpdpbusd'``, append `-C cmake.args="-DLLAMA_NATIVE=off"` to `pip install` command.
43+
44+
3. Install with Apple Metal on M1/M2/M3 Macs
45+
46+
!!! note
47+
Make sure your system Python build is `Mach-O 64-bit executable arm64` by using `file -b $(command -v python)`,
48+
or if your system is setup with [pyenv](https://github.com/pyenv/pyenv) by using the `file -b $(pyenv which python)` command.
49+
50+
```shell
51+
python3 -m venv --upgrade-deps venv
52+
source venv/bin/activate
53+
pip cache remove llama_cpp_python
54+
pip install 'instructlab[mps]'
55+
```
56+
57+
4. From your `venv` environment, verify `ilab` is installed correctly, by running the `ilab` command.
58+
59+
```shell
60+
ilab
61+
```
62+
63+
*Example output of the `ilab` command*
64+
65+
```shell
66+
(venv) $ ilab
67+
Usage: ilab [OPTIONS] COMMAND [ARGS]...
68+
69+
CLI for interacting with InstructLab.
70+
71+
If this is your first time running InstructLab, it's best to start with `ilab config init` to create the environment.
72+
73+
Options:
74+
--config PATH Path to a configuration file. [default:
75+
/home/user/.config/instructlab/config.yaml]
76+
-v, --verbose Enable debug logging (repeat for even more verbosity)
77+
--version Show the version and exit.
78+
--help Show this message and exit.
79+
80+
Commands:
81+
config Command Group for Interacting with the Config of InstructLab.
82+
data Command Group for Interacting with the Data generated by...
83+
model Command Group for Interacting with the Models in InstructLab.
84+
system Command group for all system-related command calls
85+
taxonomy Command Group for Interacting with the Taxonomy of InstructLab.
86+
87+
Aliases:
88+
chat model chat
89+
convert model convert
90+
diff taxonomy diff
91+
download model download
92+
evaluate model evaluate
93+
generate data generate
94+
init config init
95+
list model model_list
96+
serve model serve
97+
sysinfo system info
98+
test model test
99+
train model train
100+
101+
```
102+
103+
!!! important
104+
Every `ilab` command needs to be run from within your Python virtual environment. You can enter the Python environment by running the `source venv/bin/activate` command.
105+
106+
5. Optional: You can enable tab completion for the `ilab` command.
107+
108+
#### Bash (version 4.4 or newer)
109+
110+
Enable tab completion in `bash` with the following command:
111+
112+
```sh
113+
eval "$(_ILAB_COMPLETE=bash_source ilab)"
114+
```
115+
116+
To have this enabled automatically every time you open a new shell,
117+
you can save the completion script and source it from `~/.bashrc`:
118+
119+
```sh
120+
_ILAB_COMPLETE=bash_source ilab > ~/.ilab-complete.bash
121+
echo ". ~/.ilab-complete.bash" >> ~/.bashrc
122+
```
123+
124+
#### Zsh
125+
126+
Enable tab completion in `zsh` with the following command:
127+
128+
```sh
129+
eval "$(_ILAB_COMPLETE=zsh_source ilab)"
130+
```
131+
132+
To have this enabled automatically every time you open a new shell,
133+
you can save the completion script and source it from `~/.zshrc`:
134+
135+
```sh
136+
_ILAB_COMPLETE=zsh_source ilab > ~/.ilab-complete.zsh
137+
echo ". ~/.ilab-complete.zsh" >> ~/.zshrc
138+
```
139+
140+
#### Fish
141+
142+
Enable tab completion in `fish` with the following command:
143+
144+
```sh
145+
_ILAB_COMPLETE=fish_source ilab | source
146+
```
147+
148+
To have this enabled automatically every time you open a new shell,
149+
you can save the completion script and source it from `~/.bashrc`:
150+
151+
```sh
152+
_ILAB_COMPLETE=fish_source ilab > ~/.config/fish/completions/ilab.fish
153+
```
154+
155+
### 🏗️ Initialize `ilab`
156+
157+
1. Initialize `ilab` by running the following command:
158+
159+
```shell
160+
ilab config init
161+
```
162+
163+
*Example output*
164+
165+
```shell
166+
Welcome to InstructLab CLI. This guide will help you set up your environment.
167+
Please provide the following values to initiate the environment [press Enter for defaults]:
168+
Path to taxonomy repo [taxonomy]: <ENTER>
169+
```
170+
171+
2. When prompted by the interface, press **Enter** to add a new default `config.yaml` file.
172+
173+
3. When prompted, clone the `https://github.com/instructlab/taxonomy.git` repository into the current directory by typing **y**.
174+
175+
**Optional**: If you want to point to an existing local clone of the `taxonomy` repository, you can pass the path interactively or alternatively with the `--taxonomy-path` flag.
176+
177+
*Example output after initializing `ilab`*
178+
179+
```shell
180+
(venv) $ ilab config init
181+
Welcome to InstructLab CLI. This guide will help you set up your environment.
182+
Please provide the following values to initiate the environment [press Enter for defaults]:
183+
Path to taxonomy repo [taxonomy]: <ENTER>
184+
`taxonomy` seems to not exists or is empty. Should I clone https://github.com/instructlab/taxonomy.git for you? [y/N]: y
185+
Cloning https://github.com/instructlab/taxonomy.git...
186+
```
187+
188+
`ilab` will use the default configuration file unless otherwise specified. You can override this behavior with the `--config` parameter for any `ilab` command.
189+
190+
4. When prompted, provide the path to your default model. Otherwise, the default of a quantized [Merlinite](https://huggingface.co/instructlab/merlinite-7b-lab-GGUF) model will be used - you can download this model with `ilab model download` (see below).
191+
192+
```shell
193+
(venv) $ ilab config init
194+
Welcome to InstructLab CLI. This guide will help you set up your environment.
195+
Please provide the following values to initiate the environment [press Enter for defaults]:
196+
Path to taxonomy repo [taxonomy]: <ENTER>
197+
`taxonomy` seems to not exists or is empty. Should I clone https://github.com/instructlab/taxonomy.git for you? [y/N]: y
198+
Cloning https://github.com/instructlab/taxonomy.git...
199+
FIX FOR MAC Path to your model [/home/user/.cache/instructlab/models/merlinite-7b-lab-Q4_K_M.gguf]: <ENTER>
200+
```
201+
202+
5. When prompted, please choose a train profile. Train profiles are GPU specific profiles that enable accelerated training behavior. **YOU ARE ON MacOS**, please choose `No Profile (CPU-Only)` by hitting Enter. There are various flags you can utilize with individual `ilab` commands that will allow you to utilize your GPU if applicable.
203+
204+
```shell
205+
Welcome to InstructLab CLI. This guide will help you to setup your environment.
206+
Please provide the following values to initiate the environment [press Enter for defaults]:
207+
Path to taxonomy repo [/home/user/.local/share/instructlab/taxonomy]:
208+
Path to your model [/home/user/.cache/instructlab/models/merlinite-7b-lab-Q4_K_M.gguf]:
209+
Generating `/home/user/.config/instructlab/config.yaml`...
210+
Please choose a train profile to use:
211+
[0] No profile (CPU-only)
212+
[1] A100_H100_x2.yaml
213+
[2] A100_H100_x4.yaml
214+
[3] A100_H100_x8.yaml
215+
[4] L40_x4.yaml
216+
[5] L40_x8.yaml
217+
[6] L4_x8.yaml
218+
Enter the number of your choice [hit enter for the default CPU-only profile] [0]:
219+
Using default CPU-only train profile.
220+
Initialization completed successfully, you're ready to start using `ilab`. Enjoy!
221+
```
222+
223+
The GPU profiles are listed by GPU type and number. If you happen to have a GPU configuration with a similar amount of VRAM as any of the above profiles, feel free to try them out!
224+
225+
### `ilab` directory layout after initializing your system
226+
227+
After running `ilab config init` your directories will look like the following on a Linux system:
228+
229+
```shell
230+
├─ ~/.cache/instructlab/models/ (1)
231+
├─ ~/.local/share/instructlab/datasets (2)
232+
├─ ~/.local/share/instructlab/taxonomy (3)
233+
├─ ~/.local/share/instructlab/checkpoints (4)
234+
```
235+
236+
1. `~/.cache/instructlab/models/`: Contains all downloaded large language models, including the saved output of ones you generate with ilab.
237+
2. `~/.local/share/instructlab/datasets/`: Contains data output from the SDG phase, built on modifications to the taxonomy repository.
238+
3. `~/.local/share/instructlab/taxonomy/`: Contains the skill and knowledge data.
239+
4. `~/.local/share/instructlab/checkpoints/`: Contains the output of the training process
240+
241+
On MacOS, these directories will be under `Library/Application Support/instructlab`. This directory setup is temporary in 0.18.0 and will mimic the Linux paths in future releases. The models directory will be under `Library/Caches/instructlab`.
242+
243+
### 📥 Download the model
244+
245+
- Run the `ilab model download` command.
246+
247+
```shell
248+
ilab model download
249+
```
250+
251+
`ilab model download` downloads a compact pre-trained version of the [model](https://huggingface.co/instructlab/) (~4.4G) from HuggingFace:
252+
253+
```shell
254+
(venv) $ ilab model download
255+
Downloading model from Hugging Face: instructlab/merlinite-7b-lab-GGUF@main to /home/user/.cache/instructlab/models...
256+
...
257+
INFO 2024-08-01 15:05:48,464 huggingface_hub.file_download:1893: Download complete. Moving file to /home/user/.cache/instructlab/models/merlinite-7b-lab-Q4_K_M.gguf
258+
```
259+
260+
> **NOTE** ⏳ This command can take few minutes or immediately depending on your internet connection or model is cached. If you have issues connecting to Hugging Face, refer to the [Hugging Face discussion forum](https://discuss.huggingface.co/) for more details.
261+
262+
#### Downloading a specific model from a Hugging Face repository
263+
264+
- Specify repository, model, and a Hugging Face token if necessary. More information about Hugging Face tokens can be found [here](https://huggingface.co/docs/hub/en/security-tokens)
265+
266+
```shell
267+
HF_TOKEN=<YOUR HUGGINGFACE TOKEN GOES HERE> ilab model download --repository=TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF --filename=mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
268+
```
269+
270+
#### Downloading an entire Hugging Face repository (Safetensors Model)
271+
272+
- Specify repository, and a Hugging Face token if necessary. For example:
273+
274+
275+
```shell
276+
HF_TOKEN=<YOUR HUGGINGFACE TOKEN GOES HERE> ilab model download --repository=TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF --filename=mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
277+
```
278+
279+
#### Downloading an entire Hugging Face repository (Safetensors Model)
280+
281+
- Specify repository, and a Hugging Face token if necessary. For example:
282+
283+
```shell
284+
HF_TOKEN=<YOUR HUGGINGFACE TOKEN GOES HERE> ilab model download --repository=instructlab/granite-7b-lab
285+
```
286+
287+
These types of models are useful for GPU-enabled systems or anyone looking to serve a model using vLLM. InstructLab provides Safetensor versions of our Granite models on HuggingFace.
288+
289+
#### Listing downloaded models
290+
291+
- All downloaded models can be seen with `ilab model list`.
292+
293+
```shell
294+
ilab model list
295+
```
296+
297+
*Example output of `ilab model list` after `ilab model download`*
298+
299+
```shell
300+
(venv) $ ilab model list
301+
+------------------------------+---------------------+--------+
302+
| Model Name | Last Modified | Size |
303+
+------------------------------+---------------------+--------+
304+
| merlinite-7b-lab-Q4_K_M.gguf | 2024-08-01 15:05:48 | 4.1 GB |
305+
+------------------------------+---------------------+--------+
306+
```
307+
308+
### 🍴 Serving the model
309+
310+
- Serve the model by running the following command:
311+
312+
```shell
313+
ilab model serve
314+
```
315+
316+
- Serve a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1):
317+
318+
```shell
319+
ilab model serve --model-path models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
320+
```
321+
322+
- Once the model is served and ready, you'll see the following output:
323+
324+
```shell
325+
(venv) $ ilab model serve
326+
INFO 2024-03-02 02:21:11,352 lab.py:201 Using model 'models/ggml-merlinite-7b-lab-Q4_K_M.gguf' with -1 gpu-layers and 4096 max context size.
327+
Starting server process
328+
After application startup complete see http://127.0.0.1:8000/docs for API.
329+
Press CTRL+C to shut down the server.
330+
```
331+
332+
> **NOTE:** If multiple `ilab` clients try to connect to the same InstructLab server at the same time, the 1st will connect to the server while the others will start their own temporary server. This will require additional resources on the host machine.
333+
334+
- Serve a non-default Safetensors model (e.g. granite-7b-lab). NOTE: this requires a GPU.
335+
336+
Ensure vllm is installed:
337+
338+
```shell
339+
pip show vllm
340+
```
341+
342+
If it is not, please run:
343+
344+
```shell
345+
pip install vllm@git+https://github.com/opendatahub-io/[email protected]
346+
```
347+
348+
```shell
349+
ilab model serve --model-path ~/.cache/instructlab/models/instructlab/granite-7b-lab
350+
```
351+
352+
### 📣 Chat with the model (Optional)
353+
354+
Because you're serving the model in one terminal window, you will have to create a new window and re-activate your Python virtual environment to run `ilab model chat` command:
355+
356+
```shell
357+
source venv/bin/activate
358+
ilab model chat
359+
```
360+
361+
Chat with a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1):
362+
363+
```shell
364+
source venv/bin/activate
365+
ilab model chat --model models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
366+
```
367+
368+
Please note that usage of `--model` necessitates that the existing server has that model. If not, you must exit the server. `--model` in `ilab model chat` has the ability to start a server on your behalf with the specified model if one is not already running on the port.
369+
370+
Before you start adding new skills and knowledge to your model, you can check its baseline performance by asking it a question such as `what is the capital of Canada?`.
371+
372+
> **NOTE:** the model needs to be trained with the generated synthetic data to use the new skills or knowledge
373+
374+
```shell
375+
(venv) $ ilab model chat
376+
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
377+
│ Welcome to InstructLab Chat w/ GGML-MERLINITE-7B-lab-Q4_K_M (type /h for help) │
378+
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
379+
>>b> what is the capital of Canada [S][default]
380+
╭────────────────────────────────────────────────────────────────────────────────────────────────────── ggml-merlinite-7b-lab-Q4_K_M ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
381+
│ The capital city of Canada is Ottawa. It is located in the province of Ontario, on the southern banks of the Ottawa River in the eastern portion of southern Ontario. The city serves as the political center for Canada, as it is home to │
382+
│ Parliament Hill, which houses the House of Commons, Senate, Supreme Court, and Cabinet of Canada. Ottawa has a rich history and cultural significance, making it an essential part of Canada's identity. │
383+
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 12.008 seconds ─╯
3384
4-
TODO

0 commit comments

Comments
 (0)