Skip to content

Commit 62e62a2

Browse files
Chore: Update README (#49)
* update * fix
1 parent e85c678 commit 62e62a2

File tree

1 file changed

+76
-47
lines changed

1 file changed

+76
-47
lines changed

README.md

+76-47
Original file line numberDiff line numberDiff line change
@@ -2,67 +2,41 @@
22
This guide will help you understand how to convert AI models into different formats that our applications can use.
33

44
## What We're Doing
5-
We convert AI models into special formats (GGUF, TensorRT, ONNX) so they can work with our applications called [Jan](https://github.com/janhq/jan) and [Cortex](https://github.com/janhq/cortex.cpp). Think of this like converting a video file from one format to another so it can play on different devices.
5+
We convert LLMs on HuggingFace into special formats (GGUF, TensorRT, ONNX) so they can work with our applications called [Jan](https://github.com/janhq/jan) and [Cortex](https://github.com/janhq/cortex.cpp). Think of this like converting a video file from one format to another so it can play on different devices.
66

77
### New Model Conversion
88

99
#### Step 1: Create a New Model Template
1010

11-
1. Go to our model storage at: https://huggingface.co/cortexso
12-
2. Click "New Model"
13-
3. Give it the same license as the original model you're converting
14-
4. Create two important files in your new model:
11+
This step will create a model repository on [Cortexso's Hugging Face account](https://huggingface.co/cortexso) and then generate two files: `model.yml` as the model's configuration file and `metadata.yml`.
1512

16-
**File 1:** `model.yml`
17-
This file tells our system how to use the model. Copy this template and fill in the details:
18-
19-
```
20-
# Basic Information
21-
id: your_model_name # Choose a unique name
22-
model: your_model_name # Same as above
23-
name: your_model_name # Same as above
24-
version: 1 # Start with 1
25-
26-
# Model Settings
27-
stop: # Where the model should stop generating
28-
- "<|eot_id|>" # You might need to change this
29-
30-
# Default Settings (Usually keep these as they are)
31-
stream: true
32-
top_p: 0.9
33-
temperature: 0.7
34-
max_tokens: 4096
35-
36-
# Technical Settings
37-
engine: llama-cpp # The engine type
38-
prompt_template: "<|begin_of_text|>..." # How to format inputs
39-
ctx_len: 8192 # Context length
40-
ngl: 34 # Usually layers + 1
41-
```
13+
1. Visit: https://github.com/janhq/models/actions/workflows/create-model-yml.yml
14+
2. Click `Run workflow` dropdown
15+
3. Fill in the required information:
16+
- **`model_name`**: Name of the model to create (will be used in repo name and files)
17+
- **`prompt_template`**: Prompt template for the model (default: `<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n`)
18+
- **`stop_tokens`**: Stop tokens for the model (comma-separated, e.g., `,</s>`) (default: `<|im_end|>`)
19+
- **`engine`**: Engine to run the model (default: `llama-cpp`)
20+
4. Click `Run workflow` button to start the conversion
4221

4322
**Common Errors:**
44-
- Wrong Stop Tokens: If the model keeps generating too much text, check the stop tokens in model.yml
45-
- Engine Errors: Make sure you picked the right engine type in model.yml
46-
- Template Issues: Double-check your prompt_template if the model gives weird outputs
47-
48-
**File 2:** `metadata.yml`
49-
50-
```
51-
version: 1
52-
name: your_model_name
53-
default: 8b-gguf-q4-km
54-
```
23+
- Wrong Stop Tokens: If the model keeps generating too much text, check the `stop` tokens in `model.yml`
24+
- Engine Errors: Make sure you picked the right engine type in `model.yml`
25+
- Template Issues: Double-check your `prompt_template` if the model gives weird outputs
5526

56-
#### Step 2: Convert Your Model 🔄
27+
#### Step 2: Convert Model
5728

58-
1. Visit: https://github.com/janhq/models/actions
29+
1. Visit: https://github.com/janhq/models/actions/workflows/convert-model-all-quant.yml
5930
2. Choose your conversion type:
6031
- For [GGUF](https://huggingface.co/docs/hub/gguf) format: Click `Convert model to gguf with specified quant`
6132
- For [TensorRT](https://github.com/NVIDIA/TensorRT-LLM): Coming soon
6233
- For [ONNX](https://onnx.ai/): Coming soon
63-
6434
3. Click `Run workflow` dropdown
65-
4. Fill in the required information
35+
4. Fill in the required information:
36+
- **`source_model_id`**: The source HuggingFace model ID (e.g., meta-llama/Meta-Llama-3.1-8B-Instruct)
37+
- **`**source_model_size`**: The model size (e.g., 8b)
38+
- **`target_model_id`**: The target HuggingFace model ID
39+
- **`quantization_level`**: Quantization level (e.g., 'q4-km') or 'all' for all levels
6640
5. Click `Run workflow` button to start the conversion
6741

6842
#### Step 3: Check If It Worked
@@ -73,11 +47,66 @@ default: 8b-gguf-q4-km
7347
- Create a "Bug Report" on GitHub
7448
- Contact Rex or Alex for help
7549

76-
#### Step 4: Final Testing 🧪
50+
#### Step 4: Final Testing
7751
After the conversion completes:
7852

7953
1. Check Huggingface page to make sure all files are there
8054
2. Test the model in Cortex:
8155

8256
- Make sure it generates text properly
8357
- Check if it stops generating when it should
58+
59+
For instructions on installing and running the model on Cortex, please refer to the [official documentation](https://cortex.so/docs/)
60+
61+
#### Step 5: Create README For New Model
62+
63+
1. Navigate to the newly created model repository on [Cortexso's Hugging Face account](https://huggingface.co/cortexso)
64+
2. Open the repository and select "Create Model Card"
65+
3. Use the template below for your model card, replacing the example content with your model's information:
66+
67+
```yml
68+
---
69+
license: your_model_license
70+
---
71+
72+
## Overview
73+
74+
[Provide a brief description of your model, including its key features, use cases, and performance characteristics]
75+
76+
## Variants
77+
78+
| No | Variant | Cortex CLI command |
79+
| --- | --- | --- |
80+
| 1 | [variant_name](variant_url) | `cortex run model_name:variant` |
81+
| 2 | [main/default](main_url) | `cortex run model_name` |
82+
83+
## Use with Jan (UI)
84+
85+
1. Install **Jan** from the [Quickstart Guide](https://jan.ai/docs/quickstart)
86+
2. In Jan model Hub, enter:
87+
```
88+
cortexso/your_model_name
89+
```
90+
91+
## Use with Cortex (CLI)
92+
93+
1. Install **Cortex** using the [Quickstart Guide](https://cortex.jan.ai/docs/quickstart)
94+
2. Run the model with:
95+
```
96+
cortex run your_model_name
97+
```
98+
99+
## Credits
100+
101+
- **Author:** [Original model creator]
102+
- **Converter:** [Converting organization/person]
103+
- **Original License:** [Link to original license]
104+
- **Papers/References:** [Relevant papers or documentation]
105+
```
106+
4. Review and verify:
107+
108+
- Model license is correctly specified
109+
- All URLs are valid and point to the correct resources
110+
- Model names and commands are accurate
111+
112+
5. Click "Commit changes" to save the model card

0 commit comments

Comments
 (0)