Ollama

Ollama

By default, ShellGPT leverages OpenAI's large language models. However, it also possible to use locally hosted models, which can be a cost-effective alternative. To use local models, you will need to run your own LLM backend server Ollama. Ollama provides an easy way to download and run Llama 2, Mistral, and other large language models locally.

Note

ShellGPT is not optimized for local models and may not work as expected.

Installation

MacOS

Download and launch Ollama app.

Linux & WSL2

curl https://ollama.ai/install.sh | sh

Setup

We can have multiple large language models installed in Ollama like Llama2, Mistral and others. It is recommended to use mistral:7b-instruct for the best results. To install the model, run the following command:

ollama pull mistral:7b-instruct

This will take some time to download the model and install it. Once the model is installed, you can start API server:

ollama serve

ShellGPT configuration

Now when we have Ollama backend running we need to configure ShellGPT to use it. To communicate with local LLM backends, ShellGPT utilizes LiteLLM. To install it run:

pip install shell-gpt[litellm]

Check if Ollama backend is running and accessible:

sgpt --model ollama/mistral:7b-instruct  "Who are you?"
# -> I'm ShellGPT, your OS and shell assistant...

If you are running ShellGPT for the first time, you will be prompted for OpenAI API key. Provide any random string to skip this step (do not just press enter with empty input). If you got an error you can ask ShellGPT community for help.

Now we need to change few settings in ~/.config/shell_gpt/.sgptrc. Open the file in your editor and change DEFAULT_MODEL to ollama/mistral:7b-instruct. Also make sure that OPENAI_USE_FUNCTIONS is set to false and USE_LITELLM is set to true. That's it, now you can use ShellGPT with Ollama backend.

sgpt "Hello Ollama"