Skip to content

Fullstack chatbot built using Rust. Made using Candle, Leptos, Actix, Tokio and Tailwind. Uses quantized Mistral 7B Instruct v0.1 GGUF models.

License

Notifications You must be signed in to change notification settings

shettysach/CandleMist

Repository files navigation

CandleMist

   
  • A simple chatbot built using Rust, in the frontend and the backend.

  • Made using candle, leptos, actix, tokio and TailwindCSS.

  • Uses quantized Mistral 7B Instruct v0.1 GGUF models.

    Credits

    • This is a fork of MoonKraken/rusty_llama by Code to the Moon.
    • This chatbot uses Mistral GGUF models and the huggingface/candle framework, which includes the candle-transformers crate, whereas the original uses GGML models and the rustformers/llm crate.
    • The frontend has some aesthetic changes, but the overall structure is the same.
    • Colours are from the Tokyo Night colorscheme.

Setup Instructions

Rust Toolchain

You'll need to use the nightly Rust toolchain, and install the wasm32-unknown-unknown target as well as the trunk and cargo-leptos tools:

rustup toolchain install nightly
rustup target add wasm32-unknown-unknown
cargo install trunk cargo-leptos

Hardware

  • For CUDA, add the cuda feature for candle_core in Cargo.toml.
candle-core = { git = "https://github.com/huggingface/candle.git", version = "0.6.0", optional = true, features = ["cuda"] }
  • For Metal, add the metal feature for candle_core in Cargo.toml.
  • For Intel's oneAPI Math Kernel Library, add the mkl feature for candle_core in Cargo.toml.
  • For Accelerate, add the accelerate feature for candle_core in Cargo.toml.

Model

TailwindCSS

  • Install TailwindCSS with npm install -D tailwindcss.

Run

git clone https://github.com/ShettySach/CandleMist.git
cd CandleMist
npx tailwindcss -i ./input.css -o ./style/output.css
cargo leptos serve --release
  1. In your browser, navigate to http://localhost:3000/?
NOTE -
  • You can modify parameters such as temperature, seed, top-k, top-p, max history and max response length, and also modify the chat template in 'src/api.rs'.