Build and deploy llama-server on fly.io

[hazelnutcloud]

Hi all, I’ve written a Dockerfile that’ll build and deploy llama-server on fly.io along with the fly.toml configuration file. Here’s the Github repo.

It uses the most minimal dependencies possible to create a small image and downloads model files on initial boot and caches them in a volume for fast subsequent cold starts.

Hope this helps!