Serve any model without boilerplate code
Meet Truss, a seamless bridge from model development to model delivery. Truss presents an open-source standard for packaging models built in any framework for sharing and deployment in any environment, local or production.
Get started with the end-to-end tutorial.
For devs using Windows, you can use DirectML for testing.
pass gpu=false, install pytorch-directml or the tensorflow equivalent.
use this code
tr.docker_run(
environment_variables={
"LD_LIBRARY_PATH": "/usr/lib/wsl/lib"
},
mounts=[[
"type=bind",
"src=/usr/lib/wsl",
"dst=/usr/lib/wsl",
]],
devices=[
"/dev/dxg"
]
)If you've ever tried to get a model out of a Jupyter notebook, Truss is for you.
Truss exposes just the right amount of complexity around things like Docker and APIs without you really having to think about them. Here are some of the things Truss does:
- 🏎 Turns your Python model into a microservice with a production-ready API endpoint, no need for Flask or Django.
- 🎚 For most popular frameworks, includes automatic model serialization and deserialization.
- 🛍 Freezes dependencies via Docker to make your training environment portable.
- 🕰 Enables rapid iteration with local development that matches your production environment.
- 🗃 Encourages shipping parsing and even business logic alongside your model with integrated pre- and post-processing functions.
- 🤖 Supports running predictions on GPUs. (Currently limited to certain hardware, more coming soon)
- 🙉 Bundles secret management to securely give your model access to API keys.
Truss requires Python >=3.7, <3.11
To install from PyPi, run:
pip install truss
To download the source code directly (for development), clone this repository and follow the setup commands in our contributors' guide.
Truss is actively developed, and we recommend using the latest version. To update your Truss installation, run:
pip install --upgrade truss
Though Truss is in beta, we do care about backward compatibility. Review the release notes before upgrading, and note that we follow semantic versioning, so any breaking changes require the release of a new major version.
Generate and serve predictions from a Truss with this Jupyter notebook.
!pip install --upgrade scikit-learn truss
import truss
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
# Load the iris data set
iris = load_iris()
data_x = iris['data']
data_y = iris['target']
# Train the model
rfc = RandomForestClassifier()
rfc.fit(data_x, data_y)
# Create the Truss (serializing & packaging model)
tr = truss.create(rfc, target_directory="iris_rfc_truss")
# Serve a prediction from the model
tr.predict([[0, 0, 0, 0]])The truss.create() command can be used with any supported framework:
But in more complex cases, you can build a Truss manually for any model. Start with truss init my_truss and follow this guide.
Serving your model with Truss, on Docker, lets you interface with your model via HTTP requests. Start your model server with:
truss run-image iris_rfc_truss
Then, as long as the container is running, you can invoke the model as an API as follows:
curl -X POST http://127.0.0.1:8080/v1/models/model:predict -d '[[0, 0, 0, 0]]'
Truss is configurable to its core. Every Truss must include a file config.yaml in its root directory, which is automatically generated when the Truss is created. However, configuration is optional. Every configurable value has a sensible default, and a completely empty config file is valid.
The Truss we generated above in the quickstart sample has a good example of a typical Truss config:
model_framework: sklearn
model_metadata:
model_binary_dir: model
supports_predict_proba: true
python_version: py39
requirements:
- scikit-learn==1.0.2
- threadpoolctl==3.0.0
- joblib==1.1.0
- numpy==1.20.3
- scipy==1.7.3Follow the configuration guide and use the complete reference of configurable properties to make your Truss perform exactly as you wish.
You can deploy a Truss anywhere that can run a Docker image, as well as purpose-built platforms like Baseten.
Follow step-by-step deployment guides for the following platforms:
We hope this vision excites you, and we gratefully welcome contributions in accordance with our contributors' guide and code of conduct.
Truss was first developed at Baseten by maintainers Phil Howes, Pankaj Gupta, and Alex Gillmor.
If your organization allows to access to GitHub Codespaces, you can launch a Codespace for truss development. If you are a GPU Codespace, make sure to use the .devcontainer/gpu/devcontainer.json configuration to have access to a GPU and be able to use it in Docker with truss.