Jan: Local Engines

[dan-homebrew]

Overview

• How do we handle Local Engines?
• How do we do this currently (in v0.5.3)
• How do we plan to do this in v0.6.0 (i.e. through inference-cortex-extension)
• How does this interact with cortex.cpp's API - e.g. dependency installation, etc

Goals

• We have a clear Local Engine Extensions API (e.g. run model, stop model)
• Cortex then supports llama.cpp, ONNX and TensorRT-LLM (possibly as separate extensions, but all routed through cortex.cpp)
• Cortex registers 3 separate Settings pages for llama.cpp, ONNX and TensorRT-LLM
• TBD: We support Ollama

Resources

v0.5.3

• I need more details on how this currently works

v0.6.0

• It seems to me that almost everything is handled via inference-cortex-extension
• We will likely need Local Engines to have their own Settings page (e.g. llama.cpp)
• Individual Engines may have their own complexities (e.g. llama.cpp having different versions based on AVX, non-AVX)

[marknguyen1302]

hi @dan-homebrew , about how do we handle Local Engines in v0.5.3:
We have 2 main parts to handle the local engine: load model and inference

• About loading model:
  I have a simple diagram to explain how the load model works in the Nitro engine
  .
  Another things about loading model, when user switch to use another model, we will kill the current cpp process and spawn a new one.
• About inference:
  Nitro will spawn a CPP engine process and we have OAIEngine in the core package to handle the inference process.
  please cmiiw @louis-jan

[dan-homebrew]

@marknguyen1302 Is this diagram in Mermaid.js? Can you replace the image with the actual Mermaid snippet

[marknguyen1302]

oh sorry, here it's

sequenceDiagram
    participant Web
    participant useActiveModel
    participant JanInferenceNitroExtension
    participant Core
    participant NitroProcess

    Web->>useActiveModel: startModel(modelId)
    useActiveModel->>JanInferenceNitroExtension: loadModel(model)
    JanInferenceNitroExtension->>JanInferenceNitroExtension: check model.engine
    alt model.engine is 'nitro'
        JanInferenceNitroExtension->>JanInferenceNitroExtension: setInterval(periodicallyGetNitroHealth, _intervalHealthCheck)
        JanInferenceNitroExtension->>Core: super.loadModel(model)

        Core->>NitroProcess: spawnNitroProcess()
        NitroProcess->>NitroProcess: killSubprocess
        NitroProcess->>NitroProcess: spawnNitroProcess
        NitroProcess->>NitroProcess: loadLLMModel
    else model.engine is not 'nitro'
        JanInferenceNitroExtension->>useActiveModel: return Promise.resolve()
    end

    Note right of JanInferenceNitroExtension: periodicallyGetNitroHealth() runs every _intervalHealthCheck

    JanInferenceNitroExtension->>Core: executeOnMain(NODE, 'getCurrentNitroProcessInfo')
    Core->>NitroProcess: getCurrentNitroProcessInfo()
    NitroProcess->>Core: return health info
    Core->>JanInferenceNitroExtension: return health info

    alt Nitro process stopped
        JanInferenceNitroExtension->>Core: events.emit(ModelEvent.OnModelStopped, {})
    end

Loading

[marknguyen1302]

about what we plan to do:
i have a simple diagram about the process to load model

The Inference process will be the same with Nitro, there is only 1 thing different, cortex extension will handle 3 engines, cortex.llamacpp, cortex.tensorrt-llm, cortex.onnx, so we have to change the OAIEngine a bit to map engine in model.json to correct extension

[dan-homebrew]

Pls attach Mermaid snippet

[marknguyen1302]

sorry here is the Mermaid

sequenceDiagram
Participant User
Participant App
Participant Cortex

User -> App: Request to loadModel

App -> Cortex: GET /v1/system (Check if Cortex is spawned)
alt Cortex is not running
    Cortex -> App: 404 Not Found
    App -> App: Spawn Cortex process
    App -> Cortex: GET /v1/system (Check if Cortex is running)
    Cortex -> App: 200 OK (Cortex is running)
else Cortex is running
    Cortex -> App: 200 OK
end

App -> Cortex: Read model metadata (Check if engine is inited)
Cortex -> App: Return model metadata

alt Engine is not inited
    App -> Cortex: POST /engines/init (Init engine)
    Cortex -> App: 200 OK (Engine inited)
else Engine is already inited
    Note right of App: Skip engine initialization
end

App -> Cortex: POST /models/:modelId/start (Start model with path)
Cortex -> App: 200 OK (Model started with gguf file path)

App -> User: Response that model is started

Loading

[dan-homebrew]

Quick question: do we have any NonOAIEngines?

[dan-homebrew]

@marknguyen1302 @louis-jan From a product perspective, I might request that we avoid "automatically" initializing engines for v0.6.0:

Scenario: If user runs a model that requires an engine that has not been initialized

• CLI: we should return an error, and provide the command for user to initialize the engine
• API: we should return an error code, that Jan/etc can then direct user to initialize the engine (e.g. in Settings)

From my POV, this allows for simpler error handling and recovery, which is important at this stage of the repo (and also for Jan integration)

We can always implement this later on, when the codebase is more stable.

Wdyt?

[marknguyen1302]

do we have any NonOAIEngines? yes, we have, OAIEngines for handling the load, unload model, and inference, it's the parent class of all Engine Extension we have, and we have other extensions for other purposes, for example model extension for showing the model in the hub, for change model setting and persist in model.json, conversational-extension for handing thread, message

[dan-homebrew]

@marknguyen1302 Ah, I meant: do we have any Engines that are non-OpenAI compatible?

[marknguyen1302]

for the local model, I think all engines are compatible with the openAI, but for the remote model there are some engines that are non-OpenAI compatible, for example anthropic engine