feat: support a local custom cortex engine

[RyanMetcalfeInt8]

Hello, I have a custom cortex engine (compiled into an engine.dll). Right now I am able to make use of it in a super hacky way, by replacing the default cortex.llamacpp engine.dll with my custom version. But obviously I don't want to have to do that..

So, I am wondering -- What the correct way to make use of my custom engine? I see in model.json that there is an 'engine' field, which is typically set to 'nitro'. My confusion here is that 'nitro' seems equivalent to 'cortex' (which I also use), but I don't see any field that can specify which specific engine to use (cortex.llamacpp vs. mine).

It's very possible that I missed / overlooked something..

Perhaps there is also some feature needed in cortex.cpp layer as well?

Thanks!
Ryan

[louis-menlo]

Hi @RyanMetcalfeInt8, that's awesome! You're right, existing models have defined nitro as the engine, but the extension will actually route to the cortex.cpp server underneath, it would run on the llama.cpp engine (version 0.5.5). We're working on multiple engines support, which is tracked here. #3690

[RyanMetcalfeInt8]

Hey @louis-jan -- Thanks for getting back to me so quickly! Makes sense.. I'll subscribe to that issue.. and happy to help test new features as they become available. Cheers!

[louis-menlo]

Hey @RyanMetcalfeInt8, I think this is a valid issue to track for improving the extension dev experience. I encountered the same issue while working on MLX support. I will reopen it for tracking.

[freelerobot]

related #3773

[imtuyethan]

Related: https://discord.com/channels/1107178041848909847/1306612519338643466

[imtuyethan]

• Related: planning: Jan and Cortex's Extension Framework #3773