feat: Add NVIDIA triton trt-llm extension

[hiro-v]

For #821

Integration diagram

NVIDIA triton inference server and TensorRT LLM setup

• https://github.com/hamelsmu/llama-inference/blob/master/triton-tensorRT/README.md
• This can be extended to triton-inference-cluster using Helm on Kubernetes on DGX clusters

[hiro-v]

1. Check if extension installed - See Inference Triton Trt Llm Extension - v 1.0.0
   

2. Find model in Hub
   

3. Update ~/jan/engines/triton_trtllm.json with base_url as remote public/ private IP
   

4. Chat with the remote llama2 7b model on remote NVIDIA Triton inference server
   

[hiro-v]

There is a current blockage is the error on Triton inference server + trt llm with missing space character: triton-inference-server/tensorrtllm_backend#34
I will check the answer to follow up.

[freelerobot]

What's the rationale for having both inference-extension (which just has the nitro binary) and inference-nitro-extension?

[hiro-v]

No, the inference-extension has been removed @0xSage

[tikikun]

LGTM