TensorRT & Flux Dev

[Woukim]

@comfyanonymous
Sorry for the ping, but maybe you can help figure it out?” I have a RTX 4080 Super with 16gb VRAM, 64gb RAM.
I've tried different weights, also tried the Q8.gguf version, all of which resulted in errors.

What am I doing wrong? TensorRT doesn't support 16gb VRAM? Thank you in advance

[Woukim]

can someone please help?

[shammyfiveducks]

As Flux is not available in this program to train what model are you selecting to train flux (before you get the error)

[comfyanonymous]

TensorRT needs more than 24GB vram at the moment to convert a Flux model, even a 4090 isn't enough.

[Woukim]

Thank you for taking the time to respond, it makes sense now

[J-Cott]

You can put ComfyUI into CPU mode and do the converting then, but you still can't run it on a 24GB card as the .engine file is 22GB and in total it needs around 28GB when ComfyUI has loaded its other bits.

[al-swaiti]

couldn't split it to chunks and save it , in loop for gpu and cpu

[BrechtCorbeel]

I am going to get so mad if Nvidia doesn't start putting out affordable 48Gb cards soon come on even a 4090 ti with 36gb vram 20.000 tensors would be great.

[Foul-Tarnished]

Couldn't someone rent a H100 to convert it ?

[ankh2054]

If renting a H100 will work, I would happily try this? I've not used TensorRT before but I am eagerly waiting for my 4090 and really wanted to try it with Flux.

I could rent a H100 from vast.ai and spin a machine with a Docker image running comfyui. Guessing it only takes a few hours to train the dynamic models, so the cost would be +-$100, depending on how to long it takes to download initial models, train and re-download new ones.

[drake7707]

I could convert it at my workplace but if I remember correctly the TRT will be specific to the GPU I used and not portable to another?
Specs of the server I could use:
2024-09-24 07:03:48,199 - root - INFO - Total VRAM 45548 MB, total RAM 257517 MB
2024-09-24 07:03:48,199 - root - INFO - pytorch version: 2.4.0
2024-09-24 07:03:49,386 - root - INFO - xformers version: 0.0.27.post2
2024-09-24 07:03:49,401 - root - INFO - Device: cuda:0 Quadro RTX 8000

I tried it with the schnell fp8 checkpoint and I run into an error:

[09/24/2024-07:07:26] [TRT] [E] IBuilder::buildSerializedNetwork: Error Code 9: API Usage Error (Networks with BF16 precision require hardwar e with BF16 support.)
!!! Exception during processing !!! a bytes-like object is required, not 'NoneType'

[comfyanonymous]

You can try converting the Flux model using a ada 6000 and then run the engine with a 4090. However this only works if nothing else is running on the 4090. This means no monitor plugged into it and no applications running on it.

Alternatively you can try fp8 + --fast + the torch compile node.

[al-swaiti]

Hey again , I faced before many issues of cuda out of memory plus out of ram , during quantization I was fixing that by save result model by chunks, I think you can apply this for tensor rt , plus missed u last week's hope u okay 👍

[doctorpangloss]

I compiled flux with tensorrt, using the defaults in the tensorrt node pack, and observed better performance*.

runtime	perf
pytorch	1.22s/it
tensorrt	1.06s/it

On RTX A5000, pytorch 2.4.1, tensorrt 10.5.0 Windows
I obviously have been misreading the secs per it line! So there's some promise here. There are a few tweaks that I needed to fix the tracing warnings.

[DuckersMcQuack]

I compiled flux with tensorrt, using the defaults in the tensorrt node pack

What's the process of doing this? As i want to try that on my 3090 to see how much the gain will be.

[doctorpangloss]

you create the workflow then run the node.
the gain is going to be exactly the same as mine. it is similar to using wsl 2 with torch.compile.