Skip to content

FLUX

Jump to bottom
Vladimir Mandic edited this page Aug 29, 2024 · 33 revisions

Black Forest Labs FLUX.1

FLUX.1 family consists of 3 variations:

  • Pro
    model weights are NOT released, model is available only via Black Forest Labs
  • Dev
    open-weight, guidance-distilled from Pro variation, available for non-commercial applications
  • Schnell
    open-weight, timestep-distilled from Dev variation, available under Apache2.0 license

Additionally SD.Next includes pre-quantized variations of FLUX.1 Dev variation: qint8, qint4 and nf4
To use either any variations or quantizations, simply select from Networks -> Reference
and model will be auto-downloaded on first use
Use of manually downloaded safetensors files is not supported at this time (see Todo section)

Notes

  • Use of FLUX.1 LoRAs is supported
  • Scheduler: FLUX.1 is based on Flow-matching scheduling, only supported sampler is Euler Flow Match (Default)
    Setting any other sampler will be ignored
  • VAE: FLUX.1 VAE does not support FP16, it is recommended to use BF16 if you have a compatible GPU
    Otherwise, VAE will be upcast to FP32 which takes more memory and time
  • To enable image previews during generate, set Settings -> Live Preview -> Method to TAESD
  • To further speed up generation, you can disable "full quality" which triggers use of TAESD instead of full VAE to decode final image

Offloading

FLUX.1 is a massive model at ~32GB and as such it is recommended to use offloading: Settings -> Diffusers -> Model offload mode:

  • Recommended for high VRAM GPUs: Balanced
    Faster but requires compatible GPU and sufficient VRAM
  • Recommended for low VRAM GPUs: Sequential
    Much slower but allows FLUX.1 to run on GPUs with 6GB VRAM

Quantization

Note: Quantization can further reduce memory requirements, but it can also slightly reduce quality of outputs

  • qint8 and qint8 quantization require optimum-quanto which will be auto-installed on first use
    note: qint quantization requires torch==2.4.0
    note: is not compatible with balanced offload
  • nf4 quantization requires bitsandbytes which will be auto-installed on first use
    note: bitsandbytes package is not compatible with all platforms and gpus

Fine-tunes

Diffusers

There are already many FLUX.1 unofficial variations available
Any Diffuser-based variation can be downloaded and loaded into SD.Next using Models -> Huggingface -> Download
For example, interesting variation is a merge of Dev and Schnell variations by sayakpaul: sayakpaul/FLUX.1-merged

Single-file Safetensors

Loading of single-file safetensors is experimental:

  • Supported for transformer (otherwise known as UNet) part of the FLUX.1 model only!
  • Safetensors that contain full model with VAE and text-encoder are not supported at the moment and will be added in the future
  • Safetensors in pre-quantized format are not supported at the moment and will be added in the future

To load a Unet safetensors file:

  1. Download safetensors file from desired source and place it in models/UNET folder
    example: FastFlux Unchained
  2. Load FLUX.1 model as usual and then
  3. Replace transformer with one in desired safetensors file using:
    Settings -> Execution & Models -> UNet

For convience, you can add that setting to your quicksettings by adding Settings -> User Interface -> Quicksettings list -> sd_unet

ToDo / Future

© SD.Next

Clone this wiki locally