Skip to content

Latest commit

 

History

History
61 lines (43 loc) · 2.59 KB

README.md

File metadata and controls

61 lines (43 loc) · 2.59 KB
Raw

deforum-stable-diffusion

Refactor of the Deforum Stable Diffusion notebook (featuring video_init) https://colab.research.google.com/github/deforum/stable-diffusion/blob/main/Deforum_Stable_Diffusion.ipynb

Layout

The notebook has been split into the following parts:

  • deforum_video.py is the main module (everything else gets imported via that if used directly)
    • Reports on the GPU using nvidia-smi
  • general_config.py:
    • sets models_path and output_path and creates them if they don't exist (they're no longer at /content/models and /content/output but under the caller's current working directory ./)
    • also sets DeforumArgs and the namespace (general_args) which gets imported to the main deforum_video module as the name args (like in the original notebook).
  • env_config.py

Installation

See CONDA_SETUP.md file indicating prerequisites to run and the reasoning behind installing them this way

git clone https://github.com/lmmx/deforum-stable-diffusion
cd deforum-stable-diffusion
conda create -n deforum
conda activate deforum
conda install -y pytorch torchvision torchaudio cudatoolkit=11.6 pytorch-lightning -c pytorch -c conda-forge
conda install ffmpeg
pip install accelerate einops ftfy jsonmerge kornia omegaconf opencv-python pandas resize-right torchdiffeq torchmetrics torchtext==0.2.3 transformers
pip install -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers
pip install -e git+https://github.com/deforum/stable-diffusion@main#egg=latent-diffusion
pip install -e git+https://github.com/deforum/k-diffusion@master#egg=k-diffusion
  • Note: somehow I ended up with a broken ffmpeg, but after conda uninstall ffmpeg and conda install ffmpeg again it fixed itself

To download the weights, you need to accept their T&Cs, e.g. for v1.4 go here

(This can be made less awkward by uncommenting the download_model function)

Usage

For now run with:

python ./src/deforum_video/deforum_video.py

and edit settings in ./src/deforum_video/anim_config.py.

  • If you choose animation_mode="2D" you get a video from the prompt(s)
  • If you choose animation_mode="Video Input" you get a video from applying the prompts to the video at ./content/input/video_in.mp4 (this path is customisable as the video_init_path).
  • Change the output length with max_frames, etc. etc.

I will improve this and make it a proper package CLI with entrypoints etc. soon !