Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Regional Prompting #5868

Closed
wants to merge 49 commits into from
Closed

WIP: Regional Prompting #5868

wants to merge 49 commits into from

Commits on Feb 14, 2024

  1. Fix avoid storing extra conditioning info in two places.

    @RyanJDick
    RyanJDick committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    16e5748 View commit details
    Browse the repository at this point in the history

  2. Remove use of **kwargs in do_unet_step(...), where full parameter lis… 

    …t is known and supported.
    @RyanJDick
    RyanJDick committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    e866e3b View commit details
    Browse the repository at this point in the history

  3. Remove outdated comments related to T2I-Adapters and ControlNets.

    @RyanJDick
    RyanJDick committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    bf72cee View commit details
    Browse the repository at this point in the history

  4. Merge sequential conditioning and cac conditioning logic to eliminate… 

    … a bunch of duplication.
    @RyanJDick
    RyanJDick committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    ee3abc1 View commit details
    Browse the repository at this point in the history

  5. Remove unused constructor declared with typo in name: __int__.

    @RyanJDick
    RyanJDick committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    382fa57 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Add a mask to the ConditioningField primitive type.

    @RyanJDick
    RyanJDick committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    58277c6 View commit details
    Browse the repository at this point in the history

  2. Add support for a list of ConditioningFields in DenoiseLatents.

    @RyanJDick
    RyanJDick committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    f590b39 View commit details
    Browse the repository at this point in the history

  3. Delete unused functions from shared_invokeai_diffusion.py.

    @RyanJDick
    RyanJDick committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    7b0326d View commit details
    Browse the repository at this point in the history

  4. Remove unused code for attention map saving.

    @RyanJDick
    RyanJDick committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    ef51005 View commit details
    Browse the repository at this point in the history

  5. Initialize a RegionalPromptAttnProcessor2_0 class by copying AttnProc… 

    …essor2_0 from diffusers.
    @RyanJDick
    RyanJDick committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    ba47880 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. Fix a minor bug in the logic of the IPAttnProcessor2_0. The change wo… 

    …n't have any functional effect, since this attention implementation was only being used for cross-attention, but the logic should be correct now in case we wanted to use it for self-attention.
    @RyanJDick
    RyanJDick committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    38248b9 View commit details
    Browse the repository at this point in the history

  2. Add concatenation of multiple text conditioning tensors, and patching… 

    … of RegionalPromptAttnProcessor2_0 into the UNet.
    @RyanJDick
    RyanJDick committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    caa690e View commit details
    Browse the repository at this point in the history

  3. Add RectangleMaskInvocation.

    @RyanJDick
    RyanJDick committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    878bbc3 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2024

  1. Route masks into the RegionalPromptAttnProcessor2_0 processors.

    @RyanJDick
    RyanJDick committed Feb 17, 2024
    Configuration menu
    Copy the full SHA
    2d5d370 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2024

  1. Get RegionalPromptAttnProcessor2_0 working with a ton of hacks.

    @RyanJDick
    RyanJDick committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    d132fb4 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Tidy invocation interfaces for RectangleMaskInvocation and AddConditi… 

    …oningMaskInvocation.
    @RyanJDick
    RyanJDick committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    b0fcbe5 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Handle conditioned and unconditioned text conditioning in the same wa… 

    …y for regional prompt attention.
    @RyanJDick
    RyanJDick committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    2966c8d View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Removed unused function: _prepare_text_embeddings(...)

    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    cfba51a View commit details
    Browse the repository at this point in the history

  2. Add symmetric support for regional negative text prompts.

    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    54971af View commit details
    Browse the repository at this point in the history

  3. Update various comments related to regional prompting, and delete dup… 

    …licate _preprocess_regional_prompt_mask(...) function.
    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    845c4e9 View commit details
    Browse the repository at this point in the history

  4. Remove dead code related to an old symmetry feature.

    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    cad3e5d View commit details
    Browse the repository at this point in the history

  5. Remove scheduler_args from ConditioningData structure.

    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    e7ec13f View commit details
    Browse the repository at this point in the history

  6. Split ip_adapter_conditioning out from ConditioningData.

    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    ee1b315 View commit details
    Browse the repository at this point in the history

  7. Rename ConditioningData to TextConditioningData.

    @RyanJDick
    RyanJDick committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    53ebca5 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. Move regional prompt concatenation further up the stack. This solves … 

    …a number of issues.
    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    5f49e7a View commit details
    Browse the repository at this point in the history

  2. Make regional prompting work with sequential conditioning.

    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    e132afb View commit details
    Browse the repository at this point in the history

  3. Raise a clear error message if prompt-to-prompt cross-attention contr… 

    …ol is triggered when using multiple prompts.
    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    e7f7ae6 View commit details
    Browse the repository at this point in the history

  4. Improve the logic for selecting SDXL pooled embeds when handling mult… 

    …i-region prompts.
    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    bdf3691 View commit details
    Browse the repository at this point in the history

  5. Fixup logic around compatibility of prompt-to-prompt, IP-Adapter, reg… 

    …ional prompting.
    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    1bbd4f7 View commit details
    Browse the repository at this point in the history

  6. Add CustomAttnProcessor2_0 class with simultaneous support for IP-Ada… 

    …pter and regional prompting.
    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    f44d3da View commit details
    Browse the repository at this point in the history

  7. Get multi-prompt attention working simultaneously with IP-adapter.

    @RyanJDick
    RyanJDick committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    8989a6c View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. Fix avoid storing extra conditioning info in two places.

    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    4a1acd4 View commit details
    Browse the repository at this point in the history

  2. Remove use of **kwargs in do_unet_step(...), where full parameter lis… 

    …t is known and supported.
    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    7d96710 View commit details
    Browse the repository at this point in the history

  3. Remove outdated comments related to T2I-Adapters and ControlNets.

    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    d87ff3a View commit details
    Browse the repository at this point in the history

  4. Merge sequential conditioning and cac conditioning logic to eliminate… 

    … a bunch of duplication.
    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    8721926 View commit details
    Browse the repository at this point in the history

  5. Remove unused constructor declared with typo in name: __int__.

    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    3e14bd6 View commit details
    Browse the repository at this point in the history

  6. Delete unused functions from shared_invokeai_diffusion.py.

    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    a5c94fb View commit details
    Browse the repository at this point in the history

  7. Remove unused code for attention map saving.

    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    5b3adf0 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'ryan/remove-attention-map-saving' into ryan/regional-co… 

    …nditioning
    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    ffc4ebb View commit details
    Browse the repository at this point in the history

  9. Implement (very slow) self-attention regional masking.

    @RyanJDick
    RyanJDick committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    942efa0 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2024

  1. Very experimentation with various regional prompting tuning params.

    @RyanJDick
    RyanJDick committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    ad18429 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2024

  1. Add ability to control regional prompt region weights.

    @RyanJDick
    RyanJDick committed Mar 3, 2024
    Configuration menu
    Copy the full SHA
    5fad379 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Merge branch 'main' into ryan/regional-conditioning-tuning

    @RyanJDick
    RyanJDick committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    271f8f2 View commit details
    Browse the repository at this point in the history

  2. Remove AddConditioningMaskInvocaton.

    @RyanJDick
    RyanJDick committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d313e5e View commit details
    Browse the repository at this point in the history

  3. Add positive_self_attn_mask_score and self_attn_adjustment_end_step_p… 

    …ercent to the prompt nodes.
    @RyanJDick
    RyanJDick committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    a665f20 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. (minor) Remove commented code.

    @RyanJDick
    RyanJDick committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    bcfb43e View commit details
    Browse the repository at this point in the history

  2. Use the correct device / dtype for RegionalPromptData calculations.

    @RyanJDick
    RyanJDick committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    41e1a9f View commit details
    Browse the repository at this point in the history

  3. Remove dispatch_progress() function that was added aciidentally durin… 

    …g conflict resolution.
    @RyanJDick
    RyanJDick committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    57266d3 View commit details
    Browse the repository at this point in the history

  4. Fix _negative_cross_attn_mask_score.

    @RyanJDick
    RyanJDick committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    b5c334d View commit details
    Browse the repository at this point in the history