[Fooocus 2.1.822] Fooocus Inpaint or Outpaint (Midjourney Left/Right/Top/Bottom)

[lllyasviel]

(Last Update, 2023 Nov 19)

Fooocus 2.1.822 added inpaint and outpaint similar to Midjourney’s “left/right/top/bottom” arrows:

In Fooocus it looks like this:

This “Fooocus Inpaint” is not only a user-friendly UI but also an algorithm production. Fooocus uses its own algorithm DPMPP Fooocus inpaint, and also uses Fooocus's own control model to minimized the influence to the style of base models. The method is partially inspired by Diffusion-based semantic image editing with mask guidance.

In the entire open source community of SDXL, the Fooocus is the only software that allows you to use control-model-based inpaint with arbitary base models.

Note that this method, in general, expects processing generated images with unchanged or minorly changed prompts. We encourage users to drag image like this:

Inpaint Example

Outpaint Example

Use the pan to outpaint:

Advanced: Detail Fix (face, hand, eyes, ...)

From time to time, you may get images like this:

Everything is perfect, but the eyes / faces are just annoying:

Now you can fix details like this:

Note that this interface also allows you to add some prompts just for this inpaint pass. If you are too lazy, you can also use the shortcuts here:

and you get result like these

Another example:

before:

after:

Advanced: Add New Objects / Change Background

You can always add new objects just using default setting. But if the modification is too large, a better idea is to use

For example:

Change Background:

This "modification" mode will completely ignore original content and is good at handling images with solid color backgrounds like

Very Advanced: Multiple-pass inpaint

You can first use general inpaint to get the image, drag the image to inpaint input again, and then change the mode to "Improve Detail" and then inpaint again. In this way you will get extremely high quality result. (Improve Detail can be used to improve details of anything, not only faces or hands).

[AssassinFlash]

hello, There's a trouble when I'm using the inpaint. I dragged a photo with a house, masked the wall and typed "battery storage on the wall". Then I clicked generate, it just stuck and the cmd just said image processing but without further more.

[Rickets99]

hit the space bar in the cmd window

[lllyasviel]

"DPMPP Fooocus Seamless 2M SDE Karras Inpaint Sampler" is designed for generated images with unchanged or minor edited prompts, similar to MJ use cases

[lllyasviel]

resolved in 2.0.65

[MoonRide303]

@lllyasviel I did some tests on v2.0.65 - it's noticeably better than v2.0.60:

But it's still lagging behind old inpainting models - output from SD 1.5 inpainting for comparison:

Made with diffusers:

import torch
from diffusers import StableDiffusionInpaintPipeline
from diffusers.utils import load_image

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
)
pipe.to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "an empty bench"
generator = torch.Generator()

generator.manual_seed(0)
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, width=1024, height=1024, generator=generator).images[0]
image.save("inpainting-test-sd15-x2.png")

[MoonRide303]

I tried with a bit bigger masks (single area) and seems to work better, then:

It looks like it confused scale at some point, though - the inpainted pattern on the bench looks about twice as big as it should be.

[lllyasviel]

have several other models but will save some for Fooocus V3
also some inpaint models are only better before offset merge, some others (especially external models) are better after changing base

[onlinedear]

cool

[holyvslin]

Fantasy!
With selected area, Generate again with same prompt and change image number to 1.
New "real" image without selected area will be output.

[lllyasviel]

Update 2.0.65:

Fooocus now uses its own inpaint model for inpaint/outpaint. The model is 1.2GB and it will be downloaded in the first time you inpaint an image.

[bluntstone]

new version is broken on linux, complains cuda version is wrong. Going to try to install on windows.

[lllyasviel]

please follow official installation guide on linux

[bluntstone]

I did, on the fresh install its broken, and the one that was working I did a git pull and it broke. It was working no issues about a week ago. I will try again and post the error. I deleted the install so I can't post the error now.

[bluntstone]

Working now, I think I didn't do a step. the install instructions maybe changed. Not really sure, I am using a fresh ubuntu server install and its all working. I don''t think it did "pip install pygit2==1.12.2" the last time.

[lllyasviel]

is image 0-255 range?

[lllyasviel]

do you have other examples or is it always more red and no other colors

[MoonRide303]

@lllyasviel I got it - in case of MRE default sampler_name ('dpmpp_2m_sde_gpu') wasn't handling inpainting colors properly. When I set sampler_name to 'dpmpp_fooocus_2m_sde_inpaint_seamless' for inpainting / outpainting workflows it looks good, now.

It would be nice to implement this inpainting-specific parts in more generic way, so it would work with any base sampler. Wouldn't proposing pull requests for ComfyUI (so we wouldn't need all that patching on the Fooocus side) be better long-term solution?

[lllyasviel]

cool

[fatinghenji]

Hello, I have not come into contact with Midjourney before, so the "arrow" mentioned in the above post confused me. Can you give a more detailed explanation?

[OperationTwelve]

those arrows when selected will outpaint in those directions.

[sirilucas]

Hello, I'm having trouble with inpainting. Please note that my coding skills are almost zero.

At the moment I'm trying to inpaint some areas masked in a previusly generated image, it seems that the code runs nominal, but at the very bottom it stops at a RuntimeError that I can't solve:

File "D:\Descargas\SETUP\Fooocus_win64_2-0-50\python_embeded\lib\site-packages\torch\serialization.py", line 283, in __init__
    super().__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

I hope some of you can help me! Thanks in advance.

[tmprabubiz]

Dear lllyasviel

Inpainting, Outpainting and Upscaling is not working for me

I am just getting error messge "connection errored out" and in the command box the number "1006" appears

Why the folders like "embeddinggs', "inpaint", "style_,models" and others are empty

I am getting following error message while loading
///missing {'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'}////
///missing {'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'}///

Should I install any models in the above empty folders to make Outpainting and Upscaling work

Please guide

[barepixels]

MoonRide303 can you provide a tree lists all the files or directories. Now am paranoid if I miss anything haha

One thing I notice, my Fooocus-MRE have "clip_vision_g.safetensors" but Fooocus-lllyasviel & Fooocus-Runew0lf does not

[barepixels]

What is this https://huggingface.co/lllyasviel/misc/blob/main/fooocus_expansion.bin

Both Fooocus-MRE and Fooocus-lllyasviel have "pytorch_model.bin" in E:\Fooocus-lllyasviel\Fooocus\models\prompt_expansion\fooocus_expansion

[captainpd]

This is a GPT-2 NLP model for extanding your current prompts for better performance. It will change your prompts and print out the result on the cmd.

[MoonRide303]

@barepixels And clip_vision_g is used for Revision workflow, only - it's not needed in vanilla.

[barepixels]

So fooocus_expansion.bin is the same as pytorch_model.bin BUT both Fooocus-MRE and Fooocus-lllyasviel prefer pytorch_model.bin ? a file with a generic name? lol

[barepixels]

This video explains why Fooocus' Outpainting area is blurry
https://youtu.be/VJymbJ97WDQ?si=TYNKhARgmvKvOEi2

[MoonRide303]

@barepixels That's kinda expected - to get high quality output you shouldn't inpaint / outpaint region bigger than 1024x1024 at a time (at least with current inpainting implementation). That's also the reason why you get better results by separately outpainting left, THEN separately right, etc.

Just please mind doing inpainting / outpainting as tiles will side effects:

1. there might be artifacts at tiles edges (as explained around 7:00 in that video),
2. process will take more time.

[AsenAsenAsen]

when the picture width exceeds the maximum width of the preview box,cannot apply a mask that exceeds the part,

[lllyasviel]

gradio bug
uncheck advanced to make it wider

[lllyasviel]

[Important] Fooocus Inpaint V2 Beta Test: #615

[funkysugar1983]

can somebody tell me why this happened? can I download the model manually and where should I put the files? thx in advance.

[ycyy]

https://huggingface.co/lllyasviel/fooocus_inpaint/tree/main download from this link. put into Fooocus\models\inpaint this path.

[funkysugar1983]

thks so much. it's working

[lllyasviel]

2023 Nov 19:
beta test ends and this post is updated.

[qicuili]

how can i use the api about Inpaint or Outpaint

[artistry-intelligence-solutions]

Hello Fooocus community. Hope everyone is having a safe, happy festive season. I'm writing a comic book, continuity is very important. So I rendrered my protagonist on a white background, and rendered his office. I'm trying to place the character in the office and hopefully get the renderer to render the correct angle of the office according to the action. It either gives me something completely different or a jumbled pixelated mess. I've tried unsuccessfully all the different options of Image Prompt and Input or Output Paint.

Obvsiously visual continuity is very important in any story telling so I'd really appreciate any and all help here. Thank you!

[simartem]

have a look at this : https://github.com/AIGCDesignGroup/ReplaceAnything

[jclevet]

Hi,
I don't know if you generated the child's image on Fooocus. But you could generate your subjects directly on a uniform green or blue background. This will allow you to embed them with keying tools on image software.
Add for exemple to your prompt : standing in front of a perfectly seamless vivid green screen, studio lighting.

I will work on 4 styles for subjects on green, blue, white and black backgrounds. I will offer them to Illyas if he is interested.

[artistry-intelligence-solutions]

@jclevet thanks for getting back to me! Yes I did generate this with fooocus ^^

[siriume]

I hope that someday we can utilize OpenPose to generate motions or mask overlays for InPaint or OutPaint purposes.

[lawsonxwl]

Will the object removal model be released? Current inpaint model is not good at removing object.

[kpagcha]

I am also struggling to create a background for an input image with a white background.

[gutsalovpavel]

trying to generate clothes, by modifying content, but it is always generating green cloth... why green? is it the color of the mask for inpaint?

[gutsalovpavel]

ok, got it. it draws green objects when you have a pink background. strange anyway, but resolvable. we need to use grey background for proper colors

[pilotnikovk]

Hi, how can I insert an object into a picture, for example, I have a chair and an interior, and this chair needs to be inserted into my interior, is this possible?

[simartem]

may be you can have a look at this project :

https://github.com/AIGCDesignGroup/ReplaceAnything

[3Diva]

@pilotnikovk If you drop your image into the inpaint tab you should be able to paint over the area you want a chair to be and prompt for the chair to add it to the image.

[hyper-realist]

fooocus version: Google Colab public GPU server
Hi all,
is anyone having trouble with an extended processing time? I feel while processing any of the image input features: Upscale or Variation/Image Prompt/Inpaint or Outpaint/Describe, never stops.
Any thoughts to fix this?

[hyper-realist]

Ahhhh, Thanks for the quick reply,
Does that mean I won't be able to use these features?

"Please check the terminal output and restart your Colab playbook execution if needed." what do you mean by this?

[mashb1t]

@hyper-realist

Does that mean I won't be able to use these features?

Most likely not, at least not reliably.
You can try to restart Colab on error or if it stopped and try again.

[hyper-realist]

I tried paying for the service yet it still seems to be doing the same thing

[mashb1t]

@hyper-realist please check if applying the proposed changes of #1710 helps you to achieve your goal.

[hyper-realist]

Thank you for this, is this edited in the script?
I'm unable to edit the script while running

[psegovias]

I have a problem when I draw a mask and set the message to "detailed girl face" instead of fixing the face, it makes me a whole new face of a detailed girl, any idea?

[calman13]

Hello.
How can I combinate two images, for example a photo with my face with a photo generated by AI or vice versa. Also, is it possible to combine two models for example of civitAI, one anime and one realistic?

[Rotciv007]

Hello. I want to share an idea to add a feature to the Focus that will favorably distinguish it from the "competitors". I would call this function "replacement". In one window, there is an image in which replacements will be made. This image marks the fragment that needs to be changed by design. An image is placed in another window, and the fragment that will be replaced is marked on it. Suppose we are not satisfied with the hand (face, items of clothing, objects), but there is another image in which the hand (face, items of clothing, objects) is exactly what we would like to receive. We highlight the "bad hand" in one image, highlight the "good hand" on the donor. Instead of taking a long time to select the necessary product, we simply demonstrate the required change. I'm sure the Focus is capable of more.

[IPv6]

try ComfyUI, it is more suitable for fine-grained workflow.

[theaccofai]

How to fix hands and fingers? I generated a female model that is laying on the sun bed, everything is great, but the fingers on her hand and toes are incorrect (

[mashb1t]

use manual inpaint, use the auto masking feature of my fork or wait for mashb1t#42 (also see #3122).

[antoniormrzz]

Inpaint them until you get something close to good, then add some lines with photoshop and try to inpaint again. You can also try to fix things in photoshop.

[theaccofai]

Inpaint default, improve detail, or modify content ?

[theaccofai]

Inpaint default, improve detail, or modify content ?

I also tried prompt: (hands,fingers:1,5)

[antoniormrzz]

I used default but you can try improve too. try putting negative prompt like deformed hand, deformed fingers, mutated, etc. You can find good examples of negatives on CivitAI.

[trumantian]

Anyone knows how to avoid the object editing during changing background?

[ErgoR3AL]

Hello there could you explain me how to change shapes and colors of the hairs of a woman the best way possible? Thank you

[Yogesh-DevHub]

Use inpaint, select hair then select inpaint method as modify content. the modify prompt can be 'red hair bun' and generate.
reference video: https://youtu.be/vRKkj_f9cJk?t=205&si=V2OcLcix_NqMv-9K

For changing hair color, you can use the advance masking feature by adding detection prompt 'hair' after getting mask, use modify content with prompt to change the color.
https://youtu.be/4oWGtIHFlCQ?t=517&si=5XMLU79vDHZfEm1Z

[ErgoR3AL]

I will try thank you

[XDM-Inc]

Note that this method, in general, expects processing generated images with unchanged or minorly changed prompts. We encourage users to drag image like this:

were you saying we can drag and drop or that you would like to see the feature? because for me dragging and dropping from a result back to a source window does not work.

OS: fedora linux
Browser:firerox