Further transform operations on mask

[nistvan86]

First, thank you for the txt2mask function, it's an amazing tool!

I have a request/question related to this. img2img seems to create much more consistent results if I'm using "Original" as the masked content instead of fill or latent noise. However this has a side affect of "infecting" the result with the dominant colors of the image under the mask. (eg. if I'm repainting a person, the original clothes' colors will determine the new colors mostly)

It seems it's possible to use mask in-paint in such way that the mask contains the image being inserted into the masked area before running the inference: masked content, mask mode

I think a simple color rotate, or by adding a custom amount of color gaussian noise on top of the mask could introduce some variety in the color of the output. An even more amazing thing would be to specify prompt=color combinations which would match sub-parts of the mask and do color change/rotate on that.

eg.
[txt2mask negative_mask="head" color_change="t-shirt=red"]person[/txt2mask]mars armored forces trooper
So the end-result would be much likely a red armored trooper instead of a blue one if the original person worn a blue shirt.

How difficult this would be?

[ThereforeGames]

Hi @nistvan86,

Thanks for reaching out!

After looking into the mask functionality, I think what you're describing is perhaps the "Inpaint sketch" tab of the img2img mode. I have not tried this feature before today.

For context:

I tried colorizing the mask via regular inpaint and found that anything other than solid white resulted in a less pronounced change, i.e. the resulting image looked closer to the original image with a blue mask vs a white mask for example.

As far as I can tell, what the sketch mode actually does is apply your colorized mask onto the original image before running the img2img operation. It "flattens" the colored mask onto your image as it were.

It wasn't easy, but I added support for this in Unprompted v7.0.0. Give it a try. :)

You'll need to use the new parameters sketch_color and optionally sketch_alpha to enable the new mode.

sketch_color takes a string like "red" or an RGB tuple like "127,127,127" and sketch_alpha is a value from 0-255 where 0 is invisible and 255 is fully opaque.

Right now, it only supports one color per mask but this may change in the future.

Let me know if you have any questions.

[nistvan86]

Wow, that was fast. Thank you! I'll certainly give it a try soon.

Speaking of parameters, what is the easiest way to find a list of available parameters, especially those which can be configured with set/sets? I've found out most of them with trial and error but I haven't found a comprehensive list of these.

[ThereforeGames]

Speaking of parameters, what is the easiest way to find a list of available parameters, especially those which can be configured with set/sets?

I try to document every parameter in the manual, which you can access from the Unprompted extension itself (you'll want the "Shortcodes" section at the bottom):

Or through the online portal here: https://github.com/ThereforeGames/unprompted/blob/main/docs/MANUAL.md#shortcodes

The vast majority of parameters are also available via Wizard -> Shortcodes if you prefer to look at them in the form of a UI.

[nistvan86]

This seems to be working great. Previously I was unable to recolor such cases to red, no matter what, the clothing ended up being either the original or a blend in between.
This very consistently produce a red overall now. The only problem I'm seeing is that it also adds a red-ish hallo, because the mask obviously wraps around the person.
This is where additional selections on the mask would make sense (just like the negative prompt, but instead of cutting out holes, recoloring that part and giving it transparency). eg.: body=red:128 or t-shirt=red:128 (+padding/smoothing) to be handled as well on these regions) instead of making the whole mask red.
But anyway, this at least shows this feature has merit.

Inpainted:

Original:

Mask:

[txt2mask padding=10 neg_padding=5 smoothing=40 neg_smoothing=20 negative_mask="head" sketch_color="red" sketch_alpha="128" show]person[/txt2mask]
white skin male formula one ferrari race driver wearing red racing suit, three quarter candid medium side shot, serious face, hands on waist, alone, outside, facing left
Negative: ugly, naked, cartoon, anime, 3d, black and white, out of frame, sepia, painting
Steps: 150, Sampler: DPM adaptive, CFG scale: 7, Seed: 2133752019, Size: 512x512, Model hash: 61a37adf76, Model: ProtoGen_X3.4, Denoising strength: 1, Mask blur: 4

[ThereforeGames]

Glad to see it's working for you!

Yes, multi-color masks would be a great improvement - let me give some thought to implementation.

[nistvan86]

I've tried sketch_color="black" sketch_alpha="0" and it produces the same result as if I was omitting these and reverting to normal inpaint, so that could serve as the fill color of the mask if you would implement sub-mask coloring.

I also think negative padding on sub mask definitions would be helpful to hide this halo effect. I think the color "overlay" is only needed to bias the denoiser to choose the expected color, it don't have to be precise but we should avoid stepping out of the shape the prompt selects.
so eg. "body" would probably select a bit more around the person, but with negative padding it's possible to reduce it's coverage to remain within the body.