Description
Is there an existing issue for this?
- I have searched the existing issues
OS
macOS
GPU
mps
VRAM
32GB
What happened?
To reproduce
- Generate or upload an image
- Upscale it using ESRGAN with any scale and any strength > 0
- Send it to canvas
- Position bounding box entirely within the image
- Click invoke (any settings)
Results
-
The image is outpainted instead of img2img'd.
-
An error like this is in my terminal on MPS:
/Users/spencer/Documents/Code/stable-diffusion/ldm/invoke/generator/base.py:144: RuntimeWarning: Mean of empty slice.
init_means = np_init_rgb_pixels_masked.mean(axis=0)
/Users/spencer/mambaforge/envs/invokeai/lib/python3.10/site-packages/numpy/core/_methods.py:265: RuntimeWarning: Degrees of freedom <= 0 for slice
ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
/Users/spencer/Documents/Code/stable-diffusion/ldm/invoke/generator/base.py:146: RuntimeWarning: Mean of empty slice.
gen_means = np_image_masked.mean(axis=0)
- On MPS, the result image is totally black. On CUDA, it appears to have messed up colors.
Examples
CUDA on Linux (thanks @JPPhoto)
Canvas state:
Result:
MPS
Canvas state:
Representation of transparency in the bounding box area:
Result:
Investigation
Root cause is ESRGAN (or our implementation) introduces many partially transparent pixels, scattered across the whole image, to the upscaled image.
We see that transparency and run outpainting on the transparent pixels.
Then we do seam painting on all the seams around all of the transparent pixels. This results in the whole image being seam-painted, which means the whole image gets img2img'd.
This non-fatally fails for some reason, resulting in a black output image on MPS and one with messed up colors on CUDA on Linux.
Solutions
We need to do both of these.
-
Fix ESRGAN outputs to not introduce transparency.
-
Fix inpainting so that it does not fail in the edge case of many scattered transparent pixels.
Screenshots
No response
Additional context
No response
Contact Details
No response