handle lora scale and clip skip in lpw sd and sdxl community pipelines #8988
Conversation
| @@ -11,15 +11,19 @@ | |||
| from diffusers import DiffusionPipeline | |||
| from diffusers.configuration_utils import FrozenDict | |||
| from diffusers.image_processor import VaeImageProcessor | |||
| from diffusers.loaders import FromSingleFileMixin, StableDiffusionLoraLoaderMixin, TextualInversionLoaderMixin | |||
| from diffusers.loaders import FromSingleFileMixin, LoraLoaderMixin, TextualInversionLoaderMixin | |||
There was a problem hiding this comment.
any reason why we're not using the newer StableDiffusionLoraLoaderMixin here?
| @@ -268,6 +292,16 @@ def get_weighted_text_embeddings( | |||
| skip_weighting (`bool`, *optional*, defaults to `False`): | |||
| Skip the weighting. When the parsing is skipped, it is forced True. | |||
| """ | |||
| # set lora scale so that monkey patched LoRA | |||
| # function of text encoder can correctly access it | |||
| if lora_scale is not None and isinstance(pipe, LoraLoaderMixin): | |||
There was a problem hiding this comment.
We recently shipped #8981. I think this should be StableDiffusionLoraLoaderMixin. Maybe @sayakpaul can provide more clarity
There was a problem hiding this comment.
LoraLoaderMixin is fine but it's just deprecated. Prefer using StableDiffusionLoraLoaderMixin as @a-r-r-o-w mentioned.
What? |
I don't mean for you to revert 8981 😨 I thought the author of current PR removed your changes by mistake |
|
@a-r-r-o-w yes, i removed those changes by mistake, it should be fixed now |
| # dynamically adjust the LoRA scale | ||
| if not USE_PEFT_BACKEND: | ||
| adjust_lora_scale_text_encoder(pipe.text_encoder, lora_scale) |
There was a problem hiding this comment.
Do we need this? Because without the PEFT backend, you cannot really do LoRA inference in the recent diffusers versions. No strong opinions either.
| if pipe.text_encoder_2 is not None: | ||
| if not USE_PEFT_BACKEND: | ||
| adjust_lora_scale_text_encoder(pipe.text_encoder_2, lora_scale) | ||
| else: | ||
| scale_lora_layers(pipe.text_encoder_2, lora_scale) |
There was a problem hiding this comment.
I just copied these lines from pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py, should i just leave scale_lora_layers(pipe.text_encoder_2, lora_scale) ?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@noskill let's fix the code quality issues and then we can merge. |
|
Thank you for your contributions! |
#8988) * handle lora scale and clip skip in lpw sd and sdxl * use StableDiffusionLoraLoaderMixin * use StableDiffusionXLLoraLoaderMixin * style --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
#8988) * handle lora scale and clip skip in lpw sd and sdxl * use StableDiffusionLoraLoaderMixin * use StableDiffusionXLLoraLoaderMixin * style --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
What does this PR do?
handle lora scale in cross_attention_kwargs and clip skip