Skip to content

Refactor CLIP&ErnieViL model and Add ChineseCLIP Model #4270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jan 6, 2023

Conversation

JunnYu
Copy link
Member

@JunnYu JunnYu commented Dec 28, 2022

PR types

Others

PR changes

Models

Description

  • 重构CLIP模型,支持使用Config加载,兼容旧版model_config.json文件。(权重参数命名没有变化,只修改了Config的参数名);
  • 添加CLIP和ErnieViL的单测(其中 ErnieViL的tokenizer与Ernie一致,因此并未添加单测。)。
  • 新增ChineseCLIP模型:
  • ChineseCLIP 当前支持的权重
CHINESE_CLIP_PRETRAINED_MODEL_ARCHIVE_LIST = [
    "OFA-Sys/chinese-clip-vit-base-patch16",
    "OFA-Sys/chinese-clip-vit-huge-patch14",
    "OFA-Sys/chinese-clip-vit-large-patch14",
    "OFA-Sys/chinese-clip-vit-large-patch14-336px",
]
  • CLIP当前支持的权重:
CLIP_PRETRAINED_MODEL_ARCHIVE_LIST = [
    "openai/clip-vit-base-patch32",  # ViT-B/32
    "openai/clip-vit-base-patch16",  # ViT-B/16
    "openai/clip-vit-large-patch14",  # ViT-L/14
    "laion/CLIP-ViT-H-14-laion2B-s32B-b79K",
    "laion/CLIP-ViT-B-32-laion2B-s34B-b79K",
    "openai/clip-rn50",  # RN50
    "openai/clip-rn101",  # RN101
    "openai/clip-rn50x4",  # RN50x4
]
  • 重构ErnieViL模型,支持使用Config加载,由于该模型使用较少,因此未兼容旧版model_config.json文件,直接在community底下新增了权重文件。(方便下载,未来的新模型几乎都会从 community 目录下下载。)
ERNIE_VIL_PRETRAINED_MODEL_ARCHIVE_LIST = [
    "PaddlePaddle/ernie_vil-2.0-base-zh",
    "PaddlePaddle/disco_diffusion_ernie_vil-2.0-base-zh",
]
  • 同步修改TaskFlow中 ERNIE_VIL 相关的链接。
  • 修改DPT及BIT的Config默认值,return_dict=True
  • 重命名 PreTrainedModel 成 PretrainedModel,大写的T改成小写。

@paddle-bot
Copy link

paddle-bot bot commented Dec 28, 2022

Thanks for your contribution!

@JunnYu
Copy link
Member Author

JunnYu commented Dec 28, 2022

TODO:ERNIEVIl还需要更新一下

DONE

@codecov
Copy link

codecov bot commented Dec 28, 2022

Codecov Report

Merging #4270 (f0d1394) into develop (943a0ff) will increase coverage by 0.99%.
The diff coverage is 61.32%.

@@             Coverage Diff             @@
##           develop    #4270      +/-   ##
===========================================
+ Coverage    38.67%   39.67%   +0.99%     
===========================================
  Files          421      432      +11     
  Lines        59600    60739    +1139     
===========================================
+ Hits         23052    24097    +1045     
- Misses       36548    36642      +94     
Impacted Files Coverage Δ
paddlenlp/taskflow/taskflow.py 75.29% <ø> (ø)
paddlenlp/transformers/auto/modeling.py 71.88% <ø> (ø)
paddlenlp/transformers/auto/tokenizer.py 81.74% <ø> (ø)
paddlenlp/transformers/chineseclip/converter.py 0.00% <0.00%> (ø)
paddlenlp/transformers/clip/converter.py 0.00% <0.00%> (ø)
paddlenlp/transformers/clip/modeling.py 60.61% <ø> (+37.96%) ⬆️
...lenlp/transformers/chineseclip/image_processing.py 20.48% <20.48%> (ø)
...ddlenlp/transformers/ernie_vil/image_processing.py 20.48% <20.48%> (ø)
...ransformers/guided_diffusion_utils/resize_right.py 21.39% <33.33%> (-0.66%) ⬇️
paddlenlp/transformers/chineseclip/procesing.py 38.77% <38.77%> (ø)
... and 25 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@JunnYu JunnYu changed the title Refactor CLIP model Refactor CLIP&ErnieViL model Dec 29, 2022
@JunnYu JunnYu changed the title Refactor CLIP&ErnieViL model Refactor CLIP&ErnieViL model and Add ChineseCLIP Model Jan 3, 2023
Copy link
Contributor

@wj-Mcat wj-Mcat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你的这个 PR 太大了,看起来很花费时间,所以我只留了几个 comment。

至于 converter 这块我还是会继续兼容你这边的工作,保证你这边能够正常工作。

@JunnYu JunnYu requested a review from wj-Mcat January 6, 2023 06:49
@wj-Mcat wj-Mcat merged commit 8269220 into PaddlePaddle:develop Jan 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants