Skip to content

Commit a50ab8a

Browse files
nemonamelessnemonameless
authored
Fix docs of internvl2 and qwen2vl (#767)
1 parent acc1530 commit a50ab8a

10 files changed

+35
-148
lines changed

‎paddlemix/examples/internvl2/README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ python paddlemix/examples/internvl2/chat_demo.py \
2222
--text "Please describe this image in detail."
2323
```
2424
可配置参数说明：
25-
* `model_name_or_path`: 指定 internvl2 的模型名字或权重路径以及tokenizer组件，默认 OpenGVLab/InternVL2-8B
25+
* `model_name_or_path`: 指定 internvl2 的模型名字或权重路径以及tokenizer组件，默认 OpenGVLab/InternVL2-8B，也可选择 OpenGVLab/InternVL2-2B
2626
* `image_path`: 指定图片路径
2727
* `text`: 用户指令, 例如 "Please describe this image in detail."
2828

@@ -34,7 +34,7 @@ python paddlemix/examples/internvl2/chat_demo_video.py \
3434
--text "Please describe this video in detail."
3535
```
3636
可配置参数说明：
37-
* `model_name_or_path`: 指定 internvl2 的模型名字或权重路径以及tokenizer组件，默认 OpenGVLab/InternVL2-8B
37+
* `model_name_or_path`: 指定 internvl2 的模型名字或权重路径以及tokenizer组件，默认 OpenGVLab/InternVL2-8B，也可选择 OpenGVLab/InternVL2-2B
3838
* `video_path`: 指定视频路径
3939
* `text`: 用户指令, 例如 "Please describe this video in detail."
4040

@@ -47,9 +47,12 @@ SFT数据集采用 InternVL2 官方公布的1.3M的SFT数据集，包括了`shar
4747

4848
PaddleMIX团队整理后的下载链接为：
4949
```
50-
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground.tar
50+
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground.tar # 50G
51+
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/LLaVA/LLaVA-SFT.tar # 116G
5152
```
5253

54+
下载后可解压或软链接在 PaddleMIX/ 目录下。
55+
5356
PaddleMIX团队也提供了其中单独的`chartqa`数据集的下载链接，作为训练示例：
5457
```
5558
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground/data/chartqa.tar
@@ -59,6 +62,8 @@ chartqa.tar需下载解压在playground/data/目录下，opensource.tar需下载
5962

6063
### 4.2 微调命令
6164

65+
注意：此微调训练为全参数微调，冻结视觉编码器而放开LLM训练，2B模型微调训练的显存大小约为40G，8B模型微调训练的显存大小约为80G。
66+
6267
```bash
6368
# 1B
6469
sh paddlemix/examples/internvl2/shell/internvl2.0/2nd_finetune/internvl2_1b_qwen2_0_5b_dynamic_res_2nd_finetune_full.sh

‎paddlemix/examples/internvl2/shell/internvl2.0/2nd_finetune/internvl2_1b_qwen2_0_5b_dynamic_res_2nd_finetune_full.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ export PYTHONPATH="${PYTHONPATH}:$(pwd)"
1111
export MASTER_PORT=34229
1212
export TF_CPP_MIN_LOG_LEVEL=3
1313

14-
OUTPUT_DIR='work_dirs/internvl_chat_v2_0/internvl2_1b_qwen2_0_5b_dynamic_res_2nd_finetune_full'
14+
OUTPUT_DIR='work_dirs/internvl2-1B'
1515

1616
if [ ! -d "$OUTPUT_DIR" ]; then
1717
mkdir -p "$OUTPUT_DIR"

‎paddlemix/examples/internvl2/shell/internvl2.0/2nd_finetune/internvl2_2b_internlm2_1_8b_dynamic_res_2nd_finetune_full.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ export PYTHONPATH="${PYTHONPATH}:$(pwd)"
1111
export MASTER_PORT=34229
1212
export TF_CPP_MIN_LOG_LEVEL=3
1313

14-
OUTPUT_DIR='work_dirs/internvl_chat_v2_0/internvl2_2b_internlm2_1_8b_dynamic_res_2nd_finetune_full'
14+
OUTPUT_DIR='work_dirs/internvl2-2B'
1515

1616
if [ ! -d "$OUTPUT_DIR" ]; then
1717
mkdir -p "$OUTPUT_DIR"

‎paddlemix/examples/internvl2/shell/internvl2.0/2nd_finetune/internvl2_8b_internlm2_7b_dynamic_res_2nd_finetune_full.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ export PYTHONPATH="${PYTHONPATH}:$(pwd)"
1111
export MASTER_PORT=34229
1212
export TF_CPP_MIN_LOG_LEVEL=3
1313

14-
OUTPUT_DIR='work_dirs/internvl_chat_v2_0/internvl2_8b_internlm2_7b_dynamic_res_2nd_finetune_full'
14+
OUTPUT_DIR='work_dirs/internvl2-8B'
1515

1616
if [ ! -d "$OUTPUT_DIR" ]; then
1717
mkdir -p "$OUTPUT_DIR"

‎paddlemix/examples/minimonkey/README.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,20 +29,28 @@ python paddlemix/examples/minimonkey/chat_demo_minimonkey.py \
2929

3030
## 4 模型微调
3131

32-
SFT数据集采用 InternVL2 官方公布的1.3M的SFT数据集中的`llava_instruct_150k_zh`、`dvqa`、`chartqa`、`ai2d`、`docvqa`、`geoqa+`、`synthdog_en`共7个。
32+
### 4.1 微调数据准备
33+
34+
SFT数据集采用 InternVL2 官方公布的1.3M的SFT数据集中的`dvqa`、`chartqa`、`ai2d`、`docvqa`、`geoqa+`、`synthdog_en`共6个。
3335

3436
PaddleMIX团队整理后的下载链接为：
3537
```
36-
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground.tar
38+
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground.tar # 50G
3739
```
3840

41+
下载后可解压或软链接在 PaddleMIX/ 目录下。
42+
3943
PaddleMIX团队也提供了其中单独的`chartqa`数据集的下载链接，作为训练示例：
4044
```
4145
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground/data/chartqa.tar
4246
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground/opensource.tar
4347
```
4448
chartqa.tar需下载解压在playground/data/目录下，opensource.tar需下载解压在playground/目录下，opensource里是数据标注的jsonl文件。
4549

50+
### 4.2 微调命令
51+
52+
注意：此微调训练为全参数微调，冻结视觉编码器而放开LLM训练，2B模型微调训练的显存大小约为40G。
53+
4654
```bash
4755
sh paddlemix/examples/minimonkey/shell/internvl2.0/2nd_finetune/minimonkey_2b_internlm2_1_8b_dynamic_res_2nd_finetune_full.sh
4856
```

‎paddlemix/examples/minimonkey/shell/data/minimonkey_finetune.json

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,4 @@
11
{
2-
"llava_instruct_150k_zh": {
3-
"root": "playground/data/coco/",
4-
"annotation": "playground/opensource/llava_instruct_150k_zh.jsonl",
5-
"data_augment": false,
6-
"repeat_time": 1,
7-
"length": 157712
8-
},
92
"dvqa_train_200k": {
103
"root": "playground/data/dvqa/",
114
"annotation": "playground/opensource/dvqa_train_200k.jsonl",

‎paddlemix/examples/minimonkey/shell/data/minimonkey_finetune_chartqa.json

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"chartqa_train_18k": {
3+
"root": "playground/data/chartqa/",
4+
"annotation": "playground/opensource/chartqa_train_18k.jsonl",
5+
"data_augment": false,
6+
"repeat_time": 1,
7+
"length": 18317
8+
}
9+
}

‎paddlemix/examples/minimonkey/shell/internvl2.0/2nd_finetune/minimonkey_2b_internlm2_1_8b_dynamic_res_2nd_finetune_full.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ export PYTHONPATH="${PYTHONPATH}:$(pwd)"
1111
export MASTER_PORT=34229
1212
export TF_CPP_MIN_LOG_LEVEL=3
1313

14-
OUTPUT_DIR='work_dirs/minimonkey_2b_internlm2_1_8b_dynamic_res_2nd_finetune_full'
14+
OUTPUT_DIR='work_dirs/minimonkey-2B'
1515

1616
if [ ! -d "$OUTPUT_DIR" ]; then
1717
mkdir -p "$OUTPUT_DIR"
@@ -35,7 +35,7 @@ ${TRAINING_PYTHON} --log_dir ${OUTPUT_DIR}/paddle_distributed_logs \
3535
--conv_style "internlm2-chat" \
3636
--output_dir ${OUTPUT_DIR} \
3737
--logging_dir ${OUTPUT_DIR}/logs \
38-
--meta_path "paddlemix/examples/minimonkey/shell/data/minimonkey_finetune.json" \
38+
--meta_path "paddlemix/examples/minimonkey/shell/data/minimonkey_finetune_chartqa.json" \
3939
--overwrite_output_dir True \
4040
--force_image_size 448 \
4141
--max_dynamic_patch 12 \

‎paddlemix/examples/qwen2_vl/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,14 +44,14 @@ SFT数据集选择6个公开的数据集，包括`dvqa`、`chartqa`、`ai2d`、`
4444

4545
PaddleMIX团队整理后的下载链接为：
4646
```
47-
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground.tar
47+
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground.tar # 50G
4848
wget https://paddlenlp.bj.bcebos.com/datasets/paddlemix/playground/opensource_json.tar
4949
```
50-
opensource_json.tar需下载解压在playground/目录下，opensource里是数据标注的jsonl文件。
50+
opensource_json.tar需下载解压在playground/目录下，opensource_json 里是数据标注的json格式文件。
5151

5252
### 4.2 微调命令
5353

54-
注意：此微调训练为冻结视觉编码器而放开LLM训练的，2B模型微调训练的显存大小约为30G，7B模型微调训练的显存大小约为75G。
54+
注意：此微调训练为全参数微调，冻结视觉编码器而放开LLM训练，2B模型微调训练的显存大小约为30G，7B模型微调训练的显存大小约为75G。
5555

5656
```bash
5757
# 2B

‎paddlemix/examples/qwen2_vl/configs/add_llavaov_doc_ocr.json

Lines changed: 0 additions & 128 deletions
This file was deleted.

0 commit comments

Comments
 (0)