[NPU] Add llava npu doc (PaddlePaddle#694)

Co-authored-by: LokeZhou <aishenghuoaiqq@163.com>

Author: Birdylx

paddlemix/examples/README.md

@@ -4,23 +4,23 @@ paddlemix `examples` 目录下提供模型的一站式体验，包括模型推
 
 ## 模型能力矩阵
 
-| Model | Inference |Pretrain | SFT | LoRA | Deploy |
-| --- | --- | --- | --- | --- | --- |
-| [qwen_vl](./qwen_vl/) | ✅  | ❌  | ✅  | ✅  | ✅  |
-| [blip2](./blip2/) | ✅  | ✅ | ✅  | ✅ |  ❌ |
-| [clip](./clip) | ✅  | ✅ | ❌ | ❌ | ❌ |
-| [coca](./coca/) |  ✅  | ✅ | ❌ | ❌ | ❌ |
-| [eva02](./eva02/)|    ✅  |  ✅  |  ✅  |  ❌   | ❌   |
-| [evaclip](./evaclip/) | ✅ | ✅ | ❌ | ❌ |  ❌ |
-| [groundingdino](./groundingdino/) | ✅ | ❌  | 🚧   | ❌  | ✅  |
-| [imagebind](./imagebind/) |   ✅  |  ❌   |  ❌  | ❌ | ❌ |
-| [minigpt4](./minigpt4) | ✅ | ✅ | ✅   |  ❌  | ✅  |
-| [sam](./sam/) | ✅ | ❌ | ❌ | ❌ | ✅  |
-| [visualglm](./visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ |
-| [llava](./llava/) | ✅  | ✅  | ✅  | ✅  | 🚧  |
-| [YOLO-World](./YOLO-World/) | ✅  | ❌  | ❌  | ❌  | ❌ |
-| [CogVLM && CogAgent](./cogvlm/) | ✅ | ❌ | ❌ | ❌ | ❌ |
-| [InternLM-XComposer2](./internlm_xcomposer2/) | ✅ | ❌ | ✅ | ❌ | ❌ |
+| Model | Inference |Pretrain | SFT | LoRA | Deploy | NPU training |
+| --- | --- | --- | --- | --- | --- | --- | 
+| [qwen_vl](./qwen_vl/) | ✅  | ❌  | ✅  | ✅  | ✅  | ❌ |
+| [blip2](./blip2/) | ✅  | ✅ | ✅  | ✅ |  ❌ | ❌ |
+| [clip](./clip) | ✅  | ✅ | ❌ | ❌ | ❌ | ❌ |
+| [coca](./coca/) |  ✅  | ✅ | ❌ | ❌ | ❌ | ❌ |
+| [eva02](./eva02/)|    ✅  |  ✅  |  ✅  |  ❌   | ❌   | ❌ |
+| [evaclip](./evaclip/) | ✅ | ✅ | ❌ | ❌ |  ❌ | ❌ |
+| [groundingdino](./groundingdino/) | ✅ | ❌  | 🚧   | ❌  | ✅  | ❌ |
+| [imagebind](./imagebind/) |   ✅  |  ❌   |  ❌  | ❌ | ❌ | ❌ |
+| [minigpt4](./minigpt4) | ✅ | ✅ | ✅   |  ❌  | ✅  | ❌ |
+| [sam](./sam/) | ✅ | ❌ | ❌ | ❌ | ✅  | ❌ |
+| [visualglm](./visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
+| [llava](./llava/) | ✅  | ✅  | ✅  | ✅  | 🚧  | ✅ |
+| [YOLO-World](./YOLO-World/) | ✅  | ❌  | ❌  | ❌  | ❌ | ❌ |
+| [CogVLM && CogAgent](./cogvlm/) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| [InternLM-XComposer2](./internlm_xcomposer2/) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
 
 
 * ✅: Supported

paddlemix/tools/README.md

@@ -4,12 +4,12 @@ PaddleMIX工具箱秉承了飞桨套件一站式体验、性能极致、生态
 [[English](README_en.md)]
 
 ##  🛠️ 支持模型列表 🛠️
-| Model | Inference |Pretrain | SFT | LoRA | Deploy |
-| --- | --- | --- | --- | --- | --- |
-| [qwen_vl](../examples/qwen_vl/) | ✅  | ❌  | ✅  | ✅  |  ✅ |
-| [blip2](../examples/blip2/) | ✅  | ✅ | ✅  | ✅ | ✅  |
-| [visualglm](../examples/visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ |
-| [llava](../examples/llava/) | ✅  | ✅   | ✅  | ✅  | 🚧  |
+| Model | Inference |Pretrain | SFT | LoRA | Deploy | NPU training |
+| --- | --- | --- | --- | --- | --- | --- |
+| [qwen_vl](../examples/qwen_vl/) | ✅  | ❌  | ✅  | ✅  |  ✅ | ❌ |
+| [blip2](../examples/blip2/) | ✅  | ✅ | ✅  | ✅ | ✅  | ❌ |
+| [visualglm](../examples/visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
+| [llava](../examples/llava/) | ✅  | ✅   | ✅  | ✅  | 🚧  | ✅ |
 
 * ✅: Supported
 * 🚧: In Progress
@@ -74,6 +74,8 @@ PaddleMIX 精调支持多个主流跨模态大模型的SFT、LoRA等精调策略
 
     “mixtoken” : #是否使用mixtoken策略，默认False,
 
+    "device": #训练硬件，npu、gpu
+
     "output_dir":  #模型存储路径
 
     "overwrite_output_dir": # 覆盖输出目录，默认False
@@ -188,3 +190,17 @@ python paddlemix/tools/merge_lora_params.py \
 --lora_path output_qwen_vl\
 --merge_model_path qwen_vl_merge
 ```
+
+**NPU硬件训练**
+
+PaddleMIX支持在NPU硬件上训练，只需要在config配置文件中增加`device`字段制定设备即可：
+```json
+{
+    ...
+    "model_name_or_path": "paddlemix/llava/llava-v1.5-7b",
+    "device": "npu",
+    "output_dir": "./checkpoints/llava_sft_ckpts",
+    ...
+}
+```
+目前支持NPU训练的模型可以参考此[文档](../examples/README.md)

paddlemix/tools/README_en.md

@@ -4,12 +4,12 @@ The PaddleMIX toolkit embodies the design philosophy of one-stop experience, ult
 [[中文文档](README.md)]
 
 ##  🛠️ Supported Model List 🛠️
-| Model | Inference |Pretrain | SFT | LoRA | Deploy |
-| --- | --- | --- | --- | --- | --- |
-| [qwen_vl](../examples/qwen_vl/) | ✅  | ❌  | ✅  | ✅  |  ✅ |
-| [blip2](../examples/blip2/) | ✅  | ✅ | ✅  | ✅ | ✅  |
-| [visualglm](../examples/visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ |
-| [llava](../examples/llava/) | ✅  | ✅   | ✅  | ✅  | 🚧  |
+| Model | Inference |Pretrain | SFT | LoRA | Deploy | NPU training |
+| --- | --- | --- | --- | --- | --- | --- |
+| [qwen_vl](../examples/qwen_vl/) | ✅  | ❌  | ✅  | ✅  |  ✅ | ❌ |
+| [blip2](../examples/blip2/) | ✅  | ✅ | ✅  | ✅ | ✅  | ❌ |
+| [visualglm](../examples/visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
+| [llava](../examples/llava/) | ✅  | ✅   | ✅  | ✅  | 🚧  | ✅ |
 
 * ✅: Supported
 * 🚧: In Progress
@@ -70,6 +70,8 @@ For convenience in testing, we also provide a dataset in the chatml_dataset form
 
     “mixtoken” : #是否使用mixtoken策略，默认False,
 
+    "device": #训练硬件，gpu、npu
+
     "output_dir":  #模型存储路径
 
     "overwrite_output_dir": # 覆盖输出目录，默认False
@@ -177,3 +179,17 @@ python paddlemix/tools/merge_lora_params.py \
 --lora_path output_qwen_vl\
 --merge_model_path qwen_vl_merge
 ```
+
+**NPU training**
+
+PaddleMIX supports training on NPU device，you cat add the `device` field in the configuration file to specify the device:
+```json
+{
+    ...
+    "model_name_or_path": "paddlemix/llava/llava-v1.5-7b",
+    "device": "npu",
+    "output_dir": "./checkpoints/llava_sft_ckpts",
+    ...
+}
+```
+For models that support NPU training, please refer to this [doc](../examples/README.md)

paddlemix/tools/supervised_finetune.py

@@ -62,11 +62,16 @@ def main():
                 "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
             )
 
+    if "npu" in training_args.device:
+        is_bfloat16_supported = True
+    else:
+        is_bfloat16_supported = paddle.amp.is_bfloat16_supported():
+
     # Load model
     if training_args.fp16_opt_level == "O2":
         if training_args.fp16:
             dtype = "float16"
-        elif training_args.bf16 and paddle.amp.is_bfloat16_supported():
+        elif training_args.bf16 and is_bfloat16_supported:
             dtype = "bfloat16"
         else:
             raise ValueError("Please specific dtype: --fp16 or --bf16")