added mllama doc

docs/source/en/model_doc/mllama.md

@@ -14,93 +14,110 @@ rendered properly in your Markdown viewer.
 
 -->
 
-# Mllama
-
-<div class="flex flex-wrap space-x-1">
-<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+        <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+    </div>
 </div>
 
-## Overview
-
-The Llama 3.2-Vision collection of multimodal large language models (LLMs) is a collection of pretrained and instruction-tuned image reasoning generative models in 11B and 90B sizes (text \+ images in / text out). The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image.
-
-**Model Architecture:** Llama 3.2-Vision is built on top of Llama 3.1 text-only model, which is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. To support image recognition tasks, the Llama 3.2-Vision model uses a separately trained vision adapter that integrates with the pre-trained Llama 3.1 language model. The adapter consists of a series of cross-attention layers that feed image encoder representations into the core LLM.
-
-## Usage Tips
-
-- For image+text and text inputs use `MllamaForConditionalGeneration`.
-- For text-only inputs use `MllamaForCausalLM` for generation to avoid loading vision tower.
-- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images across samples and to a maximum number of tiles within each image.
-- The text passed to the processor should have the `"<|image|>"` tokens where the images should be inserted.
-- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as text to the processor. If you're using `transformers>=4.49.0`, you can also get a vectorized output from `apply_chat_template`. See the **Usage Examples** below for more details on how to use it.
-
+# MLlama
 
+[MLlama](https://huggingface.co/papers/2407.21783), or Llama 3.2 Vision, is a multimodal version of [Llama 3](./llama3), available in 11B and 90B parameters, and as an instruction-tuned and base variant. This model integrates a separately trained vision adapter (image encoder, image adapter, and video adapter) to handle visual tasks like image reasoning, captioning, and answering. The instruction-tuned variants are post-trained with supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human helpfulness and safety values.
 
-<Tip warning={true}>
+You can find all the original MLlama checkpoints under the [Llama 3.2](https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf) collection.
 
-Mllama has an extra token used as a placeholder for image positions in the text. It means that input ids and an input embedding layer will have an extra token. But since the weights for input and output embeddings are not tied, the `lm_head` layer has one less token and will fail if you want to calculate loss on image tokens or apply some logit processors. In case you are training, make sure to mask out special `"<|image|>"` tokens in the `labels` as the model should not be trained on predicting them.
+> [!TIP]
+> Click on the MLlama models in the right sidebar for more examples of how to apply MLlama to different vision-language tasks like image captioning, visual question answering, and reasoning.
 
-Otherwise if you see CUDA-side index erros when generating, use the below code to expand the `lm_head` by one more token. 
+The example below demonstrates how to generate text based on an image with [`Pipeline`] or the [`AutoModel`] class.
 
+<hfoptions id="usage">
+<hfoption id="Pipeline">
 
 ```python
-old_embeddings = model.get_output_embeddings()
-
-num_tokens = model.vocab_size + 1
-resized_embeddings = model._get_resized_lm_head(old_embeddings, new_num_tokens=num_tokens, mean_resizing=True)
-resized_embeddings.requires_grad_(old_embeddings.weight.requires_grad)
-model.set_output_embeddings(resized_embeddings)
+import torch
+from transformers import pipeline
+
+pipeline = pipeline(
+    task="image-text-to-text",
+    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
+    device=0,
+    torch_dtype=torch.bfloat16
+)
+messages = [
+    [
+        {
+            "role": "user", 
+            "content": [
+                {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
+                {"type": "text", "text": "What does the image show?"}
+            ]
+        }
+    ],
+]
+pipeline(text=messages, return_full_text=False)
 ```
-</Tip>
-
 
-## Usage Example
+</hfoption>
+<hfoption id="AutoModel">
 
-#### Instruct model
 ```python
 import torch
-from transformers import MllamaForConditionalGeneration, AutoProcessor
+from transformers import BitsAndBytesConfig, MllamaForConditionalGeneration, AutoProcessor
 
-model_id = "meta-llama/Llama-3.2-11B-Vision-Instruct"
-model = MllamaForConditionalGeneration.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16)
-processor = AutoProcessor.from_pretrained(model_id)
+model = MllamaForConditionalGeneration.from_pretrained(
+    "meta-llama/Llama-3.2-11B-Vision-Instruct",
+    device_map="auto", 
+    torch_dtype=torch.bfloat16,
+    attn_implementation="sdpa"
+)
+processor = AutoProcessor.from_pretrained("meta-llama/Llama-3.2-11B-Vision-Instruct")
 
 messages = [
     [
         {
             "role": "user", 
             "content": [
-                {"type": "image", "url": "https://llava-vl.github.io/static/images/view.jpg"},
+                {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
                 {"type": "text", "text": "What does the image show?"}
             ]
         }
     ],
 ]
-inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
+inputs = processor.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt"
+).to("cuda")
 output = model.generate(**inputs, max_new_tokens=25)
 print(processor.decode(output[0]))
 ```
 
-#### Base model
-```python
-import requests
-import torch
-from PIL import Image
-from transformers import MllamaForConditionalGeneration, AutoProcessor
+</hfoption>
+</hfoptions>
 
-model_id = "meta-llama/Llama-3.2-11B-Vision"
-model = MllamaForConditionalGeneration.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16)
-processor = AutoProcessor.from_pretrained(model_id)
+<div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mllama_architecture.png"/>
+</div>
 
-prompt = "<|image|>If I had to write a haiku for this one"
-url = "https://llava-vl.github.io/static/images/view.jpg"
-raw_image = Image.open(requests.get(url, stream=True).raw)
+## Notes
 
-inputs = processor(text=prompt, images=raw_image, return_tensors="pt").to(model.device)
-output = model.generate(**inputs, do_sample=False, max_new_tokens=25)
-print(processor.decode(output[0], skip_special_tokens=True))
-```
+- Use [`MllamaForConditionalGeneration`] for image-text and text inputs.
+- Use [`MllamaForCausalLM`] for generation with text only inputs to avoid unnecessarily loading the vision components.
+- Samples can contain multiple images and each sample can have different number of images. In these cases, the processor pads the inputs to the maximum number of images across samples and to a maximum number of tiles within each image.
+- Use the `<|image|>` token to indicate where an image should be inserted.
+- MLlama's input and output embeddings are not tied which means the `lm_head` layer has one less token (the `<|image|>` placeholder token) and fails if you want to calculate loss on image tokens or apply logit processors. For training, make sure to mask the `<|image|>` token in `labels` because the model should not be trained on predicting them.
+- For CUDA index errors during generation, expand the `lm_head` by one token:
 
+```python
+old_embeddings = model.get_output_embeddings()
+num_tokens = model.vocab_size + 1
+resized_embeddings = model._get_resized_lm_head(old_embeddings, new_num_tokens=num_tokens, mean_resizing=True)
+resized_embeddings.requires_grad_(old_embeddings.weight.requires_grad)
+model.set_output_embeddings(resized_embeddings)
+```
 
 ## MllamaConfig
 
@@ -110,7 +127,6 @@ print(processor.decode(output[0], skip_special_tokens=True))
 
 [[autodoc]] MllamaProcessor
 
-
 ## MllamaImageProcessor
 
 [[autodoc]] MllamaImageProcessor
@@ -125,17 +141,17 @@ print(processor.decode(output[0], skip_special_tokens=True))
 [[autodoc]] MllamaForCausalLM
     - forward
 
+## MllamaVisionModel
+
+[[autodoc]] MllamaVisionModel
+    - forward
+
 ## MllamaTextModel
 
 [[autodoc]] MllamaTextModel
     - forward
 
-## MllamaForCausalLM
 
-[[autodoc]] MllamaForCausalLM
-    - forward
 
-## MllamaVisionModel
 
-[[autodoc]] MllamaVisionModel
-    - forward
+