update readme

README.md

@@ -3,11 +3,11 @@ library_name: transformers
 pipeline_tag: text-generation
 ---
 
-# Model Card for Model ID
+# Model Card for Magma-8B
 
 <!-- Provide a quick summary of what the model is/does. -->
 
-
+A multimodal agentic foundation for multimodal understanding and agentic tasks.
 
 ## Model Details
 
@@ -15,63 +15,142 @@ pipeline_tag: text-generation
 
 <!-- Provide a longer summary of what this model is. -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+Magma is a multimodal agentic AI model that can generate text based on the input text and image. The model is designed for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, in particular the multimodal agentic AI. The main innovation of this model lies on the introduction of two technical innovations: Set-of-Mark and Trace-of-Mark, and the leverage of a large-amount of unlabeled video data to learn the spatial-temporal grounding and planning. Please refer to our paper for more technical details. 
+
+The model is developed by Microsoft and is funded by Microsoft Research. 
+
+The model is shared by Microsoft Research and is licensed under the MIT License. 
+
+The model is developed based on Meta LLama-3 as the LLM.
+
+<!-- {{ model_description | default("", true) }}
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** {{ developers | default("[More Information Needed]", true)}}
+- **Funded by [optional]:** {{ funded_by | default("[More Information Needed]", true)}}
+- **Shared by [optional]:** {{ shared_by | default("[More Information Needed]", true)}}
+- **Model type:** {{ model_type | default("[More Information Needed]", true)}}
+- **Language(s) (NLP):** {{ language | default("[More Information Needed]", true)}}
+- **License:** {{ license | default("[More Information Needed]", true)}}
+- **Finetuned from model [optional]:** {{ base_model | default("[More Information Needed]", true)}} -->
 
-### Model Sources [optional]
+### Model Sources
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** [Magma Github Repo]("https://github.com/microsoft/Magma")
+<!-- - **Paper [optional]:** {{ paper | default("[More Information Needed]", true)}} -->
+<!-- - **Demo [optional]:** {{ demo | default("[More Information Needed]", true)}} -->
 
-## Uses
+## Intended Uses
 
 <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
+This model is intended for broad research use in English. It is designed only for research purposes and aimed at knowledge-sharing and accelerating research in multimodal AI, particularly in mutimodal agentic AI. It is intended to be used by domain experts who are independently capable of evaluating the quality of outputs before acting on them.
+
 ### Direct Use
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-[More Information Needed]
+The model takes images and text as inputs, and produces the textual outputs for the following uses:
+
+* **Image/Video-Conditoned Text Generation:** The model can generate text (e.g., descriptions, answers) based on the input text and image.
+
+* **Visual Planning Capabilities:** The model can also produce the visual trace as the future planning to accomplish a task (e.g., move object from one place to another).
+
+* **Agentic Capabilities:** The model can also generate UI grounding (e.g., click ``search'' button) and robotics manipulations (e.g., 7 DoF for the robot gripper).
+
 
-### Downstream Use [optional]
+
+### Downstream Use
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-[More Information Needed]
+<!-- {{ downstream_use | default("[More Information Needed]", true)}} -->
 
-### Out-of-Scope Use
+<!-- ### Out-of-Scope Use -->
 
 <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-[More Information Needed]
+<!-- {{ out_of_scope_use | default("[More Information Needed]", true)}} -->
+
+The model can be further finetuned for different downstream tasks, such as:
+
+* **Image Captioning and QA:** We can further finetune this model for image captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better spatial understanding and reasoning on these tasks.
+
+* **Video Captioning and QA:** We can further finetune this model for video captioning and QA tasks under the pipeline of multimodal LLMs. Based on our experiments, the model can achieve competitive performance yet better temporal understanding and reasoning on these tasks.
+
+* **UI Navigation:** We can finetune this model for specific UI navigation tasks, such as web navigation or mobile navigation. The model can achieve superior performance on these tasks.
+
+* **Robotics Manipulation:** Our model can be further finetuned for robotics tasks given its general agentic capabilities as a vision-language-action model. After finetuning, our model significantly outperms the state-of-the-art models such as OpenVLA on robotics manipulation tasks.
+
 
 ## Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
+<!-- {{ bias_risks_limitations | default("[More Information Needed]", true)}} -->
+
+Please note that this model is not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios. Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case.
+
+
 
 ### Recommendations
 
 <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+<!-- {{ bias_recommendations | default("Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.", true)}} -->
+
+Magma was developed for research purposes only. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+The recommended usage for the finetuned models is within the research settings they were trained on — namely, 
+-	an android simulator running on a computer for UI manipulation.
+-	an enclosure equipped with a robotic arm and everyday objects for Robotic manipulation
+
+For UI navigation task, researchers should make sure a human is in the loop and in control for every action the agentic system generates. Since the model cannot act by itself, the sub-module a researcher uses to actually perform the UI navigation action should ensure no unintended consequences can occur as a result of performing the UI action proposed by the model.
+
+For the robotic manipulation task, some mitigation strategies to use for human safety when operating robotic arms include:
+
+* **Safety Zones and Barriers:** Establish physical barriers or safety zones around robotic workspaces to prevent unauthorized access.
+* **Emergency Stop Systems:** Equip robotic arms with easily accessible emergency stop buttons. Implement a fail-safe mechanism that triggers an immediate stop of operations in case of an emergency
+* **Safety Standards and Compliance:** Adhere to established safety standards (e.g., ISO 10218, ISO/TS 15066) for industrial robots and collaborative robots.
+* **User Training and Awareness:** Provide comprehensive training for all personnel working around robotic arms to understand their functions, safety features, and emergency procedures. Promote awareness of the potential risks associated with robotic manipulation.
 
 ## How to Get Started with the Model
 
+<!-- {{ get_started_code | default("[More Information Needed]", true)}} -->
+
 Use the code below to get started with the model.
 
-[More Information Needed]
+```python
+from magma.image_processing_magma import MagmaImageProcessor
+from magma.processing_magma import MagmaProcessor
+from magma.modeling_magma import MagmaForConditionalGeneration
+
+# Load the model
+model_id = "microsoft/Magma-8B"
+processor = MagmaProcessor.from_pretrained(model_id, trust_remote_code=True) 
+model = MagmaForConditionalGeneration.from_pretrained(model_id, device_map="cuda", low_cpu_mem_usage=True)
+
+# Inference
+url = "https://assets-c4akfrf5b4d3f4b7.z01.azurefd.net/assets/2024/04/BMDataViz_661fb89f3845e.png" 
+image = Image.open(requests.get(url, stream=True).raw) 
+
+convs = [
+    {"role": "system", "content": "You are agent that can see, talk and act."},            
+    {"role": "user", "content": "<image_start><image><image_end>\nWhat is in this image?"},
+]
+prompt = processor.tokenizer.apply_chat_template(convs, tokenize=False, add_generation_prompt=True)
+inputs = processor(images=[image], texts=prompt, return_tensors="pt")
+inputs = inputs.to("cuda")
+
+with torch.inference_mode():
+    generate_ids = model.generate(**inputs, **generation_args)
+
+generate_ids = generate_ids[:, inputs["input_ids"].shape[-1] :]
+response = processor.decode(generate_ids[0], skip_special_tokens=True).strip()
+
+print(response)
+```
 
 ## Training Details
 
@@ -79,121 +158,166 @@ Use the code below to get started with the model.
 
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-[More Information Needed]
+<!-- {{ training_data | default("[More Information Needed]", true)}} -->
+
+Our training data consists of:
+
+* Generic Image SFT Data: [LLaVA-Next](https://llava-vl.github.io/blog/2024-01-30-llava-next/), [InfoGrpahicVQA](https://www.docvqa.org/datasets/infographicvqa), [ChartQA_Augmented](https://github.com/vis-nlp/ChartQA), [FigureQA](https://www.microsoft.com/en-us/research/project/figureqa-dataset/), [TQA](https://paperswithcode.com/dataset/tqa), [ScienceQA](https://scienceqa.github.io/).
+
+* Generic Video SFT Data: [ShareGPT4Video](https://sharegpt4video.github.io/) and [LLaVA-Video](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K).
+
+* Instructional Video Data: [Ego4d](https://ego4d-data.org/), [Somethingv2](https://www.qualcomm.com/developer/software/something-something-v-2-dataset), [Epic-Kitchen](https://epic-kitchens.github.io/2025), [COIN](https://coin-dataset.github.io/).
+
+* Robotics Manipulation Data: [Open-X-Embodiment](https://robotics-transformer-x.github.io/).
+
+* UI Grounding Data: [SeeClick](https://github.com/njucckevin/SeeClick).
+
+* UI Navigation Data: [Mind2web](https://osu-nlp-group.github.io/Mind2Web/) and [AITW](https://github.com/google-research/google-research/tree/master/android_in_the_wild).
+
+The data collection process involved sourcing information from publicly available documents, with a meticulous approach to filtering out undesirable documents and images. To safeguard privacy, we carefully filtered various image and text data sources to remove or scrub any potentially personal data from the training data.
+
+More details can be found in our paper.
+
+[Microsoft Privacy Notice](https://go.microsoft.com/fwlink/?LinkId=521839)
 
 ### Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-#### Preprocessing [optional]
+#### Preprocessing
 
-[More Information Needed]
+<!-- {{ preprocessing | default("[More Information Needed]", true)}} -->
+In addition to the text-related preprocessing, we mainly undertake the following image and video preprocessing steps:
 
+* UI Grounding and Navigation Data: For each UI screenshot, we extract the bounding boxes for the UI elements, and apply [Set-of-Mark Prompting](https://arxiv.org/abs/2310.11441) to overlay numeric marks on the raw image. The model is trained to generate the UI grounding text based on the image and the Set-of-Mark prompts.
+
+* Instruction Video Data: For each video clip, we apply [Co-Tracker](https://co-tracker.github.io/) to extract the grid traces and then apply filtering algorithm to remove the noisy or staic points. For videos that bear camera motion, we further apply homography transformation to stabilize the video clips. In tne end, we assign a numeric mark for each trace which gives us a set of trace-of-mark. The model is trained to generate the trace-of-mark given the video clips and instructional text.
+
+* Robotics Manipulation Data: For robotics data in Open-X Embodiment, we extract the 7 DoF robot gripper state and also extract the trace-of-mark from the video clips. Similar filtering and stabilization steps are applied to the video clips. The model is trained to generate the robot manipulation action as well as the trace-of-mark given the video clips and instructional text.
+
+After all these preprocessing, we combine them with existing text annotations to form our final multimodal training data. We refer to our paper for more technical details.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+<!-- - **Training regime:** {{ training_regime | default("[More Information Needed]", true)}} fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-#### Speeds, Sizes, Times [optional]
+We used bf16 mixed precision for training on H100s and MI300s. We used the following hyperparameters for training:
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+* Batch size: 1024
+* Learning rate: 1e-5
+* Max sequence length: 4096
+* Resolution: maximally 1024x1024 for image, 512x512 for video frame.
+* Pretraining Epochs: 3
 
-[More Information Needed]
 
 ## Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
+We evaluate the model in zero-shot manner on a wide range of tasks, mostly agent-related tasks.
 
 ### Testing Data, Factors & Metrics
-
-#### Testing Data
-
 <!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
+<!-- {{ testing_data | default("[More Information Needed]", true)}} -->
 
-#### Factors
+<!-- #### Factors
 
 <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
 
-[More Information Needed]
+<!-- {{ testing_factors | default("[More Information Needed]", true)}} -->
 
-#### Metrics
+#### Zero-shot Testing Data
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+We evaluate the model's zero-shot performance on the following datasets:
 
-[More Information Needed]
+* UI Grounding: [ScreenSpot](https://huggingface.co/datasets/rootsautomation/ScreenSpot) and [VisualWebArena](https://jykoh.com/vwa).
 
-### Results
+* Robotics Manipulation: [SimplerEnv](https://jykoh.com/vwa) and WidowX real robot.
 
-[More Information Needed]
+* Spatial Understanding and Reasoning: [VSR](https://github.com/cambridgeltl/visual-spatial-reasoning), [BLINK](https://zeyofu.github.io/blink/) and [SpatialEval](https://spatialeval.github.io/).
 
-#### Summary
 
 
+#### Finetuned Testing Data
 
-## Model Examination [optional]
+We evaluate the model's performance after finetuning on the following datasets:
 
-<!-- Relevant interpretability work for the model goes here -->
+* UI Navigation: [Mind2Web](https://osu-nlp-group.github.io/Mind2Web/) and [AITW](https://github.com/google-research/google-research/tree/master/android_in_the_wild).
 
-[More Information Needed]
+* Robotics Manipulation: [SimplerEnv](https://github.com/simpler-env/SimplerEnv) and WidowX real robot.
 
-## Environmental Impact
+* Multimodal Image Understanding and Reasoning: [VQAv2](https://visualqa.org/), [GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html), [MME](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation), [POPE](https://huggingface.co/datasets/lmms-lab/POPE), [TextVQA](https://textvqa.org/), [ChartQA](https://github.com/vis-nlp/ChartQA), [DocVQA](https://www.docvqa.org/).
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+* Multimodal Video Understanding and Reasoning: [Next-QA](https://github.com/doc-doc/NExT-QA), [VideoMME](https://video-mme.github.io/home_page.html), [MVBench](https://arxiv.org/abs/2311.17005).
+
+#### Metrics
+<!-- {{ testing_metrics | default("[More Information Needed]", true)}} -->
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+We follow the individual dataset's evaluation metrics for the evaluation. Please refer to the original dataset for more details.
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
+### Results
+
+TBD
+<!-- {{ results | default("[More Information Needed]", true)}} -->
+
+#### Summary
+
+TBD
+<!-- {{ results_summary | default("", true) }} -->
+
+## Technical Specifications 
+
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+<!-- {{ model_specs | default("[More Information Needed]", true)}} -->
 
-### Compute Infrastructure
+* Language Model: We use [Meta LLama-3](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) as the backbone LLM.
+* Vision Encoder: We use [CLIP-ConvneXt-XXLarge](https://huggingface.co/laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg) trained by LAION team as the vision encoder to tokenize the images and videos.
 
-[More Information Needed]
+The whole pipeline follows the common practice in the multimodal LLMs, where the vision encoder is used to tokenize the images and videos, and then the visual tokens are fed into the LLM along with the textal tokens to generate the text outputs.
 
-#### Hardware
 
-[More Information Needed]
+### Compute Infrastructure
+<!-- {{ compute_infrastructure | default("[More Information Needed]", true)}} -->
 
-#### Software
+We used [Azure ML](https://azure.microsoft.com/en-us/products/machine-learning) for our model training.
 
-[More Information Needed]
 
-## Citation [optional]
+#### Hardware
+<!-- {{ hardware_requirements | default("[More Information Needed]", true)}} -->
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+Our model is trained on two GPUs:
 
-**BibTeX:**
+* Nvidia H100
+* AMD MI300
 
-[More Information Needed]
 
-**APA:**
 
-[More Information Needed]
+#### Software
+<!-- {{ software | default("[More Information Needed]", true)}} -->
 
-## Glossary [optional]
+Our model is built based on:
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+* [Pytorch](https://pytorch.org/)
+* [Transformers](https://huggingface.co/transformers/)
+* [TorchVision](https://pytorch.org/vision/stable/index.html)
+* [DeepSpeed](https://www.deepspeed.ai/)
+* [FlashAttenton](https://github.com/HazyResearch/flash-attention)
 
-[More Information Needed]
 
-## More Information [optional]
 
-[More Information Needed]
+## Citation [optional]
 
-## Model Card Authors [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
 
-[More Information Needed]
+N/A
+<!-- {{ citation_bibtex | default("[More Information Needed]", true)}} -->
 
-## Model Card Contact
+**APA:**
 
-[More Information Needed]
+N/A
+<!-- {{ citation_apa | default("[More Information Needed]", true)}} -->