Phi-3 Visio is a lightweight, state-of-the-art ope multimodal model built upo datasets which iclude - sythetic data ad filtered publicly available websites - with a focus o very high-quality, reasoig dese data both o text ad visio. The model belogs to the Phi-3 model family, ad the multimodal versio comes with 128K cotext legth (i tokes) it ca support. The model uderwet a rigorous ehacemet process, icorporatig both supervised fie-tuig ad direct preferece optimizatio to esure precise istructio adherece ad robust safety measures. Resources ad Techical Documetatio: The model is iteded for broad commercial ad research use i Eglish. The model provides uses for geeral purpose AI systems ad applicatios with visual ad text iput capabilities which require 1) memory/compute costraied eviromets;
2) latecy boud scearios;
3) geeral image uderstadig;
4) OCR;
5) chart ad table uderstadig. Our model is desiged to accelerate research o efficiet laguage ad multimodal models, for use as a buildig block for geerative AI powered features. Our models are ot specifically desiged or evaluated for all dowstream purposes. Developers should cosider commo limitatios of laguage models as they select use cases, ad evaluate ad mitigate for accuracy, safety, ad fairess before usig withi a specific dowstream use case, particularly for high-risk scearios.
Developers should be aware of ad adhere to applicable laws or regulatios (icludig privacy, trade compliace laws, etc.) that are relevat to their use case. Nothig cotaied i this Model Card should be iterpreted as or deemed a restrictio or modificatio to the licese the model is released uder. Phi-3-visio-128K-Istruct has bee itegrated i the developmet versio (4.40.2) of Whe loadig the model, esure that Update your local The curret Examples of required packages: Phi-3-Visio-128K-Istruct is also available i Azure AI Studio. Give the ature of the traiig data, the Phi-3-Visio-128K-Istruct model is best suited for a sigle image iput wih prompts usig the chat format as follows.
You ca provide the prompt as a sigle image with a geeric template as follow: where the model geerates the text after This code sippets show how to get quickly started with ruig the model o a GPU: Additioal basic examples are provided here. Like other models, the Phi family of models ca potetially behave i ways that are ufair, ureliable, or offesive. Some of the limitig behaviors to be aware of iclude: Developers should apply resposible AI best practices ad are resposible for esurig that a specific use case complies with relevat laws ad regulatios (e.g. privacy, trade, etc.). Importat areas for cosideratio iclude: Our traiig data icludes a wide variety of sources, ad is a combiatio of 1) publicly available documets filtered rigorously for quality, selected high-quality educatioal data ad code;
2) selected high-quality image-text iterleave;
3) ewly created sythetic, “textbook-like” data for the purpose of teachig math, codig, commo sese reasoig, geeral kowledge of the world (sciece, daily activities, theory of mid, etc.), ewly created image data, e.g., chart/table/diagram/slides;
4) high quality chat format supervised data coverig various topics to reflect huma prefereces o differet aspects such as istruct-followig, truthfuless, hoesty ad helpfuless. The data collectio process ivolved sourcig iformatio from publicly available documets, with a meticulous approach to filterig out udesirable documets ad images. To safeguard privacy, we carefully filtered various image ad text data sources to remove or scrub ay potetially persoal data from the traiig data. More details ca be foud i the Phi-3 Techical Report. To uderstad the capabilities, we compare Phi-3 Visio-128K-Istruct with a set of models over a variety of zero-shot bechmarks usig our iteral bechmark platform. Note that by default, the Phi-3-Visio-128K model uses flash attetio, which requires certai types of GPU hardware to ru. We have tested o the followig GPU types: To eable the model o these eviromet here are steps that you may cosider to follow: Step 1: commet flash attetio import code i modeligphi3v.py from lie 52 to lie 56. Step 2: chage "attimplemetatio" from "flashattetio2" to "eager" i cofig.jso or disable flash attetio whe you create the model as below. The model is licesed uder the MIT licese. This project may cotai trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to ad must follow Microsoft’s Trademark & Brad Guidelies. Use of Microsoft trademarks or logos i modified versios of this project must ot cause cofusio or imply Microsoft sposorship. Ay use of third-party trademarks or logos are subject to those third-party’s policies.Model Summary
Short Cotext
Log Cotext
Mii
4K [HF] ; [ONNX] ; [GGUF]
128K [HF] ; [ONNX]
Small
8K [HF] ; [ONNX]
128K [HF] ; [ONNX]
Medium
4K [HF] ; [ONNX]
128K [HF] ; [ONNX]
Visio
128K [HF]
Iteded Uses
How to Use
trasformers. Util the official versio is released through pip, esure that you are doig oe of the followig:
trust_remote_code=True is passed as a argumet of the from_pretraied() fuctio.trasformers to the developmet versio: pip uistall -y trasformers && pip istall git+https://github.com/huggigface/trasformers. The previous commad is a alterative to cloig ad istallig from the source.trasformers versio ca be verified with: pip list | grep trasformers.flash_att==2.5.8
umpy==1.24.4
Pillow==10.3.0
Requests==2.31.0
torch==2.3.0
torchvisio==0.18.0
trasformers==4.40.2
Chat Format
<|user|>\<|image_1|>\{prompt}<|ed|>\<|assistat|>\
<|assistat|> . I case of multi-tur coversatio, the prompt ca be formatted as follows:<|user|>\<|image_1|>\{prompt_1}<|ed|>\<|assistat|>\{respose_1}<|ed|>\<|user|>\{prompt_2}<|ed|>\<|assistat|>\
Sample iferece code
from PIL import Image
import requests
from modelscope import sapshot_dowload
from trasformers import AutoModelForCausalLM
from trasformers import AutoProcessor
model_id = sapshot_dowload("LLM-Research/Phi-3-visio-128k-istruct" )
model = AutoModelForCausalLM.from_pretraied(model_id, device_map="cuda", trust_remote_code=True, torch_dtype="auto")
processor = AutoProcessor.from_pretraied(model_id, trust_remote_code=True)
messages = [
{"role": "user", "cotet": "<|image_1|>\What is show i this image?"},
{"role": "assistat", "cotet": "The chart displays the percetage of respodets who agree with various statemets about their preparedess for meetigs. It shows five categories: 'Havig clear ad pre-defied goals for meetigs', 'Kowig where to fid the iformatio I eed for a meetig', 'Uderstadig my exact role ad resposibilities whe I'm ivited', 'Havig tools to maage admi tasks like ote-takig or summarizatio', ad 'Havig more focus time to sufficietly prepare for meetigs'. Each category has a associated bar idicatig the level of agreemet, measured o a scale from 0% to 100%."},
{"role": "user", "cotet": "Provide isightful questios to spark discussio."}
]
url = "https://assets-c4akfrf5b4d3f4b7.z01.azurefd.et/assets/2024/04/BMDataViz_661fb89f3845e.pg"
image = Image.ope(requests.get(url, stream=True).raw)
prompt = processor.tokeizer.apply_chat_template(messages, tokeize=False, add_geeratio_prompt=True)
iputs = processor(prompt, [image], retur_tesors="pt").to("cuda:0")
geeratio_args = {
"max_ew_tokes": 500,
"temperature": 0.0,
"do_sample": False,
}
geerate_ids = model.geerate(**iputs, eos_toke_id=processor.tokeizer.eos_toke_id, **geeratio_args)
# remove iput tokes
geerate_ids = geerate_ids[:, iputs['iput_ids'].shape[1]:]
respose = processor.batch_decode(geerate_ids, skip_special_tokes=True, clea_up_tokeizatio_spaces=False)[0]
prit(respose)
Resposible AI Cosideratios
Traiig
Model
Datasets
Bechmarks
Bechmark
Phi-3 Visio-128K-I
LlaVA-1.6 Vicua-7B
QWEN-VL Chat
Llama3-Llava-Next-8B
Claude-3 Haiku
Gemii 1.0 Pro V
GPT-4V-Turbo
MMMU
40.4
34.2
39.0
36.4
40.7
42.0
55.5
MMBech
80.5
76.3
75.8
79.4
62.4
80.0
86.1
ScieceQA
90.8
70.6
67.2
73.7
72.0
79.7
75.7
MathVista
44.5
31.5
29.4
34.8
33.2
35.0
47.5
IterGPS
38.1
20.5
22.3
24.6
32.1
28.6
41.0
AI2D
76.7
63.1
59.8
66.9
60.3
62.8
74.7
ChartQA
81.4
55.0
50.9
65.8
59.3
58.0
62.3
TextVQA
70.9
64.6
59.4
55.7
62.7
64.7
68.1
POPE
85.8
87.2
82.6
87.0
74.4
84.2
83.7
Software
Hardware
Ruig o Widows or without flash attetio
# if is_flash_att_2_available():
# from flash_att import flash_att_fuc, flash_att_varle_fuc
# from flash_att.bert_paddig import idex_first_axis, pad_iput, upad_iput # oqa
# _flash_supports_widow_size = "widow_size" i list(ispect.sigature(flash_att_fuc).parameters)
model = AutoModelForCausalLM.from_pretraied('microsoft/Phi-3-visio-128k-istruct', device_map="cuda", trust_remote_code=True, torch_dtype="auto", _att_implemetatio="eager")
Licese
Trademarks
点击空白处退出提示







评论