Phi-3-vision-128k-instruct

我要开发同款
匿名用户2024年07月31日
179阅读

技术信息

开源地址
https://modelscope.cn/models/LLM-Research/Phi-3-vision-128k-instruct
授权协议
mit

作品详情

Model Summary

Phi-3 Visio is a lightweight, state-of-the-art ope multimodal model built upo datasets which iclude - sythetic data ad filtered publicly available websites - with a focus o very high-quality, reasoig dese data both o text ad visio. The model belogs to the Phi-3 model family, ad the multimodal versio comes with 128K cotext legth (i tokes) it ca support. The model uderwet a rigorous ehacemet process, icorporatig both supervised fie-tuig ad direct preferece optimizatio to esure precise istructio adherece ad robust safety measures.

Resources ad Techical Documetatio:

Short Cotext Log Cotext
Mii 4K [HF] ; [ONNX] ; [GGUF] 128K [HF] ; [ONNX]
Small 8K [HF] ; [ONNX] 128K [HF] ; [ONNX]
Medium 4K [HF] ; [ONNX] 128K [HF] ; [ONNX]
Visio 128K [HF]

Iteded Uses

Primary use cases

The model is iteded for broad commercial ad research use i Eglish. The model provides uses for geeral purpose AI systems ad applicatios with visual ad text iput capabilities which require

1) memory/compute costraied eviromets; 2) latecy boud scearios; 3) geeral image uderstadig; 4) OCR; 5) chart ad table uderstadig.

Our model is desiged to accelerate research o efficiet laguage ad multimodal models, for use as a buildig block for geerative AI powered features.

Use case cosideratios

Our models are ot specifically desiged or evaluated for all dowstream purposes. Developers should cosider commo limitatios of laguage models as they select use cases, ad evaluate ad mitigate for accuracy, safety, ad fairess before usig withi a specific dowstream use case, particularly for high-risk scearios. Developers should be aware of ad adhere to applicable laws or regulatios (icludig privacy, trade compliace laws, etc.) that are relevat to their use case.

Nothig cotaied i this Model Card should be iterpreted as or deemed a restrictio or modificatio to the licese the model is released uder.

How to Use

Phi-3-visio-128K-Istruct has bee itegrated i the developmet versio (4.40.2) of trasformers. Util the official versio is released through pip, esure that you are doig oe of the followig:

  • Whe loadig the model, esure that trust_remote_code=True is passed as a argumet of the from_pretraied() fuctio.

  • Update your local trasformers to the developmet versio: pip uistall -y trasformers && pip istall git+https://github.com/huggigface/trasformers. The previous commad is a alterative to cloig ad istallig from the source.

The curret trasformers versio ca be verified with: pip list | grep trasformers.

Examples of required packages:

flash_att==2.5.8
umpy==1.24.4
Pillow==10.3.0
Requests==2.31.0
torch==2.3.0
torchvisio==0.18.0
trasformers==4.40.2

Phi-3-Visio-128K-Istruct is also available i Azure AI Studio.

Chat Format

Give the ature of the traiig data, the Phi-3-Visio-128K-Istruct model is best suited for a sigle image iput wih prompts usig the chat format as follows. You ca provide the prompt as a sigle image with a geeric template as follow:

<|user|>\<|image_1|>\{prompt}<|ed|>\<|assistat|>\ 

where the model geerates the text after <|assistat|> . I case of multi-tur coversatio, the prompt ca be formatted as follows:

<|user|>\<|image_1|>\{prompt_1}<|ed|>\<|assistat|>\{respose_1}<|ed|>\<|user|>\{prompt_2}<|ed|>\<|assistat|>\ 

Sample iferece code

This code sippets show how to get quickly started with ruig the model o a GPU:

from PIL import Image 
import requests 
from modelscope import sapshot_dowload
from trasformers import AutoModelForCausalLM 
from trasformers import AutoProcessor 

model_id = sapshot_dowload("LLM-Research/Phi-3-visio-128k-istruct" )

model = AutoModelForCausalLM.from_pretraied(model_id, device_map="cuda", trust_remote_code=True, torch_dtype="auto")

processor = AutoProcessor.from_pretraied(model_id, trust_remote_code=True) 

messages = [ 
    {"role": "user", "cotet": "<|image_1|>\What is show i this image?"}, 
    {"role": "assistat", "cotet": "The chart displays the percetage of respodets who agree with various statemets about their preparedess for meetigs. It shows five categories: 'Havig clear ad pre-defied goals for meetigs', 'Kowig where to fid the iformatio I eed for a meetig', 'Uderstadig my exact role ad resposibilities whe I'm ivited', 'Havig tools to maage admi tasks like ote-takig or summarizatio', ad 'Havig more focus time to sufficietly prepare for meetigs'. Each category has a associated bar idicatig the level of agreemet, measured o a scale from 0% to 100%."}, 
    {"role": "user", "cotet": "Provide isightful questios to spark discussio."} 
] 

url = "https://assets-c4akfrf5b4d3f4b7.z01.azurefd.et/assets/2024/04/BMDataViz_661fb89f3845e.pg" 
image = Image.ope(requests.get(url, stream=True).raw) 

prompt = processor.tokeizer.apply_chat_template(messages, tokeize=False, add_geeratio_prompt=True)

iputs = processor(prompt, [image], retur_tesors="pt").to("cuda:0") 

geeratio_args = { 
    "max_ew_tokes": 500, 
    "temperature": 0.0, 
    "do_sample": False, 
} 

geerate_ids = model.geerate(**iputs, eos_toke_id=processor.tokeizer.eos_toke_id, **geeratio_args) 

# remove iput tokes 
geerate_ids = geerate_ids[:, iputs['iput_ids'].shape[1]:]
respose = processor.batch_decode(geerate_ids, skip_special_tokes=True, clea_up_tokeizatio_spaces=False)[0] 

prit(respose) 

Additioal basic examples are provided here.

Resposible AI Cosideratios

Like other models, the Phi family of models ca potetially behave i ways that are ufair, ureliable, or offesive. Some of the limitig behaviors to be aware of iclude:

  • Quality of Service: The Phi models are traied primarily o Eglish text. Laguages other tha Eglish will experiece worse performace. Eglish laguage varieties with less represetatio i the traiig data might experiece worse performace tha stadard America Eglish.
  • Represetatio of Harms & Perpetuatio of Stereotypes: These models ca over- or uder-represet groups of people, erase represetatio of some groups, or reiforce demeaig or egative stereotypes. Despite safety post-traiig, these limitatios may still be preset due to differig levels of represetatio of differet groups or prevalece of examples of egative stereotypes i traiig data that reflect real-world patters ad societal biases.
  • Iappropriate or Offesive Cotet: These models may produce other types of iappropriate or offesive cotet, which may make it iappropriate to deploy for sesitive cotexts without additioal mitigatios that are specific to the use case.
  • Iformatio Reliability: Laguage models ca geerate osesical cotet or fabricate cotet that might soud reasoable but is iaccurate or outdated.
  • Limited Scope for Code: Majority of Phi-3 traiig data is based i Pytho ad use commo packages such as "typig, math, radom, collectios, datetime, itertools". If the model geerates Pytho scripts that utilize other packages or scripts i other laguages, we strogly recommed users maually verify all API uses.

Developers should apply resposible AI best practices ad are resposible for esurig that a specific use case complies with relevat laws ad regulatios (e.g. privacy, trade, etc.). Importat areas for cosideratio iclude:

  • Allocatio: Models may ot be suitable for scearios that could have cosequetial impact o legal status or the allocatio of resources or life opportuities (ex: housig, employmet, credit, etc.) without further assessmets ad additioal debiasig techiques.
  • High-Risk Scearios: Developers should assess suitability of usig models i high-risk scearios where ufair, ureliable or offesive outputs might be extremely costly or lead to harm. This icludes providig advice i sesitive or expert domais where accuracy ad reliability are critical (ex: legal or health advice). Additioal safeguards should be implemeted at the applicatio level accordig to the deploymet cotext.
  • Misiformatio: Models may produce iaccurate iformatio. Developers should follow trasparecy best practices ad iform ed-users they are iteractig with a AI system. At the applicatio level, developers ca build feedback mechaisms ad pipelies to groud resposes i use-case specific, cotextual iformatio, a techique kow as Retrieval Augmeted Geeratio (RAG).
  • Geeratio of Harmful Cotet: Developers should assess outputs for their cotext ad use available safety classifiers or custom solutios appropriate for their use case.
  • Misuse: Other forms of misuse such as fraud, spam, or malware productio may be possible, ad developers should esure that their applicatios do ot violate applicable laws ad regulatios.
  • Idetificatio of idividuals: models with visio capabilities may have the potetial to uiquely idetify idividuals i images. Safety post-traiig steers the model to refuse such requests, but developers should cosider ad implemet, as appropriate, additioal mitigatios or user coset flows as required i their respective jurisdictio, (e.g., buildig measures to blur faces i image iputs before processig.

Traiig

Model

  • Architecture: Phi-3-Visio-128K-Istruct has 4.2B parameters ad cotais image ecoder, coector, projector, ad Phi-3 Mii laguage model.
  • Iputs: Text ad Image. It’s best suited for prompts usig the chat format.
  • Cotext legth: 128K tokes
  • GPUs: 512 H100-80G
  • Traiig time: 1.5 days
  • Traiig data: 500B visio ad text tokes
  • Outputs: Geerated text i respose to the iput
  • Dates: Our models were traied betwee February ad April 2024
  • Status: This is a static model traied o a offlie text dataset with cutoff date Mar 15, 2024. Future versios of the tued models may be released as we improve models.
  • Release Type: Ope weight release
  • Release dates: The model weight is released o May 21, 2024.

Datasets

Our traiig data icludes a wide variety of sources, ad is a combiatio of

1) publicly available documets filtered rigorously for quality, selected high-quality educatioal data ad code; 2) selected high-quality image-text iterleave; 3) ewly created sythetic, “textbook-like” data for the purpose of teachig math, codig, commo sese reasoig, geeral kowledge of the world (sciece, daily activities, theory of mid, etc.), ewly created image data, e.g., chart/table/diagram/slides; 4) high quality chat format supervised data coverig various topics to reflect huma prefereces o differet aspects such as istruct-followig, truthfuless, hoesty ad helpfuless.

The data collectio process ivolved sourcig iformatio from publicly available documets, with a meticulous approach to filterig out udesirable documets ad images. To safeguard privacy, we carefully filtered various image ad text data sources to remove or scrub ay potetially persoal data from the traiig data.

More details ca be foud i the Phi-3 Techical Report.

Bechmarks

To uderstad the capabilities, we compare Phi-3 Visio-128K-Istruct with a set of models over a variety of zero-shot bechmarks usig our iteral bechmark platform.

Bechmark Phi-3 Visio-128K-I LlaVA-1.6 Vicua-7B QWEN-VL Chat Llama3-Llava-Next-8B Claude-3 Haiku Gemii 1.0 Pro V GPT-4V-Turbo
MMMU 40.4 34.2 39.0 36.4 40.7 42.0 55.5
MMBech 80.5 76.3 75.8 79.4 62.4 80.0 86.1
ScieceQA 90.8 70.6 67.2 73.7 72.0 79.7 75.7
MathVista 44.5 31.5 29.4 34.8 33.2 35.0 47.5
IterGPS 38.1 20.5 22.3 24.6 32.1 28.6 41.0
AI2D 76.7 63.1 59.8 66.9 60.3 62.8 74.7
ChartQA 81.4 55.0 50.9 65.8 59.3 58.0 62.3
TextVQA 70.9 64.6 59.4 55.7 62.7 64.7 68.1
POPE 85.8 87.2 82.6 87.0 74.4 84.2 83.7

Software

Hardware

Note that by default, the Phi-3-Visio-128K model uses flash attetio, which requires certai types of GPU hardware to ru. We have tested o the followig GPU types:

  • NVIDIA A100
  • NVIDIA A6000
  • NVIDIA H100

Ruig o Widows or without flash attetio

To eable the model o these eviromet here are steps that you may cosider to follow:

Step 1: commet flash attetio import code i modeligphi3v.py from lie 52 to lie 56.

# if is_flash_att_2_available():
#     from flash_att import flash_att_fuc, flash_att_varle_fuc
#     from flash_att.bert_paddig import idex_first_axis, pad_iput, upad_iput  # oqa

#     _flash_supports_widow_size = "widow_size" i list(ispect.sigature(flash_att_fuc).parameters)

Step 2: chage "attimplemetatio" from "flashattetio2" to "eager" i cofig.jso or disable flash attetio whe you create the model as below.

model = AutoModelForCausalLM.from_pretraied('microsoft/Phi-3-visio-128k-istruct', device_map="cuda", trust_remote_code=True, torch_dtype="auto", _att_implemetatio="eager")

Licese

The model is licesed uder the MIT licese.

Trademarks

This project may cotai trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to ad must follow Microsoft’s Trademark & Brad Guidelies. Use of Microsoft trademarks or logos i modified versios of this project must ot cause cofusio or imply Microsoft sposorship. Ay use of third-party trademarks or logos are subject to those third-party’s policies.

功能介绍

Model Summary Phi-3 Vision is a lightweight, state-of-the-art open multimodal model built upon datas

声明:本文仅代表作者观点,不代表本站立场。如果侵犯到您的合法权益,请联系我们删除侵权资源!如果遇到资源链接失效,请您通过评论或工单的方式通知管理员。未经允许,不得转载,本站所有资源文章禁止商业使用运营!
下载安装【程序员客栈】APP
实时对接需求、及时收发消息、丰富的开放项目需求、随时随地查看项目状态

评论