Model Summary
The Phi-3-Small-128K-Istruct is a 7B parameters, lightweight, state-of-the-art ope model traied with the Phi-3 datasets that icludes both sythetic data ad the filtered publicly available websites data with a focus o high-quality ad reasoig dese properties.
The model belogs to the Phi-3 family with the Small versio i two variats 8K ad 128K which is the cotext legth (i tokes) that it ca support.
The model has uderwet a post-traiig process that icorporates both supervised fie-tuig ad direct preferece optimizatio for the istructio followig ad safety measures.
Whe assessed agaist bechmarks testig commo sese, laguage uderstadig, math, code, log cotext ad logical reasoig, Phi-3-Small-128K-Istruct showcased a robust ad state-of-the-art performace amog models of the same-size ad ext-size-up.
Resources ad Techical Documetatio:
Iteded Uses
Primary use cases
The model is iteded for broad commercial ad research use i Eglish. The model provides uses for geeral purpose AI systems ad applicatios which require :
1) Memory/compute costraied eviromets
2) Latecy boud scearios
3) Strog reasoig (especially code, math ad logic)
Our model is desiged to accelerate research o laguage ad multimodal models, for use as a buildig block for geerative AI powered features.
Use case cosideratios
Our models are ot specifically desiged or evaluated for all dowstream purposes. Developers should cosider commo limitatios of laguage models as they select use cases, ad evaluate ad mitigate for accuracy, safety, ad fariess before usig withi a specific dowstream use case, particularly for high risk scearios. Developers should be aware of ad adhere to applicable laws or regulatios (icludig privacy, trade compliace laws, etc.) that are relevat to their use case.
Nothig cotaied i this Model Card should be iterpreted as or deemed a restrictio or modificatio to the licese the model is released uder.
How to Use
Phi-3-Small-128K-Istruct has bee itegrated i the developmet versio (4.40.2) of trasformers. Util the official versio is released through pip, esure that you are doig oe of the followig:
Istall tiktoke (0.6.0) as trito (2.3.0)
Whe loadig the model, esure that trust_remote_code=True is passed as a argumet of the from_pretraied() fuctio.
Update your local trasformers to the developmet versio: pip uistall -y trasformers && pip istall git+https://github.com/huggigface/trasformers. The previous commad is a alterative to cloig ad istallig from the source.
The curret trasformers versio ca be verified with: pip list | grep trasformers.
Phi-3-Small-128K-Istruct is also available i Azure AI.
Tokeizer
Phi-3-Small-128K-Istruct supports a vocabulary size of up to 100352 tokes.
Chat Format
Give the ature of the traiig data, the Phi-3-Small-128K-Istruct model is best suited for prompts usig the chat format as follows.
You ca provide the prompt as a questio with a geeric template as follow:
<|edoftext|><|user|>\Questio <|ed|>\<|assistat|>
For example:
<|edoftext|><|user|>
How to explai Iteret for a medieval kight?<|ed|>
<|assistat|>
where the model geerates the text after <|assistat|> . I case of few-shots prompt, the prompt ca be formatted as the followig:
<|edoftext|><|user|>
I am goig to Paris, what should I see?<|ed|>
<|assistat|>
Paris, the capital of Frace, is kow for its stuig architecture, art museums, historical ladmarks, ad romatic atmosphere. Here are some of the top attractios to see i Paris:\\1. The Eiffel Tower: The icoic Eiffel Tower is oe of the most recogizable ladmarks i the world ad offers breathtakig views of the city.\2. The Louvre Museum: The Louvre is oe of the world's largest ad most famous museums, housig a impressive collectio of art ad artifacts, icludig the Moa Lisa.\3. Notre-Dame Cathedral: This beautiful cathedral is oe of the most famous ladmarks i Paris ad is kow for its Gothic architecture ad stuig staied glass widows.\\These are just a few of the may attractios that Paris has to offer. With so much to see ad do, it's o woder that Paris is oe of the most popular tourist destiatios i the world."<|ed|>
<|user|>
What is so great about #1?<|ed|>
<|assistat|>
Sample iferece code
This code sippets show how to get quickly started with ruig the model o a GPU:
import torch
from trasformers import AutoModelForCausalLM, AutoTokeizer, pipelie
torch.radom.maual_seed(0)
model_id = "microsoft/Phi-3-small-128k-istruct"
model = AutoModelForCausalLM.from_pretraied(
model_id,
torch_dtype="auto",
trust_remote_code=True,
)
assert torch.cuda.is_available(), "This model eeds a GPU to ru ..."
device = torch.cuda.curret_device()
model = model.to(device)
tokeizer = AutoTokeizer.from_pretraied(model_id)
messages = [
{"role": "user", "cotet": "Ca you provide ways to eat combiatios of baaas ad dragofruits?"},
{"role": "assistat", "cotet": "Sure! Here are some ways to eat baaas ad dragofruits together: 1. Baaa ad dragofruit smoothie: Bled baaas ad dragofruits together with some milk ad hoey. 2. Baaa ad dragofruit salad: Mix sliced baaas ad dragofruits together with some lemo juice ad hoey."},
{"role": "user", "cotet": "What about solvig a 2x + 3 = 7 equatio?"},
]
pipe = pipelie(
"text-geeratio",
model=model,
tokeizer=tokeizer,
device=device
)
geeratio_args = {
"max_ew_tokes": 500,
"retur_full_text": False,
"temperature": 0.0,
"do_sample": False,
}
output = pipe(messages, **geeratio_args)
prit(output[0]['geerated_text'])
Some applicatios/frameworks might ot iclude a BOS toke (<|edoftext|>) at the start of the coversatio. Please esure that it is icluded sice it provides more reliable results.
Resposible AI Cosideratios
Like other laguage models, the Phi series models ca potetially behave i ways that are ufair, ureliable, or offesive. Some of the limitig behaviors to be aware of iclude:
- Quality of Service: the Phi models are traied primarily o Eglish text. Laguages other tha Eglish will experiece worse performace. Eglish laguage varieties with less represetatio i the traiig data might experiece worse performace tha stadard America Eglish.
- Represetatio of Harms & Perpetuatio of Stereotypes: These models ca over- or uder-represet groups of people, erase represetatio of some groups, or reiforce demeaig or egative stereotypes. Despite safety post-traiig, these limitatios may still be preset due to differig levels of represetatio of differet groups or prevalece of examples of egative stereotypes i traiig data that reflect real-world patters ad societal biases.
- Iappropriate or Offesive Cotet: these models may produce other types of iappropriate or offesive cotet, which may make it iappropriate to deploy for sesitive cotexts without additioal mitigatios that are specific to the use case.
- Iformatio Reliability: Laguage models ca geerate osesical cotet or fabricate cotet that might soud reasoable but is iaccurate or outdated.
- Limited Scope for Code: Majority of Phi-3 traiig data is based i Pytho ad use commo packages such as "typig, math, radom, collectios, datetime, itertools". If the model geerates Pytho scripts that utilize other packages or scripts i other laguages, we strogly recommed users maually verify all API uses.
Developers should apply resposible AI best practices ad are resposible for esurig that a specific use case complies with relevat laws ad regulatios (e.g. privacy, trade, etc.). Importat areas for cosideratio iclude:
- Allocatio: Models may ot be suitable for scearios that could have cosequetial impact o legal status or the allocatio of resources or life opportuities (ex: housig, employmet, credit, etc.) without further assessmets ad additioal debiasig techiques.
- High-Risk Scearios: Developers should assess suitability of usig models i high-risk scearios where ufair, ureliable or offesive outputs might be extremely costly or lead to harm. This icludes providig advice i sesitive or expert domais where accuracy ad reliability are critical (ex: legal or health advice). Additioal safeguards should be implemeted at the applicatio level accordig to the deploymet cotext.
- Misiformatio: Models may produce iaccurate iformatio. Developers should follow trasparecy best practices ad iform ed-users they are iteractig with a AI system. At the applicatio level, developers ca build feedback mechaisms ad pipelies to groud resposes i use-case specific, cotextual iformatio, a techique kow as Retrieval Augmeted Geeratio (RAG).
- Geeratio of Harmful Cotet: Developers should assess outputs for their cotext ad use available safety classifiers or custom solutios appropriate for their use case.
- Misuse: Other forms of misuse such as fraud, spam, or malware productio may be possible, ad developers should esure that their applicatios do ot violate applicable laws ad regulatios.
Traiig
Model
- Architecture: Phi-3 Small-128K-Istruct has 7B parameters ad is a dese decoder-oly Trasformer model with alteratig dese ad blocksparse attetios. The model is fie-tued with Supervised fie-tuig (SFT) ad Direct Preferece Optimizatio (DPO) to esure aligmet with huma prefereces ad safety guidlies.
- Iputs: Text. It is best suited for prompts usig chat format.
- Cotext legth: 128K tokes
- GPUs: 1024 H100-80G
- Traiig time: 18 days
- Traiig data: 4.8T tokes
- Outputs: Geerated text i respose to the iput
- Dates: Our models were traied betwee February ad April 2024
- Status: This is a static model traied o a offlie dataset with cutoff date October 2023. Future versios of the tued models may be released as we improve models.
- Release dates The model weight is released o May 21, 2024.
Datasets
Our traiig data icludes a wide variety of sources, totalig 4.8 trillio tokes (icludig 10% multiligual), ad is a combiatio of
1) Publicly available documets filtered rigorously for quality, selected high-quality educatioal data, ad code;
2) Newly created sythetic, “textbook-like” data for the purpose of teachig math, codig, commo sese reasoig, geeral kowledge of the world (sciece, daily activities, theory of mid, etc.);
3) High quality chat format supervised data coverig various topics to reflect huma prefereces o differet aspects such as istruct-followig, truthfuless, hoesty ad helpfuless.
We are focusig o the quality of data that could potetially improve the reasoig ability for the model, ad we filter the publicly available documets to cotai the correct level of kowledge. As a example, the result of a game i premier league i a particular day might be good traiig data for frotier models, but we eed to remove such iformatio to leave more model capacity for reasoig for the small size models. More details about data ca be foud i the Phi-3 Techical Report.
Bechmarks
We report the results for Phi-3-Small-128K-Istruct o stadard ope-source bechmarks measurig the model's reasoig ability (both commo sese reasoig ad logical reasoig). We compare to Mixtral-8x7b, Gemii-Pro, Gemma 7B, Llama-3-8B-Istruct, GPT-3.5-Turbo-1106, ad GPT-4-Turbo-1106.
All the reported umbers are produced with the exact same pipelie to esure that the umbers are comparable. These umbers might differ from other published umbers due to slightly differet choices i the evaluatio.
As is ow stadard, we use few-shot prompts to evaluate the models, at temperature 0.
The prompts ad umber of shots are part of a Microsoft iteral tool to evaluate laguage models, ad i particular we did o optimizatio to the pipelie for Phi-3.
More specifically, we do ot chage prompts, pick differet few-shot examples, chage prompt format, or do ay other form of optimizatio for the model.
The umber of k–shot examples is listed per-bechmark.
| Bechmark |
Phi-3-Small-128K-Istruct 7b |
Gemma 7B |
Mixtral 8x7B |
Llama-3-Istruct 8b |
GPT-3.5-Turbo versio 1106 |
Gemii Pro |
GPT-4-Turbo versio 1106 (Chat) |
AGI Eval 5-shot |
43.9 |
42.1 |
45.2 |
42.0 |
48.4 |
49.0 |
59.6 |
MMLU 5-shot |
75.5 |
63.6 |
70.5 |
66.5 |
71.4 |
66.7 |
84.0 |
BigBech Hard 3-shot |
77.6 |
59.6 |
69.7 |
51.5 |
68.3 |
75.6 |
87.7 |
ANLI 7-shot |
55.8 |
48.7 |
55.2 |
57.3 |
58.1 |
64.2 |
71.7 |
HellaSwag 5-shot |
79.6 |
49.8 |
70.4 |
71.1 |
78.8 |
76.2 |
88.3 |
ARC Challege 10-shot |
90.8 |
78.3 |
87.3 |
82.8 |
87.4 |
88.3 |
95.6 |
ARC Easy 10-shot |
97.3 |
91.4 |
95.6 |
93.4 |
96.3 |
96.1 |
98.8 |
BoolQ 2-shot |
83.7 |
66.0 |
76.6 |
80.9 |
79.1 |
86.4 |
91.3 |
CommoseseQA 10-shot |
80.8 |
76.2 |
78.1 |
79.0 |
79.6 |
81.8 |
86.7 |
MedQA 2-shot |
46.3 |
49.6 |
62.2 |
60.5 |
63.4 |
58.2 |
83.7 |
OpeBookQA 10-shot |
87.8 |
78.6 |
85.8 |
82.6 |
86.0 |
86.4 |
93.4 |
PIQA 5-shot |
88.1 |
78.1 |
86.0 |
75.7 |
86.6 |
86.2 |
90.1 |
Social IQA 5-shot |
78.7 |
65.5 |
75.9 |
73.9 |
68.3 |
75.4 |
81.7 |
TruthfulQA (MC2) 10-shot |
69.6 |
52.1 |
60.1 |
63.2 |
67.7 |
72.6 |
85.2 |
WioGrade 5-shot |
80.1 |
55.6 |
62.0 |
65.0 |
68.8 |
72.2 |
86.7 |
TriviaQA 5-shot |
66.0 |
72.3 |
82.2 |
67.7 |
85.8 |
80.2 |
73.3 |
GSM8K Chai of Thought 8-shot |
87.3 |
59.8 |
64.7 |
77.4 |
78.1 |
80.4 |
94.2 |
HumaEval 0-shot |
59.1 |
34.1 |
37.8 |
60.4 |
62.2 |
64.4 |
79.9 |
MBPP 3-shot |
70.3 |
51.5 |
60.2 |
67.7 |
77.8 |
73.2 |
86.7 |
| Average |
74.6 |
61.8 |
69.8 |
69.4 |
74.3 |
75.4 |
85.2 |
We take a closer look at differet categories across 80 public bechmark datasets at the table below:
| Bechmark |
Phi-3-Small-128K-Istruct 7b |
Gemma 7B |
Mixtral 8x7B |
Llama-3-Istruct 8b |
GPT-3.5-Turbo versio 1106 |
Gemii Pro |
GPT-4-Turbo versio 1106 (Chat) |
| Popular aggregated bechmark |
70.6 |
59.4 |
66.2 |
59.9 |
67.0 |
67.5 |
80.5 |
| Reasoig |
80.3 |
69.1 |
77.0 |
75.7 |
78.3 |
80.4 |
89.3 |
| Laguage uderstadig |
67.4 |
58.4 |
64.9 |
65.4 |
70.4 |
75.3 |
81.6 |
| Code geeratio |
60.0 |
45.6 |
52.7 |
56.4 |
70.4 |
66.7 |
76.1 |
| Math |
48.1 |
35.8 |
40.3 |
41.1 |
52.8 |
50.9 |
67.1 |
| Factual kowledge |
41.7 |
46.7 |
58.6 |
43.1 |
63.4 |
54.6 |
45.9 |
| Multiligual |
62.6 |
63.2 |
63.4 |
65.0 |
69.1 |
76.5 |
82.0 |
| Robustess |
68.7 |
38.4 |
51.0 |
64.5 |
69.3 |
69.7 |
84.6 |
Software
Hardware
Note that by default, the Phi-3-Small model uses flash attetio 2 ad Trito blocksparse attetio, which requires certai types of GPU hardware to ru. We have tested o the followig GPU types:
- NVIDIA A100
- NVIDIA A6000
- NVIDIA H100
If you wat to ru the model o:
- Optimized iferece o GPU, CPU, ad Mobile: use the ONNX models 128K
Cross Platform Support
ONNX rutime ecosystem ow supports Phi3 small models across platforms ad hardware.
Optimized phi-3 models are also published here i ONNX format, to ru with ONNX Rutime o CPU ad GPU across devices, icludig server platforms, Widows, Liux ad Mac desktops, ad mobile CPUs, with the precisio best suited to each of these targets. DirectML GPU acceleratio is supported for Widows desktops GPUs (AMD, Itel, ad NVIDIA).
Alog with DML, ONNX Rutime provides cross platform support for Phi3 Small across a rage of devices CPU, GPU, ad mobile.
Here are some of the optimized cofiguratios we have added:
- ONNX models for it4 DML: Quatized to it4 via AWQ
- ONNX model for fp16 CUDA
- ONNX model for it4 CUDA: Quatized to it4 via RTN
- ONNX model for it4 CPU ad Mobile: Quatized to it4 via RTN
Licese
The model is licesed uder the MIT licese.
Trademarks
This project may cotai trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to ad must follow Microsoft’s Trademark & Brad Guidelies. Use of Microsoft trademarks or logos i modified versios of this project must ot cause cofusio or imply Microsoft sposorship. Ay use of third-party trademarks or logos are subject to those third-party’s policies.
评论