This repository hosts the optimized versios of Phi-3-medium-4k-istruct to accelerate iferece with DirectML ad ONNX Rutime for your machies with GPUs. Phi-3 Medium is a 14B parameter, lightweight, state-of-the-art ope model traied with the Phi-3 datasets, which iclude both sythetic data ad the filtered publicly available websites data, with a focus o high-quality ad reasoig dese properties. The model belogs to the Phi-3 family with the medium versio i two variats: 4K ad 128K, which are the cotext legths (i tokes) that they ca support. The model has uderwet a post-traiig process that icorporates both supervised fie-tuig ad direct preferece optimizatio for the istructio followig ad safety measures. Whe assessed agaist bechmarks testig commo sese, laguage uderstadig, math, code, log cotext, ad logical reasoig, Phi-3-Medium-4K-Istruct showcased a robust ad state-of-the-art performace amog models of the same-size ad ext-size-up. Optimized variats of the Phi-3 Medium models are published here i ONNX format ad ru with DirectML. This lets developers brig hardware acceleratio to Widows devices at scale across AMD, Itel, ad NVIDIA GPUs. Here are some of the optimized cofiguratios we have added: How do you kow which is the best ONNX model for you: To support the Phi-3 models across a rage of devices, platforms, ad EP backeds, we itroduce a ew API to wrap several aspects of geerative AI iferecig. This API makes it easy to drag ad drop LLMs straight ito your app. To ru the early versio of these models with ONNX, follow the steps here. You ca also test this with a chat app. The model has bee tested o: Miimum Cofiguratio Required: We measured the performace of DirectML ad ONNX Rutime's ew Geerate() API with Phi-3 medium quatized with Activatio-Aware Quatizatio AWQ ad with a block size of 128 o Widows. Our test machie had a NVIDIA GeForce RTX 4090 GPU ad a Itel Core i9-13900K CPU. DirectML lets developers ot oly achieve great performace but also lets developers deploy models across the etire Widows ecosystem with support from AMD, Itel, ad NVIDIA. Best of all, AWQ meas that developers get this scale while also maitaiig high model accuracy. Stay tued for additioal performace improvemets i the comig weeks thaks to optimized drivers from our hardware parters, alog with additioal updates to the ONNX Rutime Geerate() API. AWQ works by idetifyig the top 1% most saliet weights that are most importat for maitaiig accuracy ad quatizig the remaiig 99% of weights. This leads to less accuracy loss from quatizatio compared to may other quatizatio techiques. For more o AWQ see here. pariitarahi, kvaishavi, atke Kual Vaishavi, Sughoo Choi, Yufeg Li, Sheetal Aru Kadam, Natalie Kershaw, Pariita Rahi, Patrice Vigola, Xiag Zhag, Chai Chaoweeraprasit, Loga Iyer, Vicete Rivera, Jacques Va RhyPhi-3 Medium-4K-Istruct ONNX DirectML models
ONNX Models
How to Get Started with the Model
Hardware Supported
Model Descriptio
Additioal Details
Performace Metrics
DirectML
Batch Size, Prompt Legth
Block Size = 32
Block Size = 128
1, 16
66.36
72.39
Package Versios
Pip package ame
Versio
torch
2.2.0
trito
2.2.0
oxrutime-gpu
1.18.0
trasformers
4.39.0
bitsadbytes
0.42.0
Appedix
Activatio Aware Quatizatio
Model Card Cotact
Cotributors
点击空白处退出提示







评论