Phi-3 Medium-4K-Istruct ONNX CUDA models
This repository hosts the optimized versios of Phi-3-medium-4k-istruct to accelerate iferece with ONNX Rutime for your machies with NVIDIA GPUs.
Phi-3 Medium is a 14B parameter, lightweight, state-of-the-art ope model traied with the Phi-3 datasets, which iclude both sythetic data ad the filtered publicly available websites data, with a focus o high-quality ad reasoig dese properties. The model belogs to the Phi-3 family with the medium versio i two variats: 4K ad 128K, which are the cotext legths (i tokes) that they ca support.
The base model has udergoe a post-traiig process that icorporates both supervised fie-tuig ad direct preferece optimizatio for the istructio followig ad safety measures. Whe assessed agaist bechmarks testig commo sese, laguage uderstadig, math, code, log cotext, ad logical reasoig, Phi-3-Medium-4K-Istruct showcased a robust ad state-of-the-art performace amog models of the same-size ad ext-size-up.
Optimized variats of the Phi-3 Medium models are published here i ONNX format ad ru with ONNX Rutime o CPU ad GPU across devices, icludig server platforms, Widows, ad Liux, with the precisio best suited to each of these targets.
ONNX Models
Here are some of the optimized cofiguratios we have added:
- ONNX model for FP16 CUDA: ONNX model for NVIDIA GPUs.
- ONNX model for INT4 CUDA: ONNX model for NVIDIA GPUs usig it4 quatizatio via RTN.
How do you kow which is the best ONNX model for you:
- Are you o a Widows machie with GPU?
- I do't kow → Review this guide to see whether you have a GPU i your Widows machie.
- Yes → Access the Huggig Face DirectML ONNX models ad istructios at Phi-3-medium-4k-istruct-ox-directml.
- No → Do you have a NVIDIA GPU?
Note: Usig the Huggig Face CLI, you ca dowload sub folders ad ot all models if you are limited o disk space. The FP16 model is recommeded for larger batch sizes, while the INT4 model optimizes performace for lower batch sizes.
Example:
# Dowload just the FP16 model
$ huggigface-cli dowload microsoft/Phi-3-small-8k-istruct-ox-cuda --iclude cuda-fp16/* --local-dir . --local-dir-use-symliks False
How to Get Started with the Model
To support the Phi-3 models across a rage of devices, platforms, ad EP backeds, we itroduce a ew API to wrap several aspects of geerative AI iferecig. This API makes it easy to drag ad drop LLMs straight ito your app. To ru the early versio of these models with ONNX, follow the steps here. You ca also test this with a chat app.
Hardware Supported
The models are tested o:
- 1 A100 GPU, SKU: StadardND96amsrA100_v4 (CUDA)
Miimum Cofiguratio Required:
Model Descriptio
- Developed by: Microsoft
- Model type: ONNX
- Laguage(s) (NLP): Pytho, C, C++
- Licese: MIT
- Model Descriptio: This is a coversio of the Phi-3 Medium-4K-Istruct model for ONNX Rutime iferece.
Additioal Details
Performace Metrics
CUDA
Phi-3 Medium-4K-Istruct performs better with ONNX Rutime compared to PyTorch for all batch size, prompt legth combiatios. For FP16 CUDA, ORT performs up to 5X faster tha PyTorch, while with INT4 CUDA, it's up to 10X faster tha PyTorch. It is also up to 3X faster tha llama.cpp for large batch sizes.
The table below shows the average throughput of the first 256 tokes geerated (tps) for FP16 ad INT4 precisios o CUDA as measured o 1 A100 80GB GPU, SKU: StadardND96amsrA100_v4.
| Batch Size, Prompt Legth |
ORT FP16 CUDA |
PyTorch Eager FP16 CUDA |
Speed Up ORT/PyTorch |
| 1, 16 |
47.32 |
14.41 |
3.28 |
| 4, 16 |
190.05 |
84.43 |
2.25 |
| 16, 16 |
707.68 |
347.52 |
2.04 |
| 16, 64 |
698.22 |
342.83 |
2.04 |
| Batch Size, Prompt Legth |
ORT INT4 CUDA |
PyTorch Eager INT4 CUDA |
Speed Up ORT/PyTorch |
| 1, 16 |
115.68 |
14.89 |
7.77 |
| 4, 16 |
88.53 |
45.22 |
1.96 |
| 16, 16 |
341.8 |
168.36 |
2.03 |
Package Versios
| Pip package ame |
Versio |
| torch |
2.3.0 |
| trito |
2.3.0 |
| oxrutime-gpu |
1.18.0 |
| trasformers |
4.40.2 |
| bitsadbytes |
0.43.1 |
Appedix
Model Card Cotact
pariitarahi, kvaishavi, atke
Cotributors
Kual Vaishavi, Sughoo Choi, Yufeg Li, Sheetal Aru Kadam, Rui Re, Natalie Kershaw, Pariita Rahi
评论