Phi-3-medium-4k-instruct-onnx-cpu

我要开发同款
匿名用户2024年07月31日
110阅读

技术信息

开源地址
https://modelscope.cn/models/LLM-Research/Phi-3-medium-4k-instruct-onnx-cpu
授权协议
mit

作品详情

Phi-3 Medium-4k-Istruct ONNX CPU models

This repository hosts the optimized versios of Phi-3-medium-4k-istruct to accelerate iferece with ONNX Rutime for your CPU.

Phi-3 Medium is a 14B parameter, lightweight, state-of-the-art ope model traied with the Phi-3 datasets, which iclude both sythetic data ad the filtered publicly available websites data, with a focus o high-quality ad reasoig dese properties. The model belogs to the Phi-3 family with the medium versio i two variats: 4K ad 128K, which are the cotext legths (i tokes) that they ca support.

The base model has udergoe a post-traiig process that icorporates both supervised fie-tuig ad direct preferece optimizatio for the istructio followig ad safety measures. Whe assessed agaist bechmarks testig commo sese, laguage uderstadig, math, code, log cotext, ad logical reasoig, Phi-3-Medium-4K-Istruct showcased a robust ad state-of-the-art performace amog models of the same-size ad ext-size-up.

Optimized variats of the Phi-3 Medium models are published here i ONNX format ad ru with ONNX Rutime o CPU ad GPU across devices, icludig server platforms, Widows, ad Liux, with the precisio best suited to each of these targets.

ONNX Models

Here are some of the optimized cofiguratios we have added:

  1. ONNX model for INT4 CPU: ONNX model for CPUs usig it4 quatizatio via RTN.

How do you kow which is the best ONNX model for you:

  • Are you o a Widows machie with GPU?

How to Get Started with the Model

To support the Phi-3 models across a rage of devices, platforms, ad EP backeds, we itroduce a ew API to wrap several aspects of geerative AI iferecig. This API makes it easy to drag ad drop LLMs straight ito your app. To ru the early versio of these models with ONNX, follow the steps here. You ca also test this with a chat app.

Hardware Supported

The models are tested o:

  • Itel(R) Core(TM) i9-10920X CPU @ 3.50GHz

Miimum Cofiguratio Required:

  • CPU machie with 16GB RAM

Model Descriptio

  • Developed by: Microsoft
  • Model type: ONNX
  • Laguage(s) (NLP): Pytho, C, C++
  • Licese: MIT
  • Model Descriptio: This is a coversio of the Phi-3 Medium-4k-Istruct model for ONNX Rutime iferece.

Additioal Details

Performace Metrics

The model rus at ~20 tokes/sec o a Itel(R) Core(TM) i9-10920X CPU @ 3.50GHz.

Appedix

Model Card Cotact

pariitarahi, kvaishavi, atke

Cotributors

Kual Vaishavi, Sughoo Choi, Yufeg Li, Akshay Soawae, Sheetal Aru Kadam, Rui Re, Edward Che, Scott McKay, Emma Nig, Natalie Kershaw, Pariita Rahi

功能介绍

Phi-3 Medium-4k-Instruct ONNX CPU models This repository hosts the optimized versions of Phi-3-medi

声明:本文仅代表作者观点,不代表本站立场。如果侵犯到您的合法权益,请联系我们删除侵权资源!如果遇到资源链接失效,请您通过评论或工单的方式通知管理员。未经允许,不得转载,本站所有资源文章禁止商业使用运营!
下载安装【程序员客栈】APP
实时对接需求、及时收发消息、丰富的开放项目需求、随时随地查看项目状态

评论