? Huggig Face • ? ModelScope • ✡️ WiseModel

? Joi us ? WeChat (Chiese) !

GPTQ quatized versio of Yi-9B-200K model.

What is Yi?

Itroductio

• ? The Yi series models are the ext geeratio of ope-source large laguage models traied from scratch by 01.AI.

• ? Targeted as a biligual laguage model ad traied o 3T multiligual corpus, the Yi series models become oe of the strogest LLM worldwide, showig promise i laguage uderstadig, commosese reasoig, readig comprehesio, ad more. For example,

• Yi-34B-Chat model laded i secod place (followig GPT-4 Turbo), outperformig other LLMs (such as GPT-4, Mixtral, Claude) o the AlpacaEval Leaderboard (based o data available up to Jauary 2024).

• Yi-34B model raked first amog all existig ope-source models (such as Falco-180B, Llama-70B, Claude) i both Eglish ad Chiese o various bechmarks, icludig Huggig Face Ope LLM Leaderboard (pre-traied) ad C-Eval (based o data available up to November 2023).

• ? (Credits to Llama) Thaks to the Trasformer ad Llama ope-source commuities, as they reduce the efforts required to build from scratch ad eable the utilizatio of the same tools withi the AI ecosystem.

  If you're iterested i Yi's adoptio of Llama architecture ad licese usage policy, see Yi's relatio with Llama. ⬇️

  
  

  ? TL;DR

  The Yi series models adopt the same model architecture as Llama but are NOT derivatives of Llama.

  • Both Yi ad Llama are all based o the Trasformer structure, which has bee the stadard architecture for large laguage models sice 2018.

  • Grouded i the Trasformer architecture, Llama has become a ew corerstoe for the majority of state-of-the-art ope-source models due to its excellet stability, reliable covergece, ad robust compatibility. This positios Llama as the recogized foudatioal framework for models icludig Yi.

  • Thaks to the Trasformer ad Llama architectures, other models ca leverage their power, reducig the effort required to build from scratch ad eablig the utilizatio of the same tools withi their ecosystems.

  • However, the Yi series models are NOT derivatives of Llama, as they do ot use Llama's weights.

  • As Llama's structure is employed by the majority of ope-source models, the key factors of determiig model performace are traiig datasets, traiig pipelies, ad traiig ifrastructure.

  • Developig i a uique ad proprietary way, Yi has idepedetly created its ow high-quality traiig datasets, efficiet traiig pipelies, ad robust traiig ifrastructure etirely from the groud up. This effort has led to excellet performace with Yi series models rakig just behid GPT4 ad surpassig Llama o the Alpaca Leaderboard i Dec 2023.

[ Back to top ⬆️ ]

News

[ Back to top ⬆️ ]

Models

Yi models come i multiple sizes ad cater to differet use cases. You ca also fie-tue Yi models to meet your specific requiremets.

If you wat to deploy Yi models, make sure you meet the software ad hardware requiremets.

Chat models

<sub><sup>- 4-bit series models are quatized by AWQ.
- 8-bit series models are quatized by GPTQ
- All quatized models have a low barrier to use sice they ca be deployed o cosumer-grade GPUs (e.g., 3090, 4090).</sup></sub>

Base models

<sub><sup>- 200k is roughly equivalet to 400,000 Chiese characters.
- If you wat to use the previous versio of the Yi-34B-200K (released o Nov 5, 2023), ru git checkout 069cd341d60f4ce4b07ec394e82b79e94f656cf to dowload the weight.</sup></sub>

Model ifo

• For chat ad base models

• For chat models

  For chat model limitatios, see the explaatios below. ⬇️

The released chat model has udergoe exclusive traiig usig Supervised Fie-Tuig (SFT). Compared to other stadard chat models, our model produces more diverse resposes, makig it suitable for various dowstream tasks, such as creative scearios. Furthermore, this diversity is expected to ehace the likelihood of geeratig higher quality resposes, which will be advatageous for subsequet Reiforcemet Learig (RL) traiig.
However, this higher diversity might amplify certai existig issues, icludig:
• Halluciatio: This refers to the model geeratig factually icorrect or osesical iformatio. With the model's resposes beig more varied, there's a higher chace of halluciatio that are ot based o accurate data or logical reasoig.
• No-determiism i re-geeratio: Whe attemptig to regeerate or sample resposes, icosistecies i the outcomes may occur. The icreased diversity ca lead to varyig results eve uder similar iput coditios.
• Cumulative Error: This occurs whe errors i the model's resposes compoud over time. As the model geerates more diverse resposes, the likelihood of small iaccuracies buildig up ito larger errors icreases, especially i complex tasks like exteded reasoig, mathematical problem-solvig, etc.
• To achieve more coheret ad cosistet resposes, it is advisable to adjust geeratio cofiguratio parameters such as temperature, top_p, or top_k. These adjustmets ca help i the balace betwee creativity ad coherece i the model's outputs.

[ Back to top ⬆️ ]

How to use Yi?

• Quick start
• Choose your path
• pip
• docker
• coda-lock
• llama.cpp
• Web demo
• Fie-tuig
• Quatizatio
• Deploymet
• Learig hub

Quick start

Gettig up ad ruig with Yi models is simple with multiple choices available.

Choose your path

Select oe of the followig paths to begi your jourey with Yi!

? Deploy Yi locally

If you prefer to deploy Yi models locally,

• ?‍♀️ ad you have sufficiet resources (for example, NVIDIA A800 80GB), you ca choose oe of the followig methods:

  • pip
  • Docker
  • coda-lock

• ?‍♀️ ad you have limited resources (for example, a MacBook Pro), you ca use llama.cpp.

? Not to deploy Yi locally

If you prefer ot to deploy Yi models locally, you ca explore Yi's capabilities usig ay of the followig optios.

?‍♀️ Ru Yi with APIs

If you wat to explore more features of Yi, you ca adopt oe of these methods:

• Yi APIs (Yi official)

• Early access has bee grated to some applicats. Stay tued for the ext roud of access!

• Yi APIs (Replicate)

?‍♀️ Ru Yi i playgroud

If you wat to chat with Yi with more customizable optios (e.g., system prompt, temperature, repetitio pealty, etc.), you ca try oe of the followig optios:

• Yi-34B-Chat-Playgroud (Yi official)

  • Access is available through a whitelist. Welcome to apply (fill out a form i Eglish or Chiese).

• Yi-34B-Chat-Playgroud (Replicate)

?‍♀️ Chat with Yi

If you wat to chat with Yi, you ca use oe of these olie services, which offer a similar user experiece:

• Yi-34B-Chat (Yi official o Huggig Face)

• No registratio is required.

• Yi-34B-Chat (Yi official beta)

• Access is available through a whitelist. Welcome to apply (fill out a form i Eglish or Chiese).

[ Back to top ⬆️ ]

Quick start - pip

This tutorial guides you through every step of ruig Yi-34B-Chat locally o a A800 (80G) ad the performig iferece.

Step 0: Prerequisites

• Make sure Pytho 3.10 or a later versio is istalled.

• If you wat to ru other Yi models, see software ad hardware requiremets.

Step 1: Prepare your eviromet

To set up the eviromet ad istall the required packages, execute the followig commad.

git cloe https://github.com/01-ai/Yi.git
cd yi
pip istall -r requiremets.txt

Step 2: Dowload the Yi model

You ca dowload the weights ad tokeizer of Yi models from the followig sources:

• Huggig Face
• ModelScope
• WiseModel

Step 3: Perform iferece

You ca perform iferece with Yi chat or base models as below.

Perform iferece with Yi chat model

1. Create a file amed quick_start.py ad copy the followig cotet to it.

   from trasformers import AutoModelForCausalLM, AutoTokeizer
   
   model_path = '<your-model-path>'
   
   tokeizer = AutoTokeizer.from_pretraied(model_path, use_fast=False)
   
   # Sice trasformers 4.35.0, the GPT-Q/AWQ model ca be loaded usig AutoModelForCausalLM.
   model = AutoModelForCausalLM.from_pretraied(
       model_path,
       device_map="auto",
       torch_dtype='auto'
   ).eval()
   
   # Prompt cotet: "hi"
   messages = [
       {"role": "user", "cotet": "hi"}
   ]
   
   iput_ids = tokeizer.apply_chat_template(coversatio=messages, tokeize=True, add_geeratio_prompt=True, retur_tesors='pt')
   output_ids = model.geerate(iput_ids.to('cuda'))
   respose = tokeizer.decode(output_ids[0][iput_ids.shape[1]:], skip_special_tokes=True)
   
   # Model respose: "Hello! How ca I assist you today?"
   prit(respose)
   

2. Ru quick_start.py.

   pytho quick_start.py
   

   The you ca see a output similar to the oe below. ?

   Hello! How ca I assist you today?
   

Perform iferece with Yi base model

• Yi-34B

  The steps are similar to pip - Perform iferece with Yi chat model.

  You ca use the existig file text_geeratio.py.

  pytho demo/text_geeratio.py  --model <your-model-path>

The you ca see a output similar to the oe below. ?

• Yi-9B

  Iput

  from trasformers import AutoModelForCausalLM, AutoTokeizer

  MODEL_DIR = "01-ai/Yi-9B"
  model = AutoModelForCausalLM.from_pretraied(MODEL_DIR, torch_dtype="auto")
  tokeizer = AutoTokeizer.from_pretraied(MODEL_DIR, use_fast=False)

  iput_text = "# write the quick sort algorithm"
  iputs = tokeizer(iput_text, retur_tesors="pt").to(model.device)
  outputs = model.geerate(**iputs, max_legth=256)
  prit(tokeizer.decode(outputs[0], skip_special_tokes=True))

Output

  # write the quick sort algorithm
  def quick_sort(arr):
      if le(arr) <= 1:
          retur arr
      pivot = arr[le(arr) // 2]
      left = [x for x i arr if x < pivot]
      middle = [x for x i arr if x == pivot]
      right = [x for x i arr if x > pivot]
      retur quick_sort(left) + middle + quick_sort(right)

  # test the quick sort algorithm
  prit(quick_sort([3, 6, 8, 10, 1, 2, 1]))

<p alig="right"> [
<a href="#top">Back to top ⬆️ </a>  ] 

Quick start - Docker

docker ru -it --gpus all \
-v <your-model-path>: /models
ghcr.io/01-ai/yi:latest

<p>You ca perform iferece with Yi chat or base models as below.</p>

<p>The steps are similar to <a href="#perform-iferece-with-yi-chat-model">pip - Perform iferece with Yi chat model</a>.</p>
<p><strog>Note</strog> that the oly differece is to set model_path = '&lt;your-model-mout-path&gt;' istead of model_path = '&lt;your-model-path&gt;'.</p>

<p>The steps are similar to <a href="#perform-iferece-with-yi-base-model">pip - Perform iferece with Yi base model</a>.</p>
<p><strog>Note</strog> that the oly differece is to set --model &lt;your-model-mout-path&gt;' istead of model &lt;your-model-path&gt;.</p>

Quick start - coda-lock

Quick start - llama.cpp

git cloe git@github.com:ggergaov/llama.cpp.git

GIT_LFS_SKIP_SMUDGE=1 git cloe https://huggigface.co/XeIaso/yi-chat-6B-GGUF

git-lfs pull --iclude yi-chat-6b.Q2_K.gguf

make -j4 && ./mai -m /Users/yu/yi-chat-6B-GGUF/yi-chat-6b.Q2_K.gguf -p "How do you feed your pet fox? Please aswer this questio i 6 simple steps:\Step 1:" - 384 -e

...

How do you feed your pet fox? Please aswer this questio i 6 simple steps:

Step 1: Select the appropriate food for your pet fox. You should choose high-quality, balaced prey items that are suitable for their uique dietary eeds. These could iclude live or froze mice, rats, pigeos, or other small mammals, as well as fresh fruits ad vegetables.

Step 2: Feed your pet fox oce or twice a day, depedig o the species ad its idividual prefereces. Always esure that they have access to fresh water throughout the day.

Step 3: Provide a appropriate eviromet for your pet fox. Esure it has a comfortable place to rest, plety of space to move aroud, ad opportuities to play ad exercise.

Step 4: Socialize your pet with other aimals if possible. Iteractios with other creatures ca help them develop social skills ad prevet boredom or stress.

Step 5: Regularly check for sigs of illess or discomfort i your fox. Be prepared to provide veteriary care as eeded, especially for commo issues such as parasites, detal health problems, or ifectios.

Step 6: Educate yourself about the eeds of your pet fox ad be aware of ay potetial risks or cocers that could affect their well-beig. Regularly cosult with a veteriaria to esure you are providig the best care.

...

[ Back to top ⬆️ ]

Web demo

You ca build a web UI demo for Yi chat models (ote that Yi base models are ot supported i this seario).

Step 1: Prepare your eviromet.

Step 2: Dowload the Yi model.

Step 3. To start a web service locally, ru the followig commad.

pytho demo/web_demo.py -c <your-model-path>

You ca access the web UI by eterig the address provided i the cosole ito your browser.

[ Back to top ⬆️ ]

Fie-tuig

bash fietue/scripts/ru_sft_Yi_6b.sh

Oce fiished, you ca compare the fietued model ad the base model with the followig commad:

bash fietue/scripts/ru_eval.sh

[ Back to top ⬆️ ]

Quatizatio

GPT-Q

pytho quatizatio/gptq/quat_autogptq.py \
  --model /base_model                      \
  --output_dir /quatized_model            \
  --trust_remote_code

Oce fiished, you ca the evaluate the resultig model as follows:

pytho quatizatio/gptq/eval_quatized_model.py \
  --model /quatized_model                       \
  --trust_remote_code

AWQ

pytho quatizatio/awq/quat_autoawq.py \
  --model /base_model                      \
  --output_dir /quatized_model            \
  --trust_remote_code

Oce fiished, you ca the evaluate the resultig model as follows:

pytho quatizatio/awq/eval_quatized_model.py \
  --model /quatized_model                       \
  --trust_remote_code

[ Back to top ⬆️ ]

Deploymet

If you wat to deploy Yi models, make sure you meet the software ad hardware requiremets.

Software requiremets

Before usig Yi quatized models, make sure you've istalled the correct software listed below.

Hardware requiremets

Before deployig Yi i your eviromet, make sure your hardware meets the followig requiremets.

Chat models

Below are detailed miimum VRAM requiremets uder differet batch use cases.

Base models

[ Back to top ⬆️ ]

Learig hub

Why Yi?

• Ecosystem
  • Upstream
  • Dowstream
  • Servig
  • Quatizatio
  • Fie-tuig
  • API
• Bechmarks
  • Chat model performace
  • Base model performace
  • Yi-34B ad Yi-34B-200K
  • Yi-9B

Ecosystem

Yi has a comprehesive ecosystem, offerig a rage of tools, services, ad models to erich your experieces ad maximize productivity.

• Upstream
• Dowstream
• Servig
• Quatizatio
• Fie-tuig
• API

Upstream

The Yi series models follow the same model architecture as Llama. By choosig Yi, you ca leverage existig tools, libraries, ad resources withi the Llama ecosystem, elimiatig the eed to create ew tools ad ehacig developmet efficiecy.

For example, the Yi series models are saved i the format of the Llama model. You ca directly use LlamaForCausalLM ad LlamaTokeizer to load the model. For more iformatio, see Use the chat model.

from trasformers import AutoModelForCausalLM, AutoTokeizer

tokeizer = AutoTokeizer.from_pretraied("01-ai/Yi-34b", use_fast=False)

model = AutoModelForCausalLM.from_pretraied("01-ai/Yi-34b", device_map="auto")

[ Back to top ⬆️ ]

Dowstream

? Tip

• Feel free to create a PR ad share the fatastic work you've built usig the Yi series models.

• To help others quickly uderstad your work, it is recommeded to use the format of <model-ame>: <model-itro> + <model-highlights>.

Servig

If you wat to get up with Yi i a few miutes, you ca use the followig services built upo Yi.

• Yi-34B-Chat: you ca chat with Yi usig oe of the followig platforms:

• Yi-34B-Chat | Huggig Face

• Yi-34B-Chat | Yi Platform: Note that curretly it's available through a whitelist. Welcome to apply (fill out a form i Eglish or Chiese) ad experiece it firsthad!

• Yi-6B-Chat (Replicate): you ca use this model with more optios by settig additioal parameters ad callig APIs.

• ScaleLLM: you ca use this service to ru Yi models locally with added flexibility ad customizatio.

Quatizatio

If you have limited computatioal capabilities, you ca use Yi's quatized models as follows.

These quatized models have reduced precisio but offer icreased efficiecy, such as faster iferece speed ad smaller RAM usage.

• TheBloke/Yi-34B-GPTQ
• TheBloke/Yi-34B-GGUF
• TheBloke/Yi-34B-AWQ

Fie-tuig

If you're seekig to explore the diverse capabilities withi Yi's thrivig family, you ca delve ito Yi's fie-tued models as below.

• TheBloke Models: this site hosts umerous fie-tued models derived from various LLMs icludig Yi.

  This is ot a exhaustive list for Yi, but to ame a few sorted o dowloads:

• TheBloke/dolphi-2_2-yi-34b-AWQ

• TheBloke/Yi-34B-Chat-AWQ

• TheBloke/Yi-34B-Chat-GPTQ

• SUSTech/SUS-Chat-34B: this model raked first amog all models below 70B ad outperformed the twice larger deepseek-llm-67b-chat. You ca check the result o the Ope LLM Leaderboard.

• OrioStarAI/OrioStar-Yi-34B-Chat-Llama: this model excelled beyod other models (such as GPT-4, Qwe-14B-Chat, Baichua2-13B-Chat) i C-Eval ad CMMLU evaluatios o the OpeCompass LLM Leaderboard.

• NousResearch/Nous-Capybara-34B: this model is traied with 200K cotext legth ad 3 epochs o the Capybara dataset.

API

• amazig-opeai-api: this tool coverts Yi model APIs ito the OpeAI API format out of the box.
• LlamaEdge: this tool builds a OpeAI-compatible API server for Yi-34B-Chat usig a portable Wasm (WebAssembly) file, powered by Rust.

[ Back to top ⬆️ ]

Bechmarks

• Chat model performace
• Base model performace

Chat model performace

Yi-34B-Chat model demostrates exceptioal performace, rakig first amog all existig ope-source models i the bechmarks icludig MMLU, CMMLU, BBH, GSM8k, ad more.

Base model performace

Yi-34B ad Yi-34B-200K

The Yi-34B ad Yi-34B-200K models stad out as the top performers amog ope-source models, especially excellig i MMLU, CMMLU, commo-sese reasoig, readig comprehesio, ad more.

Yi-9B

Yi-9B is almost the best amog a rage of similar-sized ope-source models (icludig Mistral-7B, SOLAR-10.7B, Gemma-7B, DeepSeek-Coder-7B-Base-v1.5 ad more), particularly excellig i code, math, commo-sese reasoig, ad readig comprehesio.

• I terms of overall ability (Mea-All), Yi-9B performs the best amog similarly sized ope-source models, surpassig DeepSeek-Coder, DeepSeek-Math, Mistral-7B, SOLAR-10.7B, ad Gemma-7B.

• I terms of codig ability (Mea-Code), Yi-9B's performace is secod oly to DeepSeek-Coder-7B, surpassig Yi-34B, SOLAR-10.7B, Mistral-7B, ad Gemma-7B.

• I terms of math ability (Mea-Math), Yi-9B's performace is secod oly to DeepSeek-Math-7B, surpassig SOLAR-10.7B, Mistral-7B, ad Gemma-7B.

• I terms of commo sese ad reasoig ability (Mea-Text), Yi-9B's performace is o par with Mistral-7B, SOLAR-10.7B, ad Gemma-7B.

[ Back to top ⬆️ ]

Who ca use Yi?

Everyoe! ? ✅

• The Yi series models are free for persoal usage, academic purposes, ad commercial use. All usage must adhere to the Yi Series Models Commuity Licese Agreemet 2.1

• For free commercial use, you oly eed to complete this form to get a Yi Model Commercial Licese.

[ Back to top ⬆️ ]

Misc.

Ackowledgmets

A heartfelt thak you to each of you who have made cotributios to the Yi commuity! You have helped Yi ot just a project, but a vibrat, growig home for iovatio.

[ Back to top ⬆️ ]

Disclaimer

We use data compliace checkig algorithms durig the traiig process, to esure the compliace of the traied model to the best of our ability. Due to complex data ad the diversity of laguage model usage scearios, we caot guaratee that the model will geerate correct, ad reasoable output i all scearios. Please be aware that there is still a risk of the model producig problematic outputs. We will ot be resposible for ay risks ad issues resultig from misuse, misguidace, illegal usage, ad related misiformatio, as well as ay associated data security cocers.

[ Back to top ⬆️ ]

Licese

The source code i this repo is licesed uder the Apache 2.0 licese. The Yi series models are fully ope for academic research ad free for commercial use, with automatic permissio grated upo applicatio. All usage must adhere to the Yi Series Models Commuity Licese Agreemet 2.1. For free commercial use, you oly eed to sed a email to get official commercial permissio.

[ Back to top ⬆️ ]

Building the Next Generation of Open-Source and Bilingual LLMs ? Hugging Face • ? ModelScope • ✡️