书生·浦语-对话-20B-4bit

LMDeploy supports LLM model iferece of 4-bit weight, with the miimum requiremet for NVIDIA graphics cards beig sm80, such as A10, A100, Geforce 30/40 series.

Before proceedig with the iferece of iterlm-chat-20b-4bit, please esure that lmdeploy is istalled.

pip istall 'lmdeploy>=0.0.11'

Iferece

Please dowload iterlm-chat-20b-4bit model as follows,

git-lfs istall
git cloe --depth=1 https://www.modelscope.c/Shaghai_AI_Laboratory/iterlm-chat-20b-4bit.git

As demostrated i the commad below, first covert the model's layout usig turbomid.deploy, ad the you ca iteract with the AI assistat i the termial

# Covert the model's layout ad store it i the default path, ./workspace.
pytho3 -m lmdeploy.serve.turbomid.deploy \
    --model-ame iterlm-chat-20b \
    --model-path ./iterlm-chat-20b-4bit \
    --model-format awq \
    --group-size 128

# iferece
pytho3 -m lmdeploy.turbomid.chat ./workspace

Serve with gradio

If you wish to iteract with the model via web UI, please iitiate the gradio server as idicated below:

pytho3 -m lmdeploy.serve.gradio.app ./workspace --server_ame {ip_addr} --server_port {port}

Subsequetly, you ca ope the website http://{ip_addr}:{port} i your browser ad iteract with the model.

Besides servig with gradio, there are two more servig methods. Oe is servig with Trito Iferece Server (TIS), ad the other is a OpeAI-like server amed as api_server.

Please refer to the user guide for detailed iformatio if you are iterested.

Iferece Performace

LMDeploy provides scripts for bechmarkig toke throughput ad request throughput.

toke throughput tests the speed of geeratig ew tokes, give a specified umber of prompt tokes ad completio tokes, while request throughput measures the umber of requests processed per miute with real dialogue data.

We coducted bechmarks o iterlm-chat-20b-4bit. Ad toke_throughput was measured by settig 256 prompt tokes ad geeratig 512 tokes i respose o A100-80G.

Note: The sessio_le i workspace/trito_models/weights/cofig.ii is chaged to 2056 i our test.

toke throughput

Ru the followig commad,

pytho bechmark/profile_geeratio.py \
  --model-path ./workspace \
  --cocurrecy 1 8 16 --prompt-tokes 256 512 512 1024 --completio-tokes 512 512 1024 1024
  --dst-csv ./toke_throughput.csv

You will fid the toke_throughput metrics i ./toke_throughput.csv

request throughput

LMDeploy uses ShareGPT dataset to test request throughput. Try the ext commads, ad you will get the rpm (request per miute) metric.

# dowload the ShareGPT dataset
wget https://huggigface.co/datasets/ao8231489123/ShareGPT_Vicua_ufiltered/resolve/mai/ShareGPT_V3_ufiltered_cleaed_split.jso

# ru bechmark script
pytho profile_throughput.py \
 ShareGPT_V3_ufiltered_cleaed_split.jso \
 ./workspace \
 --cocurrecy 16

LMDeploy supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graph