本模型是qwen/Qwen2-1.5B的DashInfer格式模型。

requirements

pip install modelscope dashinfer jinja2 tabulate torch transformers

下载模型

from modelscope import snapshot_download
model_dir = snapshot_download("dash-infer/Qwen2-1.5B-DI")

模型推理

参考如下python代码。

```python import copy import random

from modelscope import snapshot_download from dashinfer.helper import EngineHelper, ConfigManager

modelpath = snapshotdownload("dash-infer/Qwen2-1.5B-DI")

configfile = modelpath + "/" + "diconfig.json" config = ConfigManager.getconfigfromjson(configfile) config["modelpath"] = model_path

init EngineHelper class

enginehelper = EngineHelper(config) enginehelper.verbose = True enginehelper.inittokenizer(model_path)

init engine

enginehelper.initengine()

prepare inputs and generation configs

userinput = "浙江的省会在哪" prompt = "<|imstart|>" + "system\n" + "You are a helpful assistant." + "<|imend|>\n" + \ "<|imstart|>" + "user\n" + userinput + "<|imend|>\n" + \ "<|imstart|>" + "assistant\n" gencfg = copy.deepcopy(enginehelper.defaultgencfg) gencfg["seed"] = random.randint(0, 10000) requestlist = enginehelper.createrequest([prompt], [gencfg])

inference

enginehelper.processonerequest(requestlist[0]) enginehelper.printinferenceresultall(request_list)

enginehelper.uninitengine()