PolyLM多语言文本生成模型（PolyLM-文本生成模型-多语言-13B-演示）

模型简介

PolyLM是一个通晓多种语言的大规模语言模型，涵盖中文、英文、西班牙语、法语、德语、俄语、葡萄牙语、意大利语、阿拉伯语、日语、韩语、泰语、越南语和印尼语等18个语言。该模型可以应用于对话问答、文本生成、机器翻译和情感分析等领域，能够自动生成高质量的多语言文本，从而为跨语言、文化的交流提供便利。

Abstract i Eglish

Large laguage models (LLMs) demostrate remarkable ability to comprehed, reaso, ad geerate followig ature laguage istructios. However, the developmet of LLMs has bee primarily focused o high-resource laguages, such as Eglish, thereby limitig their applicability ad research i other laguages. Cosequetly, we preset PolyLM, a multiligual LLM traied o 640 billio (B) tokes, avaliable i two model sizes: 1.7B ad 13B. To ehace its multiligual capabilities, we 1) itegrate biligual data ito traiig data; ad 2) adopt a curriculum learig strategy that icreases the proportio of o-Eglish data from 30% i the first stage to 60% i the fial stage durig pre-traiig. Further, we propose a multiligual self-istruct method which automatically geerates 132.7K diverse multiligual istructios for model fie-tuig. To assess the model's performace, we collect several existig multiligual tasks, icludig multiligual uderstadig, questio aswerig, geeratio, ad traslatio. Extesive experimets show that PolyLM surpasses other ope-source models such as LLaMA ad BLOOM o multiligual tasks while maitaiig comparable performace i Eglish.

Our models, aloe with the multiligual istructio data, are available at Github ad Huggigface.

模型版本

本项目提供了一系列不同规模和用途的模型，参数规模包括1.7B/13B版本（当前模型为13B版本），同时涵盖了预训练底座模型以及指令精调后的Chat版本（即MultiAlpaca系列）。全部版本如下表所示：

实验结果

多语言理解实验结果

多语言问答和机器翻译实验结果

模型下载

 git lfs istall
 git cloe https://www.modelscope.c/damo/lp_polylm_13b_text_geeratio.git

模型使用

# git cloe https://github.com/modelscope/modelscope
# cd modelscope
# pip istall .

from modelscope.pipelies import pipelie
from modelscope.utils.costat import Tasks
from modelscope import sapshot_dowload

polylm_13b_model_id = 'damo/lp_polylm_13b_text_geeratio'
revisio = 'v1.0.3'

model_dir = sapshot_dowload(polylm_13b_model_id, revisio)

iput_text = f"Beijig is the capital of Chia.\Traslate this setece from Eglish to Chiese."

kwargs = {"do_sample": False, "um_beams": 4, "max_ew_tokes": 128, "early_stoppig": True, "eos_toke_id": 2}
pipelie_is = pipelie(Tasks.text_geeratio, model=model_dir)

result = pipelie_is(iput_text, **kwargs)
prit(result['text'])

微调(SFT)

代码链接: https://github.com/modelscope/swift/tree/mai/examples/pytorch/llm

1. 支持的sft方法: lora, qlora, 全参数微调, …
2. 支持的模型: qwe系列, qwe-vl系列, baichua系列, chatglm2系列, llama系列, opebuddy-llama系列, iterlm系列, xverse系列, …
3. 支持的特性: 模型量化, DDP, 模型并行, gradiet checkpoitig, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Aget SFT, 多轮对话, …

使用qlora+ddp+deepspeed SFT polylm-13b的脚本 (需要2*13GB显存)

# https://github.com/modelscope/swift/blob/mai/examples/pytorch/llm/scripts/polylm_13b/qlora_ddp_ds/sft.sh
# Experimetal eviromet: 2 * A10
# 2 * 13GB GPU memory
proc_per_ode=2

PYTHONPATH=../../.. \
CUDA_VISIBLE_DEVICES=0,1 \
torchru \
    --proc_per_ode=$proc_per_ode \
    --master_port 29500 \
    src/llm_sft.py \
    --model_type polylm-13b \
    --sft_type lora \
    --template_type default-geeratio \
    --dtype bf16 \
    --output_dir output \
    --ddp_backed ccl \
    --dtype bf16 \
    --dataset advertise-ge-zh \
    --trai_dataset_sample 20000 \
    --um_trai_epochs 1 \
    --max_legth 2048 \
    --quatizatio_bit 4 \
    --bb_4bit_comp_dtype bf16 \
    --lora_rak 8 \
    --lora_alpha 32 \
    --lora_dropout_p 0. \
    --lora_target_modules ALL \
    --gradiet_checkpoitig true \
    --batch_size 1 \
    --weight_decay 0. \
    --learig_rate 1e-4 \
    --gradiet_accumulatio_steps $(expr 16 / $proc_per_ode) \
    --max_grad_orm 0.5 \
    --warmup_ratio 0.03 \
    --eval_steps 100 \
    --save_steps 100 \
    --save_total_limit 2 \
    --loggig_steps 10 \
    --push_to_hub false \
    --hub_model_id polylm-13b-qlora \
    --hub_private_repo true \
    --hub_toke 'your-sdk-toke' \
    --deepspeed_cofig_path 'ds_cofig/zero2.jso' \
    --oly_save_model true \

论文引用

如果你觉得这个该模型对有所帮助，请考虑引用下面的相关的论文：

@misc{wei2023polylm,
      title={PolyLM: A Ope Source Polyglot Large Laguage Model}, 
      author={Xiagpeg Wei ad Haora Wei ad Hua Li ad Tiahao Li ad Pei Zhag ad Xigzhag Re ad Mei Li ad Yu Wa ad Zhiwei Cao ad Bibi Xie ad Tiaxiag Hu ad Shagjie Li ad Biyua Hui ad Bowe Yu ad Dayiheg Liu ad Baosog Yag ad Fei Huag ad Ju Xie},
      year={2023},
      eprit={2307.06018},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

PolyLM多语言文本生成模型（PolyLM-文本生成模型-多语言-13B-演示） 模型简介 PolyLM是一个通晓多种语言的大规模语言模型，涵盖中文、英文、西班牙语、法语、德语、俄语、葡萄牙语、意大