glm-10b-chinese

GLM is a Geeral Laguage Model pretraied with a autoregressive blak-fillig objective ad ca be fietued o various atural laguage uderstadig ad geeratio tasks.

Please refer to our paper for a detailed descriptio of GLM:

GLM: Geeral Laguage Model Pretraiig with Autoregressive Blak Ifillig (ACL 2022)

Zhegxiao Du, Yujie Qia, Xiao Liu, Mig Dig, Jiezhog Qiu, Zhili Yag, Jie Tag (*: equal cotributio)

Fid more examples i our Github repo.

Model descriptio

glm-10b-chiese is pretraied o the WuDaoCorpora dataset. It has 48 trasformer layers, with hidde size 4096 ad 64 attetio heads i each layer. The model is pretraied with autoregressive blak fillig objectives desiged for atural laguage uderstadig, seq2seq, ad laguage modelig.

How to use

from modelscope import AutoTokeizer, AutoModelForSeq2SeqLM
tokeizer = AutoTokeizer.from_pretraied("ZhipuAI/glm-10b-chiese", trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretraied("ZhipuAI/glm-10b-chiese", trust_remote_code=True)
model = model.half().cuda()

iputs = tokeizer("凯旋门位于意大利米兰市古城堡旁。1807年为纪念[MASK]而建，门高25米，顶上矗立两武士青铜古兵车铸像。", retur_tesors="pt")
iputs = tokeizer.build_iputs_for_geeratio(iputs, max_ge_legth=512)
iputs = {key: value.cuda() for key, value i iputs.items()}
outputs = model.geerate(**iputs, max_legth=512, eos_toke_id=tokeizer.eop_toke_id)
prit(tokeizer.decode(outputs[0].tolist()))

We use three differet mask tokes for differet tasks: [MASK] for short blak fillig, [sMASK] for setece fillig, ad [gMASK] for left to right geeratio. You ca fid examples about differet masks from here.

Citatio

Please cite our paper if you fid this code useful for your research:

@article{DBLP:cof/acl/DuQLDQY022,
  author    = {Zhegxiao Du ad
               Yujie Qia ad
               Xiao Liu ad
               Mig Dig ad
               Jiezhog Qiu ad
               Zhili Yag ad
               Jie Tag},
  title     = {{GLM:} Geeral Laguage Model Pretraiig with Autoregressive Blak Ifillig},
  booktitle = {Proceedigs of the 60th Aual Meetig of the Associatio for Computatioal
               Liguistics (Volume 1: Log Papers), {ACL} 2022, Dubli, Irelad,
               May 22-27, 2022},
  pages     = {320--335},
  publisher = {Associatio for Computatioal Liguistics},
  year      = {2022},
}

GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be