近年来，随着端到端语音识别的流行，基于Trasformer结构的语音识别系统逐渐成为了主流。Trasformer通过self-attetio模块来获取语音的全局信息，但对于语音识别任务，语音序列的局部信息更为关键，
例如DFSMN、TDNN等建模局部信息的网络结构在语音识别任务上取得了较好的效果。2020年，Google在Trasformer结构的基础上提出了Coformer。具体网络结构图如下，通过在self-atteio基础上叠加卷积模块来加强
模型的局部信息建模能力，进一步提升了模型的效果。Coformer已经在AISHELL-1、AISHELL-2、LibriSpeech等多个开源数据上取得了SOTA结果。 更详细的描述见：论文 识别结果输出路径结构如下： score：识别路径得分 text：语音识别结果文件 可根据使用需求对VAD和PUNC标点模型进行自由组合，使用方式如下： 若不使用PUNC模型，可配置pucmodel=""，或不传入pucmodel参数，如需加入LM模型，可增加配置lmmodel='damo/speechtrasformerlmzh-c-commo-vocab8404-pytorch'，并设置lmweight和beamsize参数。 在命令行终端执行： 注：支持单条音频文件识别，也支持文件列表，列表为kaldi风格wav.scp： 注： 注： 更多详细用法（示例） 详细用法（示例） 可以直接采用原始音频作为输入进行训练，也可以先对音频进行预处理，提取FBak特征，再进行模型训练，加快训练速度。 运行范围 使用方式 使用范围与目标场景 考虑到特征提取流程和工具以及训练工具差异，会对CER的数据带来一定的差异（<0.1%），推理GPU环境差异导致的RTF数值差异。Highlights
Coformer模型介绍
项目介绍
基于ModelScope进行推理
cat wav.scp
asr_example1 data/test/audios/asr_example1.wav
asr_example2 data/test/audios/asr_example2.wav
...
from modelscope.pipelies import pipelie
from modelscope.utils.costat import Tasks
iferece_pipelie = pipelie(
task=Tasks.auto_speech_recogitio,
model='iic/speech_coformer_asr_at-zh-c-16k-aishell1-vocab4234-pytorch', model_revisio="v2.0.4")
rec_result = iferece_pipelie('https://isv-data.oss-c-hagzhou.aliyucs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav')
prit(rec_result)
rec_result = iferece_pipelie('https://isv-data.oss-c-hagzhou.aliyucs.com/ics/MaaS/ASR/test_audio/asr_example_zh.pcm', fs=16000)
rec_result = iferece_pipelie('asr_example_zh.wav')
iferece_pipelie("wav.scp", output_dir='./output_dir')
tree output_dir/
output_dir/
└── 1best_recog
├── score
└── text
1 directory, 3 files
import soudfile
waveform, sample_rate = soudfile.read("asr_example_zh.wav")
rec_result = iferece_pipelie(waveform)
iferece_pipelie = pipelie(
task=Tasks.auto_speech_recogitio,
model='iic/speech_paraformer-large_asr_at-zh-c-16k-commo-vocab8404-pytorch', model_revisio="v2.0.4",
vad_model='iic/speech_fsm_vad_zh-c-16k-commo-pytorch', vad_model_revisio="v2.0.4",
puc_model='iic/puc_ct-trasformer_zh-c-commo-vocab272727-pytorch', puc_model_revisio="v2.0.4",
# spk_model="iic/speech_campplus_sv_zh-c_16k-commo",
# spk_model_revisio="v2.0.2",
)
基于FuASR进行推理
可执行命令行
fuasr +model=paraformer-zh +vad_model="fsm-vad" +puc_model="ct-puc" +iput=vad_example.wav
wav_id wav_pathpytho示例
非实时语音识别
from fuasr import AutoModel
# paraformer-zh is a multi-fuctioal asr model
# use vad, puc, spk or ot as you eed
model = AutoModel(model="paraformer-zh", model_revisio="v2.0.4",
vad_model="fsm-vad", vad_model_revisio="v2.0.4",
puc_model="ct-puc-c", puc_model_revisio="v2.0.4",
# spk_model="cam++", spk_model_revisio="v2.0.2",
)
res = model.geerate(iput=f"{model.model_path}/example/asr_example.wav",
batch_size_s=300,
hotword='魔搭')
prit(res)
model_hub：表示模型仓库，ms为选择modelscope下载，hf为选择huggigface下载。实时语音识别
from fuasr import AutoModel
chuk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
ecoder_chuk_look_back = 4 #umber of chuks to lookback for ecoder self-attetio
decoder_chuk_look_back = 1 #umber of ecoder chuks to lookback for decoder cross-attetio
model = AutoModel(model="paraformer-zh-streamig", model_revisio="v2.0.4")
import soudfile
import os
wav_file = os.path.joi(model.model_path, "example/asr_example.wav")
speech, sample_rate = soudfile.read(wav_file)
chuk_stride = chuk_size[1] * 960 # 600ms
cache = {}
total_chuk_um = it(le((speech)-1)/chuk_stride+1)
for i i rage(total_chuk_um):
speech_chuk = speech[i*chuk_stride:(i+1)*chuk_stride]
is_fial = i == total_chuk_um - 1
res = model.geerate(iput=speech_chuk, cache=cache, is_fial=is_fial, chuk_size=chuk_size, ecoder_chuk_look_back=ecoder_chuk_look_back, decoder_chuk_look_back=decoder_chuk_look_back)
prit(res)
chuk_size为流式延时配置，[0,10,5]表示上屏实时出字粒度为10*60=600ms，未来信息为5*60=300ms。每次推理输入为600ms（采样点数为16000*0.6=960），输出为对应文字，最后一个语音片段输入需要设置is_fial=True来强制输出最后一个字。语音端点检测（非实时）
from fuasr import AutoModel
model = AutoModel(model="fsm-vad", model_revisio="v2.0.4")
wav_file = f"{model.model_path}/example/asr_example.wav"
res = model.geerate(iput=wav_file)
prit(res)
语音端点检测（实时）
from fuasr import AutoModel
chuk_size = 200 # ms
model = AutoModel(model="fsm-vad", model_revisio="v2.0.4")
import soudfile
wav_file = f"{model.model_path}/example/vad_example.wav"
speech, sample_rate = soudfile.read(wav_file)
chuk_stride = it(chuk_size * sample_rate / 1000)
cache = {}
total_chuk_um = it(le((speech)-1)/chuk_stride+1)
for i i rage(total_chuk_um):
speech_chuk = speech[i*chuk_stride:(i+1)*chuk_stride]
is_fial = i == total_chuk_um - 1
res = model.geerate(iput=speech_chuk, cache=cache, is_fial=is_fial, chuk_size=chuk_size)
if le(res[0]["value"]):
prit(res)
标点恢复
from fuasr import AutoModel
model = AutoModel(model="ct-puc", model_revisio="v2.0.4")
res = model.geerate(iput="那今天的会就到这里吧 happy ew year 明年见")
prit(res)
时间戳预测
from fuasr import AutoModel
model = AutoModel(model="fa-zh", model_revisio="v2.0.4")
wav_file = f"{model.model_path}/example/asr_example.wav"
text_file = f"{model.model_path}/example/text.txt"
res = model.geerate(iput=(wav_file, text_file), data_type=("soud", "text"))
prit(res)
微调
预处理
数据评估及结果
model
dev(CER%)
test(CER%)
RTF
Coformer
4.42
4.87
0.2100
使用方式以及适用范围
模型局限性以及可能的偏差
相关论文以及引用信息
@article{gulati2020coformer,
title={Coformer: Covolutio-augmeted trasformer for speech recogitio},
author={Gulati, Amol ad Qi, James ad Chiu, Chug-Cheg ad Parmar, Niki ad Zhag, Yu ad Yu, Jiahui ad Ha, Wei ad Wag, Shibo ad Zhag, Zhegdog ad Wu, Yoghui ad others},
joural={arXiv preprit arXiv:2005.08100},
year={2020}
}
点击空白处退出提示
评论