yuekai/icefall-asr-multi-zh-hans-zipformer-xl

jielong1993

Apr 1

你好，我下载了你的模型去推理aishell_test_cuts和aishell_dev_cuts，但是很奇怪wer=95.24%和97.22%（解码结果基本是空的，或者是“哦”，“嗯”之类的词）。这可能是什么原因引起的呢。这是推理脚本:

python ./zipformer/decode.py \
  --epoch 999 \
  --avg 1 \
  --causal 1 \
  --use-averaged-model False \
  --chunk-size -1 \
  --left-context-frames -1 \
  --use-ctc 1 \
  --exp-dir zipformer/exp-xl \
  --max-duration 1200 \
  --num-encoder-layers 2,3,5,6,5,3 \
  --feedforward-dim 1536,2048,3072,4096,3072,1536 \
  --encoder-dim 512,768,1024,1536,1024,512 \
  --encoder-unmasked-dim 192,192,256,320,256,192 \
  --decoder-dim 768 --joiner-dim 768 \
  --value-head-dim 18 \
  --query-head-dim 48 \
  --num-heads 4,4,4,8,4,4 \
  --blank-penalty 0.7

推理结果实例：
%WER = 97.22
Errors: 408 insertions, 164180 deletions, 35043 substitutions, over 205341 reference words (6118 correct)
Search below for sections starting with PER-UTT DETAILS:, SUBSTITUTIONS:, DELETIONS:, INSERTIONS:, PER-WORD STATS:

PER-UTT DETAILS: corr or (ref->hyp)
BAC009S0724W0121-4379: (广州市房地产中介协会分析->嗯)
BAC009S0724W0122-4270: (广州市房地产中介协会还表示->嗯)
BAC009S0724W0123-4278: (相比于其他一线城市->嗯)
BAC009S0724W0124-4449: (广州二手住宅市场表现一直相对稳健->嗯)
BAC009S0724W0125-4326: (而在股市大幅震荡的环境下->嗯)
BAC009S0724W0126-4600: (预计第三季度将陆续有部分股市资金重归楼市->嗯)
BAC009S0724W0127-4450: (但受穗六条及二套房首付七成的制约->嗯)
BAC009S0724W0128-4425: (下半年楼市能是否能回到快速上升通道依然存在变数->)
BAC009S0724W0129-4364: (其中越秀区涨幅领先->嗯)
BAC009S0724W0130-4274: (天河区的签约面积在豪宅交投增多的带动下上升较快->请)
BAC009S0724W0131-4271: (其中天河区增长幅度近百分之三->)
BAC009S0724W0132-4377: (六万平方米的刚需户型成交份额下调一个百分点->嗯)
BAC009S0724W0133-4477: (但仍以百分之四占据市场主力位置->嗯)
BAC009S0724W0134-4517: (中心六区中介促成二手住宅的十大活跃板块中->*)
BAC009S0724W0135-4542: (位于榜首的同德围罗冲围交易量与上月持平->嗯)
BAC009S0724W0136-4323: (而六月居冠的天河北成交宗数下滑近百分之二->嗯)
BAC009S0724W0137-4564: (部分高端豪宅的交投持续回暖->嗯)
BAC009S0724W0138-4572: (同样表现出色的还有五山板块的汇景新城->嗯)
BAC009S0724W0139-4330: (多个热门板块的成交涨幅较前几月有所放缓->嗯)
BAC009S0724W0140-4397: (但是改善型置业入市积极性下降->嗯)
BAC009S0724W0141-4275: (拖累广州整个二手房市成交下滑->嗯)
BAC009S0724W0142-4515: (来自合富置业的统计数据显示->嗯嗯)
BAC009S0724W0143-4347: (在七月广州二手住宅市场上->嗯)
BAC009S0724W0144-4547: (改善型换房成交个案占比下降到百分之八->嗯)

jielong1993

Apr 1

日志信息：
2025-04-01 14:13:48,004 INFO [decode.py:679] Decoding started
2025-04-01 14:13:48,005 INFO [decode.py:685] Device: cuda:0
2025-04-01 14:13:48,007 INFO [decode.py:695] {
"avg": 1,
"batch_idx_train": 0,
"beam": 20.0,
"beam_size": 4,
"best_train_epoch": -1,
"best_train_loss": Infinity,
"best_valid_epoch": -1,
"best_valid_loss": Infinity,
"blank_id": 0,
"blank_penalty": 0.7,
"bpe_model": "data/lang_bpe_2000/bpe.model",
"bucketing_sampler": true,
"causal": true,
"chunk_size": "-1",
"cnn_module_kernel": "31,31,15,15,15,31",
"concatenate_cuts": false,
"context_size": 2,
"decoder_dim": 768,
"decoding_method": "greedy_search",
"downsampling_factor": "1,2,4,8,4,2",
"drop_last": true,
"duration_factor": 1.0,
"enable_musan": true,
"enable_spec_aug": true,
"encoder_dim": "512,768,1024,1536,1024,512",
"encoder_unmasked_dim": "192,192,256,320,256,192",
"env_info": {
"IP address": "192.168.13.228",
"hostname": "akuvox-a100",
"icefall-git-branch": "master",
"icefall-git-date": "Thu Feb 27 17:10:58 2025",
"icefall-git-sha1": "db9fb8ad-dirty",
"icefall-path": "/home/akuvox-a100/HDD8T/jielong.wang/icefall",
"k2-build-type": "Release",
"k2-git-date": "Tue Oct 29 09:02:19 2024",
"k2-git-sha1": "75e2ed6b2fd87c22b7f3f34bad48a69984bb8755",
"k2-path": "/home/akuvox-a100/anaconda3/envs/icefall/lib/python3.10/site-packages/k2/init.py",
"k2-version": "1.24.4",
"k2-with-cuda": true,
"lhotse-path": "/home/akuvox-a100/anaconda3/envs/icefall/lib/python3.10/site-packages/lhotse/init.py",
"lhotse-version": "1.27.0",
"python-version": "3.10",
"torch-cuda-available": true,
"torch-cuda-version": "12.4",
"torch-version": "2.5.0+cu124"
},
"epoch": 999,
"exp_dir": "zipformer/exp-xl",
"feature_dim": 80,
"feedforward_dim": "1536,2048,3072,4096,3072,1536",
"gap": 1.0,
"input_strategy": "PrecomputedFeatures",
"iter": 0,
"joiner_dim": 768,
"lang_dir": "data/lang_bpe_2000",
"left_context_frames": "-1",
"log_interval": 50,
"manifest_dir": "data/fbank",
"max_contexts": 8,
"max_duration": 1200,
"max_states": 64,
"max_sym_per_frame": 1,
"nbest_scale": 0.5,
"ngram_lm_scale": 0.01,
"num_buckets": 30,
"num_encoder_layers": "2,3,5,6,5,3",
"num_heads": "4,4,4,8,4,4",
"num_paths": 200,
"num_workers": 2,
"on_the_fly_feats": false,
"pos_dim": 48,
"pos_head_dim": "4",
"query_head_dim": "48",
"res_dir": "zipformer/exp-xl/greedy_search",
"reset_interval": 200,
"return_cuts": true,
"shuffle": true,
"spec_aug_time_warp_factor": 80,
"subsampling_factor": 4,
"suffix": "epoch-999-avg-1-chunk--1-left-context--1-context-2-max-sym-per-frame-1-blank-penalty-0.7",
"unk_id": 2,
"use_averaged_model": false,
"use_ctc": true,
"use_transducer": true,
"valid_interval": 3000,
"value_head_dim": "18",
"vocab_size": 2000,
"warm_step": 2000
}
2025-04-01 14:13:48,008 INFO [decode.py:697] About to create model
2025-04-01 14:13:50,539 INFO [checkpoint.py:112] Loading checkpoint from zipformer/exp-xl/epoch-999.pt
2025-04-01 14:13:53,948 INFO [decode.py:798] Number of model parameters: 757552351
2025-04-01 14:13:53,948 INFO [multi_dataset.py:163] About to get multidataset test cuts
2025-04-01 14:13:53,948 INFO [multi_dataset.py:166] Loading Aishell set in lazy mode
2025-04-01 14:13:54,504 INFO [decode.py:822] Start decoding test set: aishell_test
2025-04-01 14:13:58,017 INFO [decode.py:575] batch 0/?, cuts processed until now is 172
2025-04-01 14:14:13,893 INFO [zipformer.py:1883] name=None, attn_weights_entropy = tensor([2.9672, 1.4003, 1.7877, 1.5750, 1.0856, 1.5442, 2.2182, 1.7809],
device='cuda:0')
2025-04-01 14:14:24,267 INFO [zipformer.py:1883] name=None, attn_weights_entropy = tensor([1.0037, 2.4889, 0.0507, 1.3675, 1.5859, 0.6049, 1.8824, 1.9813],
device='cuda:0')
2025-04-01 14:14:56,464 INFO [decode.py:591] The transcripts are stored in zipformer/exp-xl/greedy_search/recogs-aishell_test-greedy_search_blank_penalty_0.7-epoch-999-avg-1-chunk--1-left-context--1-context-2-max-sym-per-frame-1-blank-penalty-0.7.txt
2025-04-01 14:14:56,589 INFO [utils.py:670] [aishell_test-greedy_search_blank_penalty_0.7] %WER 95.24% [99781 / 104765, 300 ins, 82589 del, 16892 sub ]
2025-04-01 14:14:56,831 INFO [decode.py:604] Wrote detailed error stats to zipformer/exp-xl/greedy_search/errs-aishell_test-greedy_search_blank_penalty_0.7-epoch-999-avg-1-chunk--1-left-context--1-context-2-max-sym-per-frame-1-blank-penalty-0.7.txt
2025-04-01 14:14:56,831 INFO [decode.py:620]
For aishell_test, WER of different settings are:
greedy_search_blank_penalty_0.7 95.24 best for aishell_test

2025-04-01 14:14:56,831 INFO [decode.py:822] Start decoding test set: aishell_dev
2025-04-01 14:14:59,834 INFO [decode.py:575] batch 0/?, cuts processed until now is 196
2025-04-01 14:16:30,430 INFO [decode.py:575] batch 50/?, cuts processed until now is 12523
2025-04-01 14:16:41,255 INFO [zipformer.py:1883] name=None, attn_weights_entropy = tensor([3.6555, 1.4914, 2.2520, 3.0800], device='cuda:0')
2025-04-01 14:16:46,668 INFO [decode.py:591] The transcripts are stored in zipformer/exp-xl/greedy_search/recogs-aishell_dev-greedy_search_blank_penalty_0.7-epoch-999-avg-1-chunk--1-left-context--1-context-2-max-sym-per-frame-1-blank-penalty-0.7.txt
2025-04-01 14:16:46,884 INFO [utils.py:670] [aishell_dev-greedy_search_blank_penalty_0.7] %WER 97.22% [199631 / 205341, 408 ins, 164180 del, 35043 sub ]
2025-04-01 14:16:47,402 INFO [decode.py:604] Wrote detailed error stats to zipformer/exp-xl/greedy_search/errs-aishell_dev-greedy_search_blank_penalty_0.7-epoch-999-avg-1-chunk--1-left-context--1-context-2-max-sym-per-frame-1-blank-penalty-0.7.txt
2025-04-01 14:16:47,402 INFO [decode.py:620]
For aishell_dev, WER of different settings are:
greedy_search_blank_penalty_0.7 97.22 best for aishell_dev

2025-04-01 14:16:47,402 INFO [decode.py:839] Done!

yuekai

Owner Apr 1

@jielong1993 你试试不要加 blank_penalty 参数，同时，fbank 需要使用 whisper large v2的，不是 kaldi fbank, 你可以先用 speech io 的 fbank 解码试试看，这个数据比较小 https://github.com/k2-fsa/icefall/blob/master/egs/speech_llm/ASR_LLM/prepare.sh#L44

yuekai

Owner Apr 1

https://huggingface.co/yuekai/icefall_asr_speechio/tree/main/fbank

jielong1993

Apr 1

•

edited Apr 1

@jielong1993 你试试不要加 blank_penalty 参数，同时，fbank 需要使用 whisper large v2的，不是 kaldi fbank, 你可以先用 speech io 的 fbank 解码试试看，这个数据比较小 https://github.com/k2-fsa/icefall/blob/master/egs/speech_llm/ASR_LLM/prepare.sh#L44

十分感谢！我试试whisper large v2的fbank特征。
再请教下，如果我想自己训练这个配方，数据准备的时候，也必须使用 whisper large v2的fbank么。我是自己准备了kaldi的fbank，训练脚本：

python ./zipformer/train.py \
  --world-size 1 \
  --num-epochs 30 \
  --start-epoch 1 \
  --use-fp16 1 \
  --exp-dir zipformer/exp \
  --causal 1 \
  --max-duration 1000 \
  --num-workers 32

16epoch损失降到0.22左右。但是解码时也是基本预测不出来，很奇怪，我把训练集拿去解码也是如此。

yuekai

Owner Apr 1

@jielong1993 你自己训练的话，kaldi fbank 也是可以的，只不过我当时刚好有 whisper 的 fbank. 你说的训练问题，建议到 icefall 下面提个 issue, 本质上这个模型就是个大号的 zipformer, 你可以先试下他们标准的 recipe.

jielong1993

Apr 1

@jielong1993 你自己训练的话，kaldi fbank 也是可以的，只不过我当时刚好有 whisper 的 fbank. 你说的训练问题，建议到 icefall 下面提个 issue, 本质上这个模型就是个大号的 zipformer, 你可以先试下他们标准的 recipe.

好的，感谢！

automount

Apr 24

您好！我训练的过程中也遇到了删除错误非常大的情况。请问这个模型通过长时间的训练可以纠正删除错误偏高的问题吗？我看你从10epoch出现这个问题然后一直训练到999epoch是吗？

yuekai
/

icefall-asr-multi-zh-hans-zipformer-xl

解码异常