码迷,mamicode.com
首页 > Web开发 > 详细

对Kaldi nnet3进行奇异值分解(SVD)以减小模型大小

时间:2019-06-21 13:08:58      阅读:463      评论:0      收藏:0      [点我收藏+]

标签:percent   upd   功耗   基础   tdd   std   gradient   组件   raw   

用处

基于SVD实现模型压缩以适配低功耗平台

? ?

根据nnet3bin/nnet3-copynnet3-copynnet3-am-copy的"--edits-config"参数中,新支持了以下选项:

apply-svd name=<name-pattern> bottleneck-dim=<dim>

查找所有名字与<name-pattern>匹配的组件,类型需要是AffineComponent或其子类。如果<dim>小于组件的输入或输出维数,则对组件参数进行奇异值分解,只保留最大<dim>奇异值,将这些组件替换为两个组件:LinearComponentNaturalGradientAffineComponent(的序列)。又见‘reduce-rank‘

? ?

示例cd

dir=`mktemp -d`

nnet3-am-copy --edits=‘apply-svd name=*.affine bottleneck-dim=64‘ $dir/final.mdl $dir/final_svd.mdl

? ?

vimdiff <(nnet3-info --print-args=false $dir/final.raw 2>&1|sort) <(nnet3-info --print-args=false $dir/final.raw 2>&1|sort)

? ?

component-node name=tdnn1.affine component=tdnn1.affine input=lda input-dim=195 output-dim=1024

component name=tdnn1.affine type=NaturalGradientAffineComponent, input-dim=195, output-dim=1024, learning-rate=0.00136, max-change=0.75, linear-params-rms=0.4864, linear-params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.92,1.0,1.1,1.2 1.3,1.6,6.4,8.8,10 11,13,14,17), mean=5.79, stddev=3.56], linear-params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(5.0,5.5,8.3,12 13,14,15,17,18 19,20,20,24), mean=15.4, stddev=2.46], bias-{mean,stddev}=-0.06099,0.2027, rank-in=20, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

component-node name=tdnn1.affine_a component=tdnn1.affine_a input=lda input-dim=195 output-dim=64

component-node name=tdnn1.affine_b component=tdnn1.affine_b input=tdnn1.affine_a input-dim=64 output-dim=1024

component name=tdnn1.affine_a type=LinearComponent, input-dim=195, output-dim=64, learning-rate=0.00136, max-change=0.75, params-rms=0.3461, params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(4.0,4.0,4.0,4.1 4.2,4.3,4.8,5.2,5.5 5.6,5.8,5.8,6.0), mean=4.81, stddev=0.496], params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.36,0.45,0.75,1.9 2.0,2.3,2.7,3.2,3.4 3.6,3.7,3.9,4.7), mean=2.7, stddev=0.618], use-natural-gradient=true, rank-in=40, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

component name=tdnn1.affine_b type=NaturalGradientAffineComponent, input-dim=64, output-dim=1024, learning-rate=0.00136, max-change=0.75, linear-params-rms=0.151, linear-params-row-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(0.12,0.13,0.14,0.16 0.19,0.23,1.1,1.6,1.8 2.1,2.5,2.6,3.8), mean=1, stddev=0.676], linear-params-col-norms=[percentiles(0,1,2,5 10,20,50,80,90 95,98,99,100)=(4.0,4.0,4.0,4.1 4.2,4.3,4.8,5.2,5.5 5.6,5.8,5.8,6.0), mean=4.81, stddev=0.496], bias-{mean,stddev}=-0.06099,0.2027, rank-in=20, rank-out=80, num-samples-history=2000, update-period=4, alpha=4

结果

经过解码测试,SVD后的模型识别率极差,完全无法使用。

需要再对模型进行retrain

? ?

使用SVD实现模型压缩后,再进行几轮迭代

在已有训练样本的情况在,假设总iteration=2000,,将final.mdl进行SVD得到final_svd.mdl,再链接为0.mdl,运行一个epochlocal/chain/run_tdnn.sh --stage 16 --num_epochs 1

在之前的epoch的基础上,再训几个epochs

local/chain/run_tdnn.sh --stage 16 --num_epochs 2 --train_stage 1155

对Kaldi nnet3进行奇异值分解(SVD)以减小模型大小

标签:percent   upd   功耗   基础   tdd   std   gradient   组件   raw   

原文地址:https://www.cnblogs.com/JarvanWang/p/11063702.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!