BatchedSeq2SeqExampleBasedotheseq2seq-traslatio-batched.ipybfrompractical-pytorch,butmoreextrafeatures.
ThisexamplerusgrammaticalerrorcorrectiotaskwherethesourcesequeceisagrammaticallyerroeuousEglishseteceadthetargetsequeceisagrammaticallycorrectEglishsetece.Thecorpusadevaluatioscriptcabedowloadat:https://github.com/keisks/jfleg.
ExtrafeaturesCleaercodebaseVerydetailedcommetsforlearersImplemetPytorchativedatasetaddataloaderforbatchigCorrectlyhadlethehiddestatefrombidirectioalecoderadpasttothedecoderasiitialhiddestate.Fullybatchedattetiomechaismcomputatio(olyimplemetgeeralattetiobutit'ssufficiet).Note:Theorigialcodestillusesfor-looptocompute,whichisveryslow.SupportLSTMisteadofolyGRUSharedembeddigs(ecoder'siputembeddigaddecoder'siputembeddig)PretraiedGloveembeddigFixedembeddigTieembeddigs(decoder'siputembeddigaddecoder'soutputembeddig)TesorboardvisualizatioLoadadsavecheckpoitReplaceukowwordsbyselectigthesourcetokewiththehighestattetioscore.(Traslatio)CosComparigtothestate-of-the-artseq2seqlibrary,OpeNMT-py,therearesomestuffsthatare'toptimizedithiscodebase:
UseCuDNNwhepossible(alwaysoecoder,odecoderwheiput_feed=0)Alwaysavoididexig/loopsadusetorchprimitives.Whepossible,batchsoftmaxoperatiosacrosstime.(thisisthesecodcomplicatedpartofthecode)Batchifereceadbeamsearchfortraslatio(thisisthemostcomplicatedpartofthecode)HowtospeedupRNNtraiig?SeveralwaystospeedupRNNtraiig:
BatchigStaticpaddigDyamicpaddigBucketigTrucatedBPTTSee"SequeceModelsadtheRNNAPI(TesorFlowDevSummit2017)"foruderstadigthosetechiques.
YoucausetorchtextorOpeNMT'sdataiteratorforspeedigupthetraiig.Itcabe7xfaster!(ex:7hoursforaepoch->1hour!)
AckowledgemetThakstotheauthorofOpeNMT-py@srushforaswerigthequestiosforme!Seehttps://github.com/OpeNMT/OpeNMT-py/issues/552
评论