Trasformer-AttetioIsAllYouNeed
Chaier-basedPythoimplemetatioofTrasformer,aattetio-basedseq2seqmodelwithoutcovolutioadrecurrece.Ifyouwattoseethearchitecture,pleaseseeet.py.
See"AttetioIsAllYouNeed",AshishVaswai,NoamShazeer,NikiParmar,JakobUszkoreit,LlioJoes,AidaN.Gomez,LukaszKaiser,IlliaPolosukhi,arxiv,2017.
Thisrepositoryispartlyderivedfrommycovolutioalseq2seqrepo,whichisalsoderivedfromChaier'sofficialseq2seqexample.
RequiremetPytho3.6.0+Chaier2.0.0+umpy1.12.1+cupy1.0.0+(ifusiggpu)ltkprogressbar(Youcaistallallthroughpip)adtheirdepedeciesPrepareDatasetYoucauseayparallelcorpus.Forexample,ru
shdowload_wmt.shwhichdowloadsaddecompressestraiigdatasetaddevelopmetdatasetfromWMT/europalitoyourcurretdirectory.Thesefilesadtheirpathsaresetitraiigscripttrai.pyasdefault.
HowtoRuPYTHONIOENCODING=utf-8pytho-utrai.py-g=0-iDATA_DIR-oSAVE_DIRDurigtraiig,logsforloss,perplexity,wordaccuracyadtimearepritedatacertaiiterval,iadditiotovalidatiotests(perplexityadBLEUforgeeratio)everyhalfepoch.Adalso,geeratiotestisperformedadpritedforcheckigtraiigprogress.
ArgumetsSomeofthemisasfollows:
-g:yourgpuid.Ifcpu,set-1.-iDATA_DIR,-sSOURCE,-tTARGET,-svalidSVALID,-tvalidTVALID:DATA_DIRdirectoryeedstoicludeapairoftraiigdatasetSOURCEadTARGETwithapairofvalidatiodatasetSVALIDadTVALID.Eachpairshouldbeparallellcorpuswithlie-by-liesetecealigmet.-oSAVE_DIR:JSONlogreportfileadamodelsapshotwillbesavediSAVE_DIRdirectory(ifitdoesotexist,itwillbeautomaticallymade).-e:maxepochsoftraiigcorpus.-b:miibatchsize.-u:sizeofuitsadwordembeddigs.-l:umberoflayersiboththeecoderadthedecoder.--source-vocab:maxsizeofvocabularysetofsourcelaguage--target-vocab:maxsizeofvocabularysetoftargetlaguagePleaseseetheothersbypythotrai.py-h.
NoteThisrepositorydoesotaimforcompletevalidatioofresultsithepaper,soIhaveoteagerlycofirmedvalidityofperformace.But,Iexpectmyimplemetatioisalmostcompatiblewithamodeldescribedithepaper.SomediffereceswhereIamawareareasfollows:
Optimizatio/traiigstrategy.Detailediformatioaboutbatchsize,parameteriitializatio,etc.isuclearithepaper.Additioally,thelearigrateproposedithepapermayworkolywithalargebatchsize(e.g.4000)fordeeplayerets.Ichagedwarmup_stepto32000from4000,thoughthereisroomforimprovemet.Ialsochagedreluitoleakyreluifeedforwardetlayersforeasygradietpropagatio.Vocabularyset,dataset,preprocessigadevaluatio.Thisrepousesacommoword-basedtokeizatio,althoughthepaperusesbyte-pairecodig.Sizeoftokesetalsodiffers.Evaluatio(validatio)islittleufairadicompatiblewithoeithepaper,e.g.,evevalidatiosetreplacesukowwordstoasigle"uk"toke.BeamsearchisuusediBLEUcalculatio.Modelsize.Thesettigofamodelithisrepoisoeof"basemodel"ithepaper,althoughyoucamodifysomeliesforusig"bigmodel".Thiscodefollowssomesettigsuseditesor2tesorrepository,whichicludesaTrasformermodel.Forexample,positioalecodiguseditherepositoryseemstodifferfromoeithepaper.Thiscodefollowstheformeroe.
评论