6.881: Natural Language Processing

huntcopywriterAI and Robotics

Oct 24, 2013 (3 years and 7 months ago)

108 views

6.881:NaturalLanguageProcessing
MachineTranslationII
PhilippKoehn
CSandAILab
MIT
November23,2004
–p.1
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Outline
p

LectureI
–IntroductiontoMachineTranslation
–PrinciplesofStatisticalMT
–Word-BasedModels
–Phrase-BasedModels

LectureII
–BeamSearchDecoding
–Evaluation
–TheChallengeofSyntax
PhilippKoehn,CSAIL,MIT2
–p.2
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Phrase-BasedTranslation
p
Morgen fliege ich nach Kanada zur Konferenz
Tomorrow I will fly to the conference in Canada

Foreigninputissegmentedinphrases
–anysequenceofwords,notnecessarilylinguisticallymotivated

EachphraseistranslatedintoEnglish

Phrasesarereordered
PhilippKoehn,CSAIL,MIT3
–p.3
6.881:NaturalLanguageProcessing—MachineTranslationII
p
DecodingAlgorithm
p

Goalofthedecodingalgorithm:
Putmodelstowork,performtheactualtranslation
PhilippKoehn,CSAIL,MIT4
–p.4
6.881:NaturalLanguageProcessing—MachineTranslationII
p
GreedyDecoder
p
Maria no daba una bofetada a la bruja verde
Mary no give a slap to the witch green
Mary no give a slap to the green witch
Mary no give a slap the green witch
Mary did not give a slap the green witch
GLOSS
SWAP
ERASE
INSERT
Mary not give a slap the green witch
CHANGE
Mary did not slap the green witch
JOIN

GreedyHill-climbing[Germann,2003]
–startwithgloss
–improveprobabilitywithactions
–use2-steplook-aheadtoavoidsomelocalminima
PhilippKoehn,CSAIL,MIT5
–p.5
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
b!"#$%$!&$
'*+,!-,.$$
-&*"'$b*/,0$-$

Buildtranslationlefttoright
–selectforeignwordstobetranslated
PhilippKoehn,CSAIL,MIT6
–p.6
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a%aria
no
%ar+
ver.e/aa
.iounabo0e1a.a

Buildtranslationlefttoright
–selectforeignwordstobetranslated
–findEnglishphrasetranslation
–addEnglishphrasetoendofpartialtranslation
PhilippKoehn,CSAIL,MIT7
–p.7
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a
nover+elaa
+iounabofe/a+a
0ar1
0aria

Buildtranslationlefttoright
–selectforeignwordstobetranslated
–findEnglishphrasetranslation
–addEnglishphrasetoendofpartialtranslation
–markforeignwordsastranslated
PhilippKoehn,CSAIL,MIT8
–p.8
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a
Maria
no
Mary
did not
/erde1aa
diounabo2etada

Onetomanytranslation
PhilippKoehn,CSAIL,MIT9
–p.9
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a
Maria
no
dio una bofe/ada
Marydid no/
12a3
4erde2aa

Manytoonetranslation
PhilippKoehn,CSAIL,MIT10
–p.10
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a
Maria
nodio una bofetada
Marydid not1la3
t4e
5erde
a la

Manytoonetranslation
PhilippKoehn,CSAIL,MIT11
–p.11
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a
Maria
nodio una bofetadaa la
Mar1did notsla3t4e
5reen
6erde

Reordering
PhilippKoehn,CSAIL,MIT12
–p.12
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Beam-SearchDecodingProcess
p
bru#a
Maria
witch
nover1e
Mary1i13not45a6the7reen
1io3una3bo8eta1aa35a

Translationfinished
PhilippKoehn,CSAIL,MIT13
–p.13
6.881:NaturalLanguageProcessing—MachineTranslationII
p
TranslationOptions
p
bof#tada
unadio
alav#rd#brujanoMaria
Mary
not
did not
giv#a4laptot6#7it86gr##n
by
to t6#
to
gr##n 7it86
t6# 7it86
did not giv#
no
a 4lap
4lap
t6#
4lap

Lookuppossiblephrasetranslations
–manydifferentwaystosegmentwordsintophrases
–manydifferentwaystotranslateeachphrase
PhilippKoehn,CSAIL,MIT14
–p.14
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisExpansion
p
dio
alaverdebrujanoMaria
Mary
not
did not
giveaslaptothewitchgreen
by
to the
to
green witch
the witch
did not give
no
a slap
slap
the
slap
e: 9: --------- p: ;
unabo9etada

Startwithnullhypothesis
–e:noEnglishwords
–f:noforeignwordscovered
–p:probability1
PhilippKoehn,CSAIL,MIT15
–p.15
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisExpansion
p
dio
alaverdebrujanoMaria
Mary
not
did not
giveaslaptothewitchgreen
by
to the
to
green witch
the witch
did not give
no
a slap
slap
the
slap
e: Mary f: *;;;;;;;;p: <=>?
e: f: ;;;;;;;;;p: @
unabofetada

Picktranslationoption

Createhypothesis
–e:addEnglishphrase
Mary
–f:firstforeignwordcovered
–p:probability0.534
PhilippKoehn,CSAIL,MIT16
–p.16
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisExpansion
p
dio
alaverdebrujanoMaria
Mary
not
did not
giveaslaptothewitchgreen
by
to the
to
green witch
the witch
did not give
no
a slap
slap
the
slap
e: Mary f: *-------- p: <=>?
e: witch f: -------*- p: <@8B
e: f: --------- p: @
unabofetada

Addanotherhypothesis
PhilippKoehn,CSAIL,MIT17
–p.17
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisExpansion
p
dio una bofetada
alaverdebrujano
Maria
Mary
not
did not
giveaslaptothewitchgreen
by
to the
to
green witch
the witch
did not give
no
a slap
slap
the
slap
e: Mary f: *-------- p: .=>?
e: witch f: -------*- p: .18B
e: f: --------- p: 1
e: ... slap f: *-***---- p: .0?>

Furtherhypothesisexpansion
PhilippKoehn,CSAIL,MIT18
–p.18
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisExpansion
p
dio una bofetada
bruja verde
Maria
Mary
not
did not
giveaslaptothewitchgreen
by
to the
to
green witch
the witch
did not give
no
a slap
slap
the
slap
e: Mary f: *-------- p: .=34
e: witch f: -------*- p: .18B
e: f: --------- p: 1
e: slap f: *-***---- p: .043
e: did not f: **------- p: .1=4
e: slap f: *****---- p: .01=
e: the f: *******-- p: .004B83
e:green witch f: ********* p: .000BD1
a lano

...untilallforeignwordscovered
–findbesthypothesisthatcoversallforeignwords
–backtracktoreadofftranslation
PhilippKoehn,CSAIL,MIT19
–p.19
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisExpansion
p
Mary
not
did not
giveaslaptothewitchgreen
by
to the
to
green witch
the witch
did not give
no
a slap
slap
the
slap
e: Mary f: *-------- p: .534
e: witch f: -------*- p: .182
e: f: --------- p: 1
e: slap f: *-***---- p: .043
e: did not f: **------- p: .154
e: slap f: *****---- p: .015
e: the f: *******-- p: .004283
e:green witch f: ********* p: .000271
no
dio
alaverdebruDanoMaria
unabofetada

Addingmorehypothesis
Explosionofsearchspace
PhilippKoehn,CSAIL,MIT20
–p.20
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ExplosionofSearchSpace
p

Numberofhypothesesisexponentialwithrespectto
sentencelength
DecodingisNP-complete[Knight,1999]
Needtoreducesearchspace
–riskfree:hypothesisrecombination
–risky:histogram/thresholdpruning
PhilippKoehn,CSAIL,MIT21
–p.21
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisRecombination
p
p=1
Mary
did not give
give
did not
p=0.534
p=0.164
p=0.092 p=0.044
p=0.092

Differentpathstothesamepartialtranslation
PhilippKoehn,CSAIL,MIT22
–p.22
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisRecombination
p
p=1
Mary
did not give
give
did not
p=0.534
p=0.164
p=0.092
p=0.092

Differentpathstothesamepartialtranslation
Combinepaths
–dropweakerhypothesis
–keeppointerfromworsepath
PhilippKoehn,CSAIL,MIT23
–p.23
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisRecombination
p
p=1
Mary
did not give
give
did not
p=0.534
p=0.164
p=0.092
Joe
did not give
p=0.092
p=0.017

Recombinedhypothesesdonothavetomatchcompletely

Nomatterwhatisadded,weakerpathcanbedropped,if:
–lasttwoEnglishwordsmatch(mattersforlanguagemodel)
–foreignwordcoveragevectorsmatch(effectsfuturepath)
PhilippKoehn,CSAIL,MIT24
–p.24
6.881:NaturalLanguageProcessing—MachineTranslationII
p
HypothesisRecombination
p
p=1
Mary
did not give
give
did not
p=0.534
p=0.164
p=0.092
Joe
did not give
p=0.092

Recombinedhypothesesdonothavetomatchcompletely

Nomatterwhatisadded,weakerpathcanbedropped,if:
–lasttwoEnglishwordsmatch(mattersforlanguagemodel)
–foreignwordcoveragevectorsmatch(effectsfuturepath)
Combinepaths
PhilippKoehn,CSAIL,MIT25
–p.25
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Pruning
p

Hypothesisrecombinationisnotsufficient
Heuristicallydiscardweakhypotheses

OrganizeHypothesisinstacks,e.g.by
–sameforeignwordscovered
–samenumberofforeignwordscovered(Pharaohdoesthis)
–samenumberofEnglishwordsproduced

Comparehypothesesinstacks,discardbadones
–histogrampruning:keeptop

hypothesesineachstack(e.g.,

=100)
–thresholdpruning:keephypothesesthatareatmost

timesthecostof
besthypothesisinstack(e.g.,

=0.001)
PhilippKoehn,CSAIL,MIT26
–p.26
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ComparingHypotheses
p

Comparinghypotheseswithsamenumberofforeign
wordscovered
Maria no
e: Mary did not f: **------- p: 0.154
a la
e: the f: -----**-- p: 0.354
dio una bofetada
bruja verde
;'--'"
1!"-#!7
-"!%>7!-#&%
?&='">
'!>#'"$1!"-
00@$7&A'"$?&>-

Hypothesisthatcoverseasypartofsentenceispreferred
Needtoconsiderfuturecost
PhilippKoehn,CSAIL,MIT27
–p.27
6.881:NaturalLanguageProcessing—MachineTranslationII
p
FutureCostEstimation
p

Estimatecosttotranslateremainingpartofinput

Step1:findcheapesttranslationoptions
–findcheapesttranslationoptionforeachinputspan
–computetranslationmodelcost
–estimatelanguagemodelcost(nopriorcontext)
–ignorereorderingmodelcost

Step2:computecheapestcost
–foreachcontiguousspan:
–findcheapestsequenceoftranslationoptions

Precomputeandlookup
–precomputefuturecostforeachcontiguousspan
–futurecostforanycoveragevector:
sumofcostofeachcontiguousspanofuncoveredwords

noexpensivecomputationduringruntime
PhilippKoehn,CSAIL,MIT28
–p.28
6.881:NaturalLanguageProcessing—MachineTranslationII
p
WordLatticeGeneration
p
p=1
Mary
did not give
give
did not
p=0.534
p=0.164
p=0.092
Joe
did not give
p=0.092

Searchgraphcanbeeasilyconvertedintoawordlattice
–canbefurtherminedforn-bestlists

enablesrerankingapproaches

enablesdiscriminativetraining
Mary
did not give
give
did not
Joe
did not give
PhilippKoehn,CSAIL,MIT29
–p.29
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Evaluation
p

ManualEvaluation
–humanjudgeoutput
–expensive

AutomaticEvaluation
–machinesjudgeoutput
–fast
–reliable?

Task-OrientedEvaluation
–humansdotaskwithMT
–testsusefulnessofMT
PhilippKoehn,CSAIL,MIT30
–p.30
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ManualEvaluation
p

Correctyes/no
–simple
–longersentencesalmostalwayshaveatleastoneerror

Correctonscale
–0=bad,5=perfect
–disagreementbetweenjudges

Moredetailedjudgments
–adequacy:howwellismeaningpreserved?
–fluency:isitgoodEnglish?
–...
PhilippKoehn,CSAIL,MIT31
–p.31
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ManualEvaluation
p

Givegradefrom0=badto5=perfect
–IntheFirstTwoMonthsGuangdong’sExportofHigh-TechProducts3.76
BillionUSDollars
–TheGuangdongprovincialforeigntradeandeconomicgrowthhasmade
importantcontributions.
–SuicideexplosioninJerusalem

Agreement
PhilippKoehn,CSAIL,MIT32
–p.32
6.881:NaturalLanguageProcessing—MachineTranslationII
p
AutomaticEvaluation
p

Whyautomaticevaluationmetrics?
–manualevaluationistooslow
–evaluationonlargetestsetsrevealsminorimprovements
–automatictuningtoimprovemachinetranslationperformance

History
–WordErrorRate
–BLEUsince2002
–BLEUinshort:overlapwithreferencetranslations
PhilippKoehn,CSAIL,MIT33
–p.33
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Bi-LingualEvaluationUnderstudy(BLEU)
p
PhilippKoehn,CSAIL,MIT34
–p.34
6.881:NaturalLanguageProcessing—MachineTranslationII
p
CorrelationwithManualEvaluation
p

Correlateswithhumanevaluation(adequacy,fluency)
PhilippKoehn,CSAIL,MIT35
–p.35
6.881:NaturalLanguageProcessing—MachineTranslationII
p
TheChallengeofSyntax
p
foreign
words
foreign
syntax
foreign
semantics
interlingua
english
semantics
english
syntax
english
words

Rememberthepyramid
PhilippKoehn,CSAIL,MIT36
–p.36
6.881:NaturalLanguageProcessing—MachineTranslationII
p
AdvantagesofSyntax-BasedTranslation
p

Reorderingforsyntacticreasons
–e.g.,moveGermanobjecttoendofsentence

Betterexplanationforfunctionwords
–e.g.,prepositions,determiners

Conditioningtosyntacticallyrelatedwords
–translationofverbmaydependonsubjectorobject

Useofsyntacticlanguagemodels
PhilippKoehn,CSAIL,MIT37
–p.37
6.881:NaturalLanguageProcessing—MachineTranslationII
p
InversionTransductionGrammars
p

GenerationofbothEnglishandforeigntrees[Wu,1997]

Rules(binaryandunary)





 
 








 
 


 





 
Commonbinarytreerequired
–limitsthecomplexityofreorderings
PhilippKoehn,CSAIL,MIT38
–p.38
6.881:NaturalLanguageProcessing—MachineTranslationII
p
SyntaxTrees
p
Mary did not slap the green witch

Englishbinarytree
PhilippKoehn,CSAIL,MIT39
–p.39
6.881:NaturalLanguageProcessing—MachineTranslationII
p
SyntaxTrees(2)
p
Maria no daba una bofetada a la bruja verde

Spanishbinarytree
PhilippKoehn,CSAIL,MIT40
–p.40
6.881:NaturalLanguageProcessing—MachineTranslationII
p
SyntaxTrees(3)
p
Mary
Maria
did
*
not
no
slap daba
*
una
*
bofetada
* a
the
la
green verde
witch bruja

CombinedtreewithreorderingofSpanish

Cansuchtreesbelearnedfromdata?

Docommontreeexistwithrealsyntaxonbothsides?
PhilippKoehn,CSAIL,MIT41
–p.41
6.881:NaturalLanguageProcessing—MachineTranslationII
p
DependencyStructure
p
gekauft
bought
hatte
had
auto
car das the
ich
I

Commondependencytree

Interestindependency-basedtranslationmodels
PhilippKoehn,CSAIL,MIT42
–p.42
6.881:NaturalLanguageProcessing—MachineTranslationII
p
StringtoTreeTranslation
p
foreign
words
foreign
syntax
foreign
semantics
interlingua
english
semantics
english
syntax
english
words

UseofEnglishsyntaxtrees[YamadaandKnight,2001]
–exploitrichresourcesontheEnglishside
–obtainedwithstatisticalparser[Collins,1997]
–flattenedtreetoallowmorereorderings
–workswellwithsyntacticlanguagemodel
PhilippKoehn,CSAIL,MIT43
–p.43
6.881:NaturalLanguageProcessing—MachineTranslationII
p
YamadaandKnight[2001]
p
VB
VB1VB2
VB
TO
TO
MN
PRP
headores
listening
tomusic
VB
VB1VB2
VB
TO
TO
MN
PRP
headores
listening
tomusic
VB
VB1VB2
VB
TO
TO
MN
PRP
headores
listening
tomusic
no
ha
ga
desu
VB
VB1VB2
VB
TO
TO
MN
PRP
hadaisuki
kiku
woongaku
no
kare
ga
desu
reorder
insert
translate
take leaves
Kare ha ongaku wo kiku no ga daisuki desu
PhilippKoehn,CSAIL,MIT44
–p.44
6.881:NaturalLanguageProcessing—MachineTranslationII
p
SyntacticLanguageModel
p

Goodsyntaxtree
6.881:NaturalLanguageProcessing—MachineTranslationII
p
StringtoTreeTransferandSyntacticLM
p

WorkpresentedatthisMTSummitbyCharniak,Knight,
Yamada
–moregrammaticalcorrectoutput
–moreperfectlytranslatedsentences
–...butnoimprovementinBLEU

SyntactictransferandLMontopofphrasetranslation
–parsealatticegeneratedbyphrase-basedMT
–noresultsyet
PhilippKoehn,CSAIL,MIT46
–p.46
6.881:NaturalLanguageProcessing—MachineTranslationII
p
AugmentModelswithSyntacticFeatures
p

Intuition:othermodelsworkfine,syntaxprovides
additionalclues

Definesyntacticpropertiesthatshouldhold
–preservationofplural
–outputshouldhaveverb
–nodroppingofcontentwords
–...

2003summerworkshopatJohnHopkins:little
improvement
PhilippKoehn,CSAIL,MIT47
–p.47
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ClauseStructure
p
S PPER-SB I*+ ,A.I/-HD 23453 ,P-6C PPER-DA I+838 /P-6A AR9-6A 5:3 AD;-/< 38=sp43*+38538 //-/< A8@34ku8g38 ,,.I/ aus+a385:g38 $  S-G6 <6US-CP 5a@:= PPER-SB S:3 ,P-6C PDS-6A 5as AD;D-G6 3v38=u3ll PP-G6 APRD-G6 b3: AR9-DA 534 //-/< Abs=:@@u8g ,,I/. u3b3483+@38 ,G.I/ kL38838$ 
I 2:ll yLu =+3 *L443spL85:8g *L@@38=s pass L8  sL =+a= yLu =+a= p34+aps :8 =+3 vL=3 :8*lu53 *a8
GAI/
CAUSE
SUB-
6RDI/A9E
CAUSE

SyntaxtreefromGermanparser
–statisticalparserbyAmitDubay,trainedonTIGERtreebank
PhilippKoehn,CSAIL,MIT48
–p.48
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ReorderingWhenTranslating
p
S PPER-SB Ich VAFIN-HD werde PPER-DA Ihnen NP-OA ART-OA die ADJ-NK entsprechenden NN-NK Anmerkungen VVFIN aushaendigen$, ,S-MO KOUS-CP damit PPER-SB Sie PDS-OA das ADJD-MO eventuell PP-MO APRD-MO bei ART-DA der NN-NK Abstimmung VVINF uebernehmen VMFIN koennen$. .
I will you the corresponding comments pass on, so that you that perhaps in the vote include can.

ReorderingwhentranslatingintoEnglish
–treeisflattened
–clauselevelconstituentslineup
PhilippKoehn,CSAIL,MIT49
–p.49
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ClauseLevelReordering
p
S PPER-SB Ich VAFIN-HD werde PPER-DA Ihnen NP-OA ART-OA die ADJ-NK entsprechenden NN-NK Anmerkungen VVFIN aushaendigen$, ,S-MO KOUS-CP damit PPER-SB Sie PDS-OA das ADJD-MO eventuell PP-MO APRD-MO bei ART-DA der NN-NK Abstimmung VVINF uebernehmen VMFIN koennen$. .
I will you the corresponding comments pass on, so that you that perhaps in the vote include can.
1 2 Q 5 3
1 2 6 Q U 5 3

Clauselevelreorderingisawelldefinedtask
–labelGermanconstituentswiththeirEnglishorder
–donethisfor300sentences,twoannotators,highagreement
PhilippKoehn,CSAIL,MIT50
–p.50
6.881:NaturalLanguageProcessing—MachineTranslationII
p
SystematicReorderingGerman
English
p

Manytypesofreorderingsaresystematic
–moveverbgrouptogether
–subject-verb-object
–movenegationinfrontofverb
Writerulesbyhand
–applyrulestotestandtrainingdata
–trainstandardphrase-basedSMTsystem
System
BLEU
baselinesystem
25.2%
withmanualrules
26.8%
PhilippKoehn,CSAIL,MIT51
–p.51
6.881:NaturalLanguageProcessing—MachineTranslationII
p
Integration
p
ff'e
transformation
phrase-based
statistical MT

Transformfintof’withourmethods

Translaten-bestrestructuringswithphrase-basedMT
–usesbothtransformationscoreandtranslation/languagemodelscore
–ifnorestructuring

baselineperformance

Transformationdoesnotneedtobeperfect
–phrase-basedmodelmaystillreorder
PhilippKoehn,CSAIL,MIT52
–p.52
6.881:NaturalLanguageProcessing—MachineTranslationII
p
ImprovedTranslations
p

we
mustalso
thiscriticism
shouldbetaken
seriously.

we
mustalsotake
thiscriticismseriously.

i
amwithhim
thatitisnecessary,theinstitutionalbalancebymeansofa
politicalrevaluationofboththecommissionandthecouncil
tomaintain
.

i
agreewithhiminthis
,thatitisnecessary
tomaintain
theinstitutional
balancebymeansofapoliticalrevaluationofboththecommissionandthe
council.

thirdly,we
believethat
theprincipleofdifferentiationofnegotiations
note
.

thirdly,we
maintain
theprincipleofdifferentiationofnegotiations.

perhaps
itwouldbe
aconstructivedialogbetweenthegovernmentand
oppositionparties,socialrepresentativeapositiveimpetusintheright
direction.

perhapsaconstructivedialogbetweengovernmentandoppositionparties
andsocialrepresentative
couldgive
apositiveimpetusintherightdirection.
PhilippKoehn,CSAIL,MIT53
–p.53