Machine Learning II

achoohomelessAI and Robotics

Oct 14, 2013 (3 years and 11 months ago)

116 views

EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
MachineLearningII
Escuderoetal.(2000)&Florianetal.(2002)
SandraK¨ubler
kuebler@sfs.uni-tuebingen.de
MachineLearningII–p.1
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
Escuderoetal.(2000a)
paper:
Escudero,Marquez,Rigau(2000):NaiveBayes
andExemplar-BasedApproachestoWordSense
DisambiguationRevisited.Proceedingsofthe14th
EuropeanConferenceonArtificialIntelligence,
ECAI’2000,Berlin.
MachineLearningII–p.2
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ModificationsofMBL
Mooney(1996):naiveBayes(NB)muchbetter
thanmemory-basedlearning(MBL)
MBLonlyslightlyabovemostfrequentsense
butMBLwithoutfeatureweightingandclass
weighting
Ng(1997):usesnewMBLapproach,designed
forsymbolicfeatures
slightlyoutperformsNB
resultsgetbetterwhenfeaturesareignored
question:findouthowcontradictoryresultscan
beexplained
MachineLearningII–p.3
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ModificationsofMBL
classweighting(exampleweighting):weight
instancebasedonitsdistancetotestinstance
featureweighting(attributeweighting):give
featureweightsdependingonhowimportant
theyareforclassification
usenewdistancemeasure:MVDM







 




EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ExperimentSettings
useDSOcorpus:ca.190000examplesof121
nounsand70verbs
1.experiment:10nouns+5verbs
nouns:age,art,car,child,church,cost,head,
interest,line,work,verbs:fall,know,set,speak,
take
2setsoffeatures
SetA:4wordsand3collocationsfromlocal
context
SetB:SetA+POStagsforSetAaswellas
broadcontextfeatures
no.offeaturesinSetB:2641–6428,
dependingonword
MachineLearningII–p.5
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ResultswithSetA
MFSNBbestMBLbestMBL+MVDM
avg.nouns
57.471.772.673.7
avg.verbs
46.657.658.160.7
all
53.366.467.268.7
allclassifiersnoticeablyaboveMFS
forMBL,highernumbersof

better,weighting
helps
MVDMisbetterthanstandarddistancemeasure
MBLbetterthanNB
MachineLearningII–p.6
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ResultsforSetB
MFSNBbestMBL
avg.nouns
57.472.264.3
avg.verbs
46.655.243.0
all
53.365.856.2
MVDMnotfeasible,tooslow
NBalmostsameresultsasforSetA
MBLdeterioratesby8.6points,onlyslightly
higherthanMFS
MachineLearningII–p.7
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ModificationsoftheAlgorithms
hypothesis:MBLsobadbecauseof
representationofbroadercontext:toomany
zeros
moodificationofMBL:similaritybasedon
numberofwordstheyhaveincommon
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
NewResults
MFS
NBposNB
MBLposMBL
avg.nouns
57.4
72.272.4
64.375.6
avg.verbs
46.6
55.255.3
43.062.1
all
53.3
65.866.0
56.270.5
posMBLmuchfaster
coulduseMVDM
resultsforposMBLmuchbetter
resultsforposNBslightlybetter
MBLbetterthanNB
posMBLresultsforSetBbetterthanMBLfor
SetA
MachineLearningII–p.9
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
Escuderoetal.(2000b)
researchquestions:
comparisonofdifferentalgorithms:naiveBayes,
Memory-Based,SNoW,decisionlists,
lazyBoosting
cross-corpusapplication
reasonforcross-corpustask:usuallyonlyone
trainingcorpusexists,notnecessarilyformthe
domainoftheintendedapplication
MachineLearningII–p.10
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ExperimentalSetting
DSOcorpusagain
13nouns,8verbs
DSOhas2parts:WallStreetJournal(A)/
BrownCorpus(B)
local+globalcontext:local=3words+POSon
eachside+collocations,global:contentwords
fromsentence
MachineLearningII–p.11
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
FirstResults
A+B/A+BA+B/AA+B/B
A/AB/B
A/BB/A
MFS
46.5553.939.21
55.9445.52
36.4038.71
NB
61.5567.2555.85
65.8656.80
41.3847.66
MBL
63.0169.0856.97
68.9857.36
45.3251.13
dec.lists
61.5867.6455.53
67.5756.56
43.0148.83
SNoW
60.9265.5756.28
67.1256.13
44.0749.76
lazyBoost
66.3271.7960.85
71.2658.96
47.1051.99
whatdothosefiguresmean?
MachineLearningII–p.12
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
KappaStatistics
measurewhichisgenerallyusedfor
inter-annotatoragreement
linguisticmyth:akappaabove0.6isgood!
here:kappabetweendifferentsystems:
agreementbetweensystems
largekappabetweensystemandDSO
good,
systemlearnswhatititshould
largekappabetweenMFCandsystem
bad,
becausesystemhastendencytoselectmost
frequentsense
MachineLearningII–p.13
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
DoesTuningHelp?
tuning:addsmallamountofnewdomaindatato
trainingdata
doesthatimproveresults?
whatdoesfigures1tellus?
MachineLearningII–p.14
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
Florianetal.(2002)
idea:combineclassifierstogetbetter
performance
problem:howtomakesurethatonecombines
thepositiveclassificationsofthesystems
manydifferentpossibilities
MachineLearningII–p.15
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
ExperimentalSetup
systems:differntversionofNaiveBayes,
decisionlists,transformation-basedlearning,
mixturemodel
data:SENSEVAL-2dataforEnglish,Spanish,
Swedish,andBasque
featureset:veryelaborate,includinglemma
information,syntacticfeatures,bigramand
trigramcollocations
MachineLearningII–p.16
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
CombiningClassifiers
simplestway:voting
eachsystemselectsonewordsenseandhas
onevote,sensewithhighestvoteisselected
forsystemswithprobabilityoutput,votefor
sensewithhighestprobability
advantage:systemswhichmakeharddecisions
canbeaddedwithoutproblems
disadvantage:close/uncertaindecisionsare
ignored
MachineLearningII–p.17
EBERHARDKARLSUNIVERSITÄTTÜBINGEN
Seminarf¨urSprachwissenschaft
[
C
L
SfS
]
MoreComplexVoting
addmixtureclassifier
sumupprobabilitiesof
differentsystemsforasense,binarizethis
number,andaddasifitweretheoutputofa
system
rank-basedvoting:systemsvoteforevery
sense,withweightwhichisinversely
proportionaltorankofsenseinoutput
problem:non-probabilisticoutpusneedstobe
convertedtoprobabilisticrepresetnation
confidence-basedcombination
performance-basedcombination
meta-voting
MachineLearningII–p.18