Information Communication Theory

addictedswimmingΤεχνίτη Νοημοσύνη και Ρομποτική

24 Οκτ 2013 (πριν από 4 χρόνια και 15 μέρες)

168 εμφανίσεις

Information

Communication

Theory

Kentaro

Inui (


健太郎
)

Naoaki

Okazaki (
岡崎

直観
)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

1

(情報伝達学)

Course Plan


Part I

Okazaki



10/04: Introduction


10/11: Classification


10/18: Part
-
of
-
speech
tagging


10/25: Syntactic
parsing


11/01: Statistical
parsing



Part II

Inui



11/08: Features
and unification


11/15: Representation
of meaning


11/22: Computational
semantics


11/29: Computational
lexical
semantics


12/06:
(no class)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

2


Part III

Inui, Okazaki, TAs




12/13, 12/20, 2013/01/10, 01/17, 01/24


Programming exercises and project from
Natural
Language Processing with Python

by Steven Bird



Lectures given at
計算機大演習室(
New Student Laboratory
Building for Information Engineering,
情報新棟
1
階)

Course Format


Text

optional



Jurafsky
, Daniel and Martin, James H.
Speech
and
Language Processing.

Prentice
-
Hall, 2009

2nd Edition





\
6,000
available at amazon.co.jp


Bird, Steven et al.
Natural
Language Processing with
Python
.
Oreilly

& Associates
Inc., 2009


萩原

正人

中山


広,水野

貴明




入門

自然言語処理


O'Reilly Japan,
2010



Grading


Exercises

given in lectures

: 40%


Final report

programming project


2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

3

Handouts


If necessary, please print out a handout and bring
it to the class by yourself


Alternatively, browse it on your laptop



Handouts will be available at (before dawn):


http://
www.cl.ecei.tohoku.ac.jp/index.php?InformationCo
mmunicationTheory


Username: nlp2012


Password:
chukougishitsu


2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

4

Contact Information


Office
hours:


Tue, 1:00
-
2:30pm
or
by appointment



Office
:


Room 305

108 after Nov

,
Electrical Engineering and
Applied Physics Research Building No.
3
(電気系
3
号館)



Contact:


inui@ecei.tohoku.ac.jp

@
inuikentaro


okazaki@ecei.ecei.tohoku.ac.jp

@
chokkanorg

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

5

Introduction

Naoaki

Okazaki

okazaki@ecei.tohoku.ac.jp

http://www.chokkan.org/

http://twitter.com/#!/
chokkanorg

#
nlptohoku

http
://
www.chokkan.org/lectures/2012nlp/p/01.pdf

Information Communication Theory (
情報伝達学
)

6

2011
-
10
-
04

Natural Language Processing (NLP)


Giving computers the ability to process human language


As old as the idea of computers themselves!


Implementations

and
implications

of the exciting idea


The long
-
awaited dream (that has not come true yet)

Information Communication Theory (
情報伝達学
)

7

Doraemon

C
-
3PO

(Star Wars)

Atom

(
Astro

boy)

2011
-
10
-
04

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

8

What are needs to be done for
understanding languages

as humans do?

Part I:
Knowledge (
disciplines)

Lexical semantics (
語彙
意味論
)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

9

How much Chinese silk was exported to Western
Europe by the end of the 18
th

century
?

Meaning of words

Compositional semantics (
合成意味論
)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

10

How much Chinese silk was exported to Western
Europe by the end of the 18
th

century
?

Meaning of constituents

1700

1720

1740

1760

1780

1800

The 18
th

Century

t
he end

of

Compositional??? (with adjectives)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

11

g
irl friend

former

hole

black

towel

wine

white

white

!?

Morphology (
形態論
)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

12

How much Chinese silk was exported to Western
Europe by the end of the 18
th

century
?

Study on word formations

(breaking words down into morphemes)


Inflection (
屈折
)


is


was


being


been


export


exports


exporting


exported


exported



Derivation (
派生
)


China


Chinese


West


Western

Syntax (
統語論,文法
)


Part
-
of
-
speech (POS): Lecture #3


Categorization of words, e.g., nouns, verbs, adjectives, adverbs



Constituency: Lectures #4 and #5


Grouping words that may behave as a single unit or phrase


e.g., noun phrase, verb phrase, prepositional phrase



Grammatical relations: Lecture #5


Relationship between words/constituents

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

13

Principles and rules for constructing

phrases and sentences

Syntactic tagging and parsing


Assign a structure to an input sentence

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

14

Economic news had little effect on financial markets .

JJ

NN

VBD

JJ

NN

IN

JJ

NNS

NP

NP

NP

PP

NP

VP

PU

S

nmod

nmod

nmod

sbj

obj

nmod

pc

p

Constituent parsing

Dependency parsing

Nivre

and
Kubler

(2006)

POS tagging

Semantic role (
意味役割
)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

15

How much Chinese silk was exported to Western
Europe
by the end of the 18
th

century
?

1700

1720

1740

1760

1780

1800

The 18
th

Century

How much Chinese silk was exported to Western
Europe
by southern merchants
?

TEMPORAL

AGENT

Coreference

(

参照
)

U: Where is The Green Hornet playing in Mountain View?

S: The Green Hornet is playing at the Century 16 theatre.

U: When is
it

playing there?

S:
It
’s playing at 2pm, 5pm, and 8pm.

U: I’d like 1 adult and 2 children for
the first show
.



How much would
that

cost?

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

16

What does “it” refers to?

What does “the first show” refers to?

What does “that” refers to?

We can guess
these easily!

Coreference

(

参照
)

U: Where is
The Green Hornet
playing in Mountain View?

S:
The Green Hornet
is playing
at the Century 16 theatre
.

U: When is
it

playing
there
?

S:
It
’s playing at
2pm
, 5pm, and 8pm.

U: I’d like
1 adult and 2 children
for
the first show
.



How much would
that

cost?

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

17

How words like
that

or pronouns like
it

refer
to previous parts of the discourse

Pragmatics (
語用論
)


Bob
: Are you coming to the party?


Jane:
I’m afraid

I can’t.



Bob: Are you coming to the party?


Jane: You know,
I’m really busy
.



Bob:
Could you
pass me the sugar?


Jane: Yes.
Here you are
.

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

18

Actions that speakers intend

by their use of text

Discourse (
談話
)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

19

http://www.isi.edu/~marcu/discourse/tagging
-
ref
-
manual.pdf

Coherent structured
groups of text

Various knowledge about languages


Morphology (
形態論
): meaningful components within words


Syntax (
文法
): structural relationships between words


Semantics (
意味論
): meanings of words, phrases, sentences


Discourse (
談話
): relationships across/beyond different
sentences or statements; contextual processing


Pragmatic (
語用論
):
relationship of meaning to the goals and
intentions of speakers; how we use languages to
communicate


World knowledge (
世界知識
): facts of the world; common
sense

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

20

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

21

What are needs to be done for
understanding languages

as humans do?

Part II: Ambiguity

Ambiguity


We may build multiple, alternative linguistic
structures

and interpretations for a single input


I made her duck

(see more examples later)




Disambiguation

(or resolution): to decide which
linguistic/semantic structure/interpretation is the
most appropriate (in the context)

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

22

Part
-
of
-
speech tagging and ambiguity

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

23

Time

flies

like

an

arrow

.

NN

VBZ

IN

DT

NN

.

(光陰
矢の
ごとし)

VB

NNS

IN

DT

NN

.


ハエの速度を矢のように測定せよ


NN

NNS

VBP

DT

NN

.


時蠅は矢を好む


Attachment ambiguity (1/3)


I saw the girl on the hill with a telescope
.







I saw the
girl on
the hill with a telescope
.



2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

24

Attachment ambiguity (2/3)


I saw the girl on the hill with a telescope
.







I saw the
girl on
the hill with a telescope
.



2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

25

Attachment ambiguity (3/3)


I saw the girl on the hill with a telescope
.







I saw the
girl on
the hill with a telescope
.



2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

26


Put
[[
the insects in the box
]

and
[
the bowl on the table
]]







Put the insects in
[[
the box
]

and
[
the bowl on the table
]]

Coordination ambiguity

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

27

Semantic ambiguity


S
yntax structure is insufficient to represent the meaning



Distinction between syntax and semantics


Colorless green ideas sleep
furiously

(Chomsky, 1957)



Opposite


John bought a book from Mary
vs

Mary sold a book to John



Lexical ambiguity


I went to the
bank…

(of the river) or (to get some money)



Quantifier


Every man loves a woman


2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

28

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

29

The state
-
of
-
the
-
art of

Natural Language Processing

Commercial world


A lot of exciting staff going on…

Information Communication Theory (
情報伝達学
)

30

2011
-
10
-
04

Machine translation (Google)


Information Communication Theory (
情報伝達学
)

31

2011
-
10
-
04

Machine translation (Google)


Information Communication Theory (
情報伝達学
)

32

2011
-
10
-
04

Watson (IBM)


Question answering system built on IBM’s
DeepQA

technology


14
-
16
February 2011,
Watson beat
two human competitors, the
biggest all
-
time money winner on Jeopardy
!
and
the
record
holder for the longest championship
streak


Hardware


2880 processor cores (3.5 GHz POWER7 eight core processors)


16 TB RAM in total



Software


Written in Java and C++


Using Apache
Hadoop

framework for distributed computing


Data


200M pages (about 1M books) of
structured and unstructured
content


Consuming 4T
of disk storage


Encyclopedias
, dictionaries, thesauri, newswire articles,
literary
works

Information Communication Theory (
情報伝達学
)

33

http://en.wikipedia.org/wiki/Watson_(computer)

2011
-
10
-
04

Jeopardy!


American quiz show featuring


history, literature, the arts, pop culture, science, sports,
geography,
wordplay, etc.


Six categories
are announced, each with
five
trivia
clues


A correct response adds the
dollar value


An
incorrect response or a failure to respond
within a five
-
second time limit
deducts
the dollar
value

Information Communication Theory (
情報伝達学
)

34

http://en.wikipedia.org/wiki/Jeopardy!

2011
-
10
-
04

Final Jeopardy! and the Future of Watson


Watch the video (08:58):


http://www.youtube.com/watch?v=Wq0XnBYC3nQ

Information Communication Theory (
情報伝達学
)

35

2011
-
10
-
04

Science behind an answer


Watch the very nice video (06:42):


http://www.youtube.com/watch?v=DywO4zksfXw

Information Communication Theory (
情報伝達学
)

36

2011
-
10
-
04

Science behind an answer


Step 1: Question analysis


What is type of question being asked?


What is the question asking for?


Step 2: Hypothesis generation


Search millions of documents for possible answers


Step 3: Hypothesis and evidence scoring


Collect positive and negative evidences to support each answer


Score evidences based on everything from source material
reliability to whether time and locations appear correct


Parallelized evidence scoring for each possible answer


Step 4: Final merging and ranking


Learn the importance of each evidence by practicing games


Yield the final ranking of possible answers


Decide whether Watson answers the question or not based on the
confidence

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

37

A shame (of NLP)


Japanese translation of the book, “Einstein: His
Life and Universe,” published on 23 June 2011


Information Communication Theory (
情報伝達学
)

38


Chapter 13 was translated by
computers, not by humans!


How this happened:
http
://
www.amazon.co.jp/review/R29GQAF5
DUOAEW/ref=cm_cr_rdp_perm


Very rare incident that an
MT’ed

book
is published


Revised version was published on 17
Aug 2011

2011
-
10
-
04

Imagine the original sentence


ボルンの妻のヘートヴィヒに最大限にしてください。(その
ヘートヴィヒは,彼の家族に関する彼の処理,今や説教さ
れた頃,彼が「自分がそのかなり不幸な回答に駆り立て
られるのを許容していない
べき
でない」と自由に彼に叱っ
た)。以上は,彼が目立つべきであり,彼女が言ったのを
「科学の人里離れている寺」に尊敬します。

Information Communication Theory (
情報伝達学
)

39


Max
Born's

wife, Hedwig, who had freely scolded Einstein
about his treatment of his family, now lectured,
“[
You
should] not have allowed yourself to be goaded into that
rather unfortunate reply
.”
He should show more respect,
she said, for
“the
secluded temple of science
.”
(P286)

2011
-
10
-
04

Passing exams for University of Tokyo


2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

40

Writing short science
fictions


2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

41

Goal of this course


Overview the issues and technologies for natural
language understanding


What is possible/easy? What is impossible/difficult?


Why is this achieved
or
not
achieved by the current
technology
?



Provide fundamental theories and techniques for
natural language processing


Some techniques are useful for other research fields



Exercise programming with real NLP tasks


You will be an experienced engineer!

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

42

Course plan

1.
4 Oct:

Introduction

2.
11 Oct:

Classification


Spam filtering, linear classifier, feature extraction,
perceptron, logistic regression, evaluation (precision,
recall, F1)

3.
18 Oct:

Part
-
of
-
speech tagging

4.
25 Oct:

Syntactic parsing

5.
1 Nov:

Statistical parsing

2011
-
10
-
04

Information Communication Theory (
情報伝達学
)

43