Music Notation, Music Representation ... - Music Informatics

almondpitterpatterAI and Robotics

Feb 23, 2014 (3 years and 1 month ago)

65 views

1

Music Notation,

Music Representation,

AND

Intelligence

Donald Byrd

School of Music, Indiana University

3 February 2005

minor rev. 5 February

2

Overview


Music representations: from abstract to concrete
(notation)


Try to assume
just the right things:

e.g., no
knowledge of music or notation


For motivation, focus on real
-
world content
-
based
music
-
IR situations


Organization of the Talk

I. Motivation: Why is this Important and/or Interesting?

II. Representation and Semantics

III. Music Notation, Representation, and Intelligence

IV. Conclusions

3

You Are Here

I. Motivation: Why is this Important and/or
Interesting?

II. Representation and Semantics

III. Music Notation, Representation, and
Intelligence

IV. Conclusions

4

Audio
-
to
-
Audio Music “Retrieval”


“Shazam
-

just hit 2580 on your mobile phone
and identify music”


Query:


Match:


Fantastically impressive to many people


Have they solved all the problems of music
IR? No, (almost) none!


Reason: intended signal & match are identical
=> no time warping, let alone higher
-
level
problems (perception/cognition)

5

Similarity Scale for Content
-
Based Music IR

1. Same music, arrangement, performance, & recording
(Shazam)

2. Same music, arrangement, performance; different recording

3. Same music, arrangement; different performance, recording

4. Same music,

different arrangement; or
different but closely
-
related
music,

e.g., simpler variations (Mozart, etc.), minor revs.
(OMRAS,
etc.)

5. Different & less closely
-
related music: freer variations (Schumann, etc.),
extensive revisions

(AI)

6. Music in same genre, etc.

(AI?)

7. Music influenced by other music

(AI!)

Relationship categories describing what’s in common between
items whose similarity is to be evaluated (from closest to most
distant)



For material in notation form, distinctions among (1), (2),
and (3) don’t apply: it’s just “Same music, arrangement”

6

OMRAS Polyphonic Audio Music IR: A Task
that Needs Note Representation


Query (audio
-
> MIDI
-
> audio)



Match (original audio recording)



Started with recordings of Bach preludes and fugues



Did polyphonic (several notes at once) music recognition



Polyphonic audio
-
> events is
an open research problem


Converted results to MIDI, used as queries against database of c.
3000 pieces in MIDI form



One of worst
-
sounding cases: Prelude in G Major from Well
-
Tempered Clavier, Book I


Outcome: the actual piece was ranked 1st!



Models built from notation database, but note data only

7

Basic Representations of Music & Audio

Audio (e.g., CD, MP3):
like speech

Time
-
stamped Events
(e.g., MIDI file): like
unformatted text

Music Notation (sheet
music): like HTML text

8

Basic Representations of Music & Audio



Audio

Time
-
stamped Events

Music Notation



Common examples

CD, MP3 file

Standard MIDI File

Sheet music


Unit

Sample


Event

Note, clef, lyric, etc.


Explicit structure

none


little (partial voicing


much (complete





information)


voicing information)



Converting to form with less explicit structure (to left): moderately
difficult


Converting to form with more explicit structure (to right):
very

difficult



MIDI = Musical Instrument Digital Interface: simple, very standard
low
-
bandwidth protocol (from early 1980’s)

9

Music
-
IR Problems that Needs More Structure


Joan Public’s problem: find a song, given some of the
melody and some lyrics


Needs notes and text (lyrics)


Common question for music librarians, esp. in public libraries


Musicologist’s problem: authorship/origin of works in
manuscripts


Full symbolic data is important, even “insignificant” details of
notation (John Howard)

10

You Are Here

I. Motivation: Why is this Important and/or
Interesting?

II. Representation and Semantics

III. Music Notation, Representation, and
Intelligence

IV. Conclusions

11

Representation, from Abstract to Concrete



Cf. Basic Representations of Music & Audio


Abstract: represention: semantics only


Intermediate: syntax (mapping rules)?


Concrete


for use by computers:
encoding


for use by humans: if visual,
notation

(involves graphics
and/or typography)


Analogous to knowledge representation vs. data
structure


12

Semantics in Music


Denotation (explicit, well
-
defined)...


vs. Connotation (implicit, ill
-
defined)


In text


Two “definitions” of pig:


1. Ugh! Dirty, evil
-
smelling creatures, wallowing in filthy
sties! (Hayakawa)


2. Mammal with short legs, cloven hoofs, bristly hair, and a
cartilaginous snout used for digging (Amer. Heritage)


Prose is “mostly” denotation


Poetry is art => connotation much more important


Music is always art, &
only

connotation!


Major issue for content
-
based music IR

13

From Representation to Notation


Choosing a representation inevitably introduces bias


Given a representation, choosing notation inevitably
introduces more bias


Important to consider the purpose (R. Davis et al;
Wiggins et al)


For huge body of important music, we have no choice:
notation is CMN (Conventional Music Notation)!


Really “CWMN” (W = Western)


Alternative for some music: tablature (guitar, lute, etc.)


CMN is among the most successful notations ever...


but enormously complex and subtle

14

Notation Says Much about Representation


CMN standard for Western music after c.1650


Evolved for “classical” music, but heavily used
for very wide range (pop, jazz, folk, etc.)


Composers/arrangers/transcribers have
pushed it hard => reveals things about music
representation in general


Will concentrate on notation (CMN)

15

You Are Here

I. Motivation: Why is this Important and/or
Interesting?

II. Representation and Semantics

III. Music Notation, Representation, and
Intelligence

IV. Conclusions

16

How to Read Music (CMN) Without Really
Trying: The Basics


Four basic parameters of a musical note

1.
Pitch:

how high or low sound is

2.
Duration:
how long the note lasts

3.
Loudness:
perceptual analog of amplitude

4.
Timbre

or tone quality


Above in decreasing order of importance for most Western music


Principles of CMN

(
& e

1)

1.
Pitch

on vertical axis: clef gives offset (“zero”)

2a.
Duration

indicated by note/rest shapes

2b. Start times (sum of durations in the voice) on horizontal axis

3.
Loudness

indicated by signs like
p , mf ,
etc.

4.
Timbre
indicated with words like “violin”, “horn”, “pizzicato”

17

Why is Musical Information Hard to
Handle?

1. Units of meaning: not clear anything in music is
analogous to words
(all representations)

2.

Polyphony: “parallel” independent voices, something
like characters in a play
(all representations)

3.

Recognizing notes
(audio only)

4.

Other reasons

18

Units of Meaning (Problem 1)


Not clear anything in music is analogous to words



No explicit delimiters (like Chinese)


Experts don’t agree on “word” boundaries (unlike Chinese)


Are
notes

like words?


No. Relative, not absolute, pitch is important


Are
pitch intervals

like words?


No. They’re too low level: more like characters


Are
pitch
-
interval sequences

like words?


In some ways, but


Ignores note durations


Ignores relationships between voices (harmony)


Probably little correlation with semantics

19

Independent Voices in Music
(Problem 2)
(
& e

2)

J.S. Bach: “St. Anne” Fugue, beginning

20

Independent Voices in Text

MARLENE. What I fancy is a rare steak. Gret?

ISABELLA. I am of course a member of the / Church of England.*

GRET. Potatoes.

MARLENE. *I haven’t been to church for years. / I like Christmas carols.

ISABELLA. Good works matter more than church attendance.




--
Caryl Churchill: “Top Girls” (1982), Act 1, Scene 1

M: What I fancy is a rare steak. Gret?



I haven’t been...


I:


I am of course a member of the Church of England.

G:


Potatoes.

Performance (time goes from left to right):

21

Complex Notation: Multiple Voices
(
& e

3)

Multiple voices on a staff rapidly gets worse with
more than 2 (Telemann “Liebe, Liebe”):



2 voices in mm. 5
-
6: not bad: stem direction is enough



3 voices in m. 7: notes must move sideways



4 voices in m. 8: almost unreadable

without
color!



Still acceptable because specific voice is rarely important

22

Problems: Example 1 (superficial but
interesting)


Ravel work has slur with 7 inflection points


Impressive, but complexity is purely graphical


No big deal in terms of representation


…but influence of performance on notation is revealing

23

Duration and Higher
-
Level Concepts of
Time


Schubert Impromptu

(
& e

4)


Measures: everything between barlines


Time signature: 3/4 = 3 quarter notes per
measure


Triplets: 3 notes in the time normally used by 2


General concept is
tuplets


24

Problems: Example 2 (Deep)


Chopin Nocturne has nasty situation
(
& e

5)



One notehead is triplet in one voice, but
normal duration in another


“Semantics” (execution) well
-
defined, obvious


Note starts 1/16 before barline…


But also

(2/3)*(1/16) before barline! How to play?


Reason: musical necessity


Solution for performer: “rubato”


Solution for music IR program: ?

25

Problems: Example 3 (Medium)


Bach: time signature change in middle of measure



(
& e

6)


“Semantics” well
-
defined and obvious


Measure has duration of 18 16ths…


But
not until the middle of the measure!


How does this make sense?


Triplets express same relationship as equivalent
simple/compound meter


Invisible (unmarked) triplets


Cf. Bach Prelude: two time signatures at once
(
& e

7)


Reason: avoid clutter

26

Problem 4 (Medium)


Brahms Capriccio
(
& e

8)


Time signature 6/8 => measure lasts 12 16ths


A dotted half note always lasts 12 16ths…


but here it clearly lasts only 11 16ths!


Reason: avoid clutter

27

Two Ways to Have Two Clefs at Once


Clef gives vertical offset to determine pitch


Debussy
(
& e

9)


Bizarrely obvious something odd involving clefs


Ravel
(
& e

10)


Only comparing time signature (3/8) and note
durations makes it clear both clefs affect whole
measure


Reason: save space (by avoiding a 3rd staff)

28

Surprise: Music Notation has Meta
-
Principles! (1)

1. Maximize readability (intelligibility)


Avoid clutter = “Omit Needless Symbols”


Try to assume
just the right things
for audience


Audience for CMN is (primarily) performers


General principle of
any

communication


Applies to talks as well as music notation!


Examples: Schubert, Bach, Brahms

29

Surprise: Music Notation has Meta
-
Principles! (2)

2. Minimize space used


Save space => fewer page turns (helps performer);
also cheaper to print (helps publisher)


Squeezing much music into little space is a
major

factor in complexity of CMN


Especially important for music: real
-
time, hands full


Examples: Telemann, Debussy, Ravel

30

The “Rules” of Music Notation


Tempting to assume that rules of such an elaborate &
successful system as CMN work (self
-
consistent, reasonably
unambiguous, etc.) in every case


But (a) “rules” evolved, with no established authority; (b) many
of the “rules” are very nebulous


In common cases, there's no problem


If you try to make every rule as precise as possible, result is
certainly
not

self
-
consistent


Trying to save space makes rules interact; something has to
give!



31

Music Notation Software and Intelligence


Despite odd notation, really nothing strange going on
in almost all of these examples


Ravel slur, Debussy & Ravel 2 simultaneous clefs, Bach &
Schubert invisible triplets, Brahms “short” dotted
-
half note,
Telemann 4 voices/staff are all simple situations


Chopin Nocturne is complex


Programmers try to help users by having programs
do things “automatically”


A good idea
if

software knows enough to do the right
thing “almost all” the time

but no program does!


Notation programs convert CMN to performance
(MIDI) and vice
-
versa => requires shallow
“semantics”; makes things
much

harder

32

You Are Here

I. Motivation: Why is this Important and/or
Interesting?

II. Representation and Semantics

III. Music Notation, Representation, and
Intelligence

IV. Conclusions

33

Conclusions: Review (1)


Representations express Semantics


Semantics of Music; Denotation & Connotation


Principles of CMN


Meta
-
Principles of CMN

1. Maximize readability; Omit Needless Symbols


Try to assume
just the right things
for audience


General principle of
any

communication

2. Minimize space used


Save space => fewer page turns, less paper

34

Conclusions: Review (2)


We need CMN or equivalent to solve spectrum
of music
-
IR (and other music
-
IT) problems


But CMN can’t represent everything we want


Even when it can, may not, at least explicitly


Need high
-
level intelligence to interpret


Solution: unknown


Likely to require major funding :
-
)

35

Conclusions: Why is this
really

Important
and/or Interesting?


Some problems directly related to other areas
of informatics


Example: Approximate string matching in
bioinformatics


Encourages progress on real semantics


Connotation is an important part of meaning in
everything


Can often ignore, but
any

semantics in arts forces
you to deal with connotation


Music is at least as quantifiable as any art, so likely
to be more tractable than others!

36

You Are Here


The End