PLOrk and machine learning - Princeton University

unknownlippsΤεχνίτη Νοημοσύνη και Ρομποτική

16 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

91 εμφανίσεις

Chuck!

Parts 3 and 4: PLOrk and machine learning

Rebecca Fiebrink

Princeton University

1

Part 3: The Princeton Laptop
Orchestra


4 years old, first of its kind


Now laptop orchestras in Stanford U., Tokyo, London,
Oslo, and many others


15+ students


Learn ChucK, some Max/MSP, and digital music
composition in two classes per week


Rehearse once per week


Pieces composed by students, faculty, and guest
artists


Very hard compositional issues!




2

Dan Trueman: Silicon/Carbon


http://www.youtube.com/watch?v=YyD96blI4
EU


http://www.youtube.com/watch?v=o71j1xGvv
fA&feature=related



3

Equipment

4

Techniques


Network to synchronize computers


Sensors & game controllers


Incorporate voice & acoustic instruments


Often improvisational within a predetermined
structure


Unique conducting gestures


5

Part 4

Machine learning for live
performance

10

Machine learning for live performance


Problem: there is a
semantic gap

between the raw
data that computers use and the musical, cultural,
aesthetic meanings that humans perceive and assign.

12

Computer

Human

Audio: a series of samples

Pitch, melody, rhythm, timbre, words,
mood, genre, artist, value judgment

Gesture: a vector or matrix of sensor
outputs, sampled in time


Gesture, expression, signal, direction,
mood. E.g., a conductor’s downbeat, an
angry motion, a sign
-
language “Hello”

Video: A matrix of pixels of different colors

People, objects, motion, mood, scene,
events, places

One solution: A lot of code


What algorithm would you design to tell a
computer whether a picture contains a human
face?

13

The problem


If your algorithm doesn’t work, how can you
fix it?


You can’t easily reuse it to do a similar task
(e.g., recognizing monkey faces that are not
human)


There’s no “theory” for how to write a good
algorithm


It’s a lot of work!

14

Another solution: Machine learning
(Classification)


Classification is a
data
-
driven

approach for
applying
labels

to
data
. Once a classifier has
been
trained

on a training set that includes
the true labels, it will
predict
labels for new
data it hasn’t seen before.


15

Classifier

Wears cape

Wears

a
mask

Height

(cm)

Is a
superhero?

Yes

No

180

Yes

Yes

No

90

No

No

No

160

Yes

Yes

Yes

184

Yes

Data Set: A feature vector and
class
for every data point

Train the classifier on a labeled dataset

Run the trained classifier on new data

Classifier

Wears cape

Wears

a
mask

Height

(cm)

Is a
superhero?

Yes

No

18

?

NO!

Candidates for machine learning


Which gesture did the performer just make with
the iCube?


Which instruments are playing right now?


Who is singing? What language are they singing?


Is this chord major or minor?


Is this dancer moving quickly or slowly?


Is this music happy or sad?


Is anyone standing near the camera?


19

An example algorithm: kNN


The features of an example are treated as its
coordinates in n
-
dimensional space


To classify an new example, the algorithm
looks for its
k

(maybe 10) nearest neighbors in
that space, and chooses the most popular
class.

20

kNN space:
Basketball
or
Sumo
?

21

Feature 1: Weight

Feature 2: Height

kNN space:
Basketball
or
Sumo
?

22

Feature 1: Weight

Feature 2: Height

?

kNN space:
Basketball
or
Sumo
?

23

Feature 1: Weight

Feature 2: Height

?

K=3

kNN space:
Basketball
or
Sumo
?

24

Feature 1: Weight

Feature 2: Height

S

Another technique: Neural
networks


One of the first machine learning
methods


Model functions as complex
connections of interconnected nodes
(neurons)


Inspired by the human brain


An input causes a cascade of neuron
firings, resulting in an output value
(e.g., a class label, or a real number)


Results in highly non
-
linear functions
from input to output


Interesting for performance!

25

Combining Techniques

26

Processing:
Extract webcam
features

ChucK: Pass features
to Java, receive
results back and use
them to make sound

Java: Train a neural
network to map
features to sounds

OSC

OSC

Example: Samson recognizer

Review


Machine learning is often easier than the alternatives


Use standard algorithms (exiting libraries) to do hard work


Present examples of inputs (features) with their outputs (desired
labels)


Requires you to choose which features to use (different for audio,
video, sensor)


Appropriate for camera, audio, sensors, and many other types
of data


Live, interactive performance is a very interesting application
area


E.g., can re
-
train classifiers as you get new data


See my handout: Classification_handout.pdf

27