Neural Networks and Clustering

chickenchairwomanAI and Robotics

Oct 19, 2013 (3 years and 9 months ago)

99 views

Neural Networks and Clustering

Professor Amit Basu

abasu@smu.edu

Neural Networks



Based upon models of human reasoning


Low level pattern recognition Artificial Neural
Networks (ANNs)


Associations in memory


Generated by exposure and progressively refined


Simulated Neural Networks


based on connectionist models of computing


bottom
-
up


adaptive



How does it work?



Give the NN a pattern, and tell it what it means


It creates a model that matches the inputs and
outputs


Give it another pattern, and tell it what it means


The NN tests its model against the pattern, and
adjusts it if needed
(learning)


Keep giving it more patterns, and see if it can
converge to a stable model


Learning Methods

Supervised (Reinforcement) Learning




requires a training set having both inputs
and outputs




partition
-

also testing set




back propagation is common rule

Unsupervised Learning




also requires a training set, but outputs are
not known




knowledge discovery




clustering


Back Propagation Learning


1.
Measure the RMS error in the output layer

2.
Modify the weights of all edges to this
layer

-

l

is the learning rate

3.
Propagate the learning backward

4.
Once the network has converged (settled
down), it yields a relatively simple
transformation


Unsupervised Learning


Used in Self
-
Organizing Neural Networks


There are no “output” parameters, only
tuples with attributes X = {x
1
, x
2
, … x
n
}


As different tuples are presented to the NN,
it organizes into a form that identifies
specific clusters

NN Features


Better used for low
-
level reasoning, where
deduction is infeasible/unimportant


Applicable to complex, nonlinear models
that do not have simple functional
representations


Adaptive


as new data indicates shift, the
network can adapt


Suitable for evolving domains


Major Design Approaches


1.
Vary Architecture



Start with simplest
network, try to modify network design, weight
symmetries, etc. based on domain knowledge
and training outcomes

2.
Vary Training
-

fix network, iterate between
training set, test set and validation set


“Overfitting”


The goal of the network is to learn the domain
and “optimize relationships/models”, not identify
optimal output



Applications of Neural Networks


Prediction


Uses input values to predict some output. e.g. pick the best stocks,
predict weather, identify people with cancer risk.



Classification and Pattern Recognition


Use inputs to determine the classification. e.g. is the input the letter A,
is the blob of the video data a plane and what kind of plane is it.



Data Conceptualization


Analyze the inputs so that grouping relationships can be inferred. e.g.
extract from a database rules for those most likely to buy a particular
product.



Data Filtering


Smooth an input signal. e.g. take the noise out of a telephone signal.



Clustering


Similar to classification, in the sense that
like items are clustered


However, the categories are not pre
-
defined


The clustering is often multidimensional,
rather than on a specific “output” attribute


The best (or target) number of clusters may
not be predetermined


Clusters may change over time

Two Views of Clustering


A cluster is a set of similar elements


i.e., elements of different clusters are NOT
alike


The distance between elements in a cluster
is less than the distance between elements in
different clusters


i.e., the element nearest to a given element is in
the same cluster

Cluster Shapes


Spherical Clusters


When clustering is based on elements being close to all
others in cluster


Min (max distance in cluster)


Useful for segmentation


Elongated Clusters


When clusters based on proximity to
some

element in
cluster


Max (min distance in cluster)


Useful in evolutionary patterns

Approaches to Clustering


Agglomerative


Start with a set of elements,
and progressively create
clusters


Initial clusters are very small,
and you only add to clusters


Partitioning


Start with large cluster, and
partition into smaller ones


Hierarchical Clustering


Start with isolated elements


Cluster with closest
neighbors


Progressively increase radius
of cluster

K
-
Means Clustering


Popular clustering method


K denotes the number of clusters


Start with k seeds (“centers”)


Cluster all elements with the seed closest to them


Re
-
compute the centers of each cluster


Re
-
cluster each element using the new centers


Continue until convergence or other terminating
condition

Example


{2, 3, 4, 10, 11, 12, 20, 25, 30}


Assume that K=2, and seeds are 2 and 5

Step

R1

R2

Cluster1

Cluster2

1

2

5

{2,3}

{4, 10, 11, 12, 20, 25, 30}

2

2.5

16

{2, 3, 4}

{10, 11, 12, 20, 25, 30}

3

3

18

{2, 3, 4, 10}

{11, 12, 20, 25, 30}

4

4.75

19.6

{2, 3, 4, 10, 11, 12}

{20, 25, 30}

5

7

25

{2, 3, 4, 10, 11, 12}

{20, 25, 30}

Example

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

Example

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

X

X

X

Example

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

X

X

X

Example

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

X

X

X

Example

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

X

X

X

Example

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

X

X

X

Using Clustering


What does each cluster mean?


Add cluster as an outcome variable


Build a decision tree or other model


Classification hierarchies


Clusters for association


Collaborative filtering


Characterize cluster by summary statistics

Market segmentation

Evaluating Clustering


How do you know whether a clustering
analysis is successful?


No test set available


Measure “distinctness” of clusters


What do you do with the cluster?


Challenge in collaborative filtering


Consequence of bottom
-
up approach