Objectives:

traderainSoftware and s/w Development

Aug 15, 2012 (4 years and 11 months ago)

324 views

Lab worksheet
5



Retrieval and Classification


Objectives




To better understand histograms



To use histogram as one of the visual features to represent image content



To develop image classification and retrieval methods


The Histogram Algorithm


To refresh

your memory, here are some highlights from the lecture notes on the
definition of a histogram and a general algorithm for histogram calculation.


For colour image (3 bands, for R, G, B respectively)



The basic idea is to quantize each of the RGB values int
o m intervals resulting
in a total number of m
3

(m=256) colour combinations (or bins). You can also
use 3 x 256 to calculate the RGB bins separately to improve the efficiency.



A colour histogram H(I) is then constructed. This colour histogram is a vector
{
h
1
, h
2
, …, h
m3
} where element h
x

represents the number of pixels in image I
falling within bin x.


The following is a sample algorithm to calculate a grey level histogram (band = 1).
For details on calculating colour histogram, please read the program disc
ussed below
from step 1.


Create an array histogram with 2
b

elements //b representing number of byte

for all grey levels, i, do

histogram[i] = 0

end for

for all pixel coordinates, x and y, do


Increment histogram [f(x,y)] by 1

end for



Overview of Deli
verables

For this lab sheet the deliverable application you produce must be capable or
performing two tasks.

Classification:

For the first deliverable you will be required to collect two sets of image data,
ideally each set should have certain similarity
in terms of their colour content. For
example, you can collect 5 images with oranges and 5 images with bananas (or 5
images with grass, 5 images with cloud and sky). Your application should be
capable of producing histogram data for each image (including t
he query image),
determining the average histogram data for each of the two classifications, and
printing to the command line which classification the query image belongs to.


Retrieval

For the second deliverable you should modify the class you have create
d above
(or create a new class which is capable of reading in 11 images as previously), but
instead of classifying the query image, your new class should retrieve and display
the image most similar from the entire set of images (i.e. the whole dataset.)


Lab Instructions


The first section of the lab is concerned with understanding what the code in
“Histogram.java” class does.
You do not have to run the Histogram.java file
. The
steps below highlight how each sentence in this algorithm above is implemented

in
Java.


Step1
In NetBeans, open “Histogram.java” which is located in the directory “javaim
-
Nick
\
Classes
\
Source
\
com
\
pearsoneduc
\
ip
\
op”.


The easiest way to do this is by selecting “Open File” from the “File” menu in
NetBeans, and navigating to the “His
togram.java” file.
There is no need to
incorporate this class into the source of your project like you may have done
with previous labs.


Step 2

from the “view” menu, choose “show line numbers”, so that you can see the
line numbers of the code, which are u
sed for reading reference for the rest of this
document. If the “show line numbers” option is greyed
-
out please click once in the
editor window (anywhere in the code) and then try to select the option again.


Step 3
. Please try to read through the whole cl
ass “Histogram” as much as you can.
The following section provides some help for you by explaining some important parts
in the class. Further explanation on the important lines are given as below:


In class “Histogram.java”


Line 44:

public final class His
togram implements Cloneable{….

The whole Histogram class is implemented very sophisticatedly, although the only
key and indispensable part is the class is the
accumulateFrequencies
( ) between line
584 and 608. The author has added a lots more functionality

into this class apart from
just calculating histogram in
accumulateFrequencies
( ).


From line 60 to line 112

defines the member variables in the Histogram class.


Line 60

private int bands;

This to be used for representing the number of bands in the sour
ce image. If band=1,
the source image to dealt with is grey level image (only deal with grey level intensity
range from 0
-
255); if band ==3, means image to be dealt with is colour image, with R
G B colour bands. Each band will have colour range from 0
-
255.



Line 66

private int samples;

This to store the total number of samples processed. Typically it will be the total
number of the pixels in the image, samples = image.getWidth()*image.getHeight();
However, if you wish to calculate the histogram for a smal
ler region of image, the
sample is the pixel number in that region.


Line 72

private int[][]freq;

2D array holding frequency data for a grey scale or colour image. This is where the
histogram information is held. If it is a grey image to be processed,

the band=1, then
freq[0][] will represent the histogram for grey image. If it is a colour image, the
band=3, then freq[0][] freq[1][] freq[2][] respectively will represent the histogram
(frequency) for bands R, G B.


From line 78
-
112
, the class also defin
es some other variables to represent statistical
data about the image, would be useful for advanced application of histogram. If you
like, you can use them in your coursework as extra criteria for possible improvement
of similarity measure (you are encoura
ged to do so if you have more time if you wish
to provide more analysis in your report, but it is beyond the minimum requirement).


Line 219

public void computeHistogram(BufferedImage image)

This function is to call the calculation function (
accumulateFreq
uenccies
() ) to
compute histogram.


Line 224,

if (image.getType()==BufferedImage.TYPE_BYTE_GREY)

is to judge whether the
input image is a grey level image or a colour image. If it is a grey image, then assign
bands ==1, otherwise it is a colour image, th
en bands = 3.

NB.
BufferedImage is a built
-
in class in Java to represent an image. In this module we
mostly will use this class for representing an image.


Line
public void write (Writer destination){

This function is to write histogram data to the specifi
ed destination. You can use this
function to write the histogram data to a file. Please note in this function, for grey and
colour images, the formats of writing the histogram is slightly different. You will see
this in step 4.


Line 338 to line 540 includ
e

methods to return some useful statistical data generated
from histogram.


Line 371
-
390
, define methods:

public int getFrequency(int value)
, and
public int getFrequency(int band, int value)

These two functions are
very important

as they are the methods

you may want to use
for the coursework to obtain histogram value of the analysed image. The reason for
using these methods is because they are public methods, and can be accessed by any
class you create. The variable associated with this method (
freq[band
][value])
is
declared as a private member and cannot be accessed outside the Histogram class.



Line 549 and line 564

contains code for allocating storage and initialisation, you may
find in here a bit more information on how the data structure has been d
esigned for
the class Histogram.


Line 584

private void accumulateFrequencies (BufferedImage image)

This is the key function that actually calculates the histogram. The histogram is stored
in private variable freq[ ][ ].


Line 592 to 596

calculate the gre
y level histogram. You can see 2D array freq here is
only used as 1D array as freq[0] [], where freq[0][i] represents the pixel frequency in
the image whose grey intensity is i.


Line 597 to 606

calculate the colour histogram, where freq[0][], freq[1][], f
req[2][]
respectively represent frequency of each band (RGB).


Line 615
-

660

private void computeStatistics()

This is the method to calculate other useful statistical information related to
histogram. It will help your understanding on histogram if you lo
ok through this code


This concludes the tour of the Histogram class. Please try and understand what has
been described above
.


Step 4
Now we will u
se a main program to call the functionality in the histogram
class. To do this we
are going to examine anot
her class


the “CalcHist.java” class
.
This class can be found in the “javaim
-
Nick
\
Apps
\
Chap06” directory and you should
include this directory into your project using the same method as you have in the
previous labs.


IMPORTANT NOTE

At this step you shoul
d make sure your project has both the “iplib.jar” and
“ipapps.jar” included in the libraries section of project properties.






The following section highlights the key functionality in the “CalcHist.java” class


Line 40

ImageDecoder input = ImageFil
e.createImageDecoder(argv[0]);

Line 41

BufferedImage image = input.decodeAsBufferedImage();

These two lines are to take input from the main(), the first argument will be the image
file (argv[0]) that you would like to process. The two lines convert the inp
ut image
file into an image object (
image),
an instance of class BufferedImage.



Line 42


Histogram histogram = new Histogram(image);

This passes the
image
variable
as a parameter to the Histogram( ) constructor, which
will construct all the

statistical data related to histogram, in particular, freq[][].


Line 43


FileWriter histFile = new FileWriter(argv[1]);

Line 44

histogram.write(histFile);

The second argument from the command line or the NetBeans arguments specifies a
text

file where the histogram data will be written to. Histogram.write() enable writing
information in freq[][] to a text file. (This is how you get the text file for histogram in
your first assignment. In this final assignment, we do not have to write the hi
stogram
to files so we do not need this.)



Both libraries
included

Line 45
-
47

performs processing that produces the cumulative histogram which is
another important statistical data about image. It is useful when calculate the
histogram equalisation. (This is not needed for this
assignment.)


Step 5
Set the appropriate arguments to run the CalcHist class. Provide the input the
image you would like to process, and indicate a filename for storing the output result.
For example, you can set the NetBeans arguments to:


matthew1.jpg hi
sto
-
out.txt


This will process a colour image (matthew1.jpg) and save the result in histo
-
out.txt.
You can later open histo
-
out.txt to check the result. For grey level image, you can
input a grey image such as “mattgrey.jpg” to see the difference. (We have

done this in
the first assignment.)


Step 6
The above process is to help you identify the flow of data when calculating
and using the histogram functionality provided in the Histogram.java class. For your
coursework, you can create a new class, for exampl
e called “Classification.java” and
copy the code from “CalcHist.java” to use as a template.


Coursework Overview:

Image classification and retrieval based on Histogram


A key issue is
what representation or encoding of the object is used in the recognition

process?

Alternatively, what features are important for recognition? We often talk
about colour, shape, and texture information being important visual cues for
recognition. Here we focus on using colour information to represent image content.


Feature vec
tor representation:

Objects may be compared for similarity based on their
representation as a vector of measurements. Suppose each object is represented by
exactly
d
measurements. The
i
th

coordinate of such a feature vector has the same
meaning for each ob
ject; In our case, we can use a colour histogram to form such
vector. (We recommend using 3x256 bins instead of 256
3
because processing a large
number of bins can be processor intensive You can use some other design, e.g., partial
accumulative histogram, f
irst 10 bins as one vector element, second 10 bins as the
second vector element and so on).


The similarity, or closeness, between the feature vector representations of two objects
can then be described using Euclidean distance between the vectors defined
in
Equation 1. Sometimes the Euclidean distance between an observed vector and a
stored class prototype can provide a useful classification function.


Definition:
the
Euclidean distance

between two
d
-
dimensional feature vectors h1
and h2 is


||h
1
-
h
2
|| =





d
i
i
h
i
h
,
1
2
]
[
]
[
2
1

Equation 1


h
1

and h
2

can be seen as two colour histograms, where h
i
[i] represents the frequency
value at bin i.


Classification


Such a method can be used for classification. Assume that w
e have 10 images
belonging to class “grass images”, another 10 images belonging to “cloud images”.
We can use the same method of calculating the centroid in the segmentation task
(which was a two dimensional problem), to calculate the centre mean of the cl
ass:


h
mean
[i] =
d
i
h
d

0
]
[


d is the number of samples, in this case, 10. h[i] is the histogram for image i.

















Figure 1 two compact classes: classification using nearest mean


After having obtained the centre mean, we

can then use above distance calculation
method (
Euclidean or Absolute distance that is mentioned in the lecture note on
Retrieval based on Colour Histogram, also see below)
to calculate the distance
between the unknown image and the two centre means, the
closer value means that it
should be more possible for this unknown image belong to that class.

x x x x x


x x x x

x x x x x


o o o o


o. o

ooooo o


o o


o class mean

x class mean



Retrieval
:


A similar method can be used for retrieval. If h
q

means a colour histogram for query
image
q
, the distance between this query image and every ima
ge
e

in the database

h
e


||h
q
-
h
e
||

=





d
i
i
h
i
h
e
q
,
1
2
]
[
]
[

Equation 2


the one in the database with the shortest distance to the query image should be
retrieved.


(we have practised
Euclidean distance
in 2
-
dim
ensional problem when we tried to
calculated the distance between two centroids, ||D|| =
2
2
1
2
2
1
)
(
)
(
y
y
x
x



)



Alternative method for colour histogram based retrieval: Absolute distance

To retrieve image from the database, the user supplies either a s
ample image or a
specification for the system to construct a colour histrogram h(Q).

A distance metric is used to measure the similarity between h(Q) and h(I). Where I
represents each of the images in the database. And example distance metric is shown
as f
ollows:


D(Q, I ) =




tbin
x
x
z
z
i
q
0
|
|


Where
q
x

and
i
x

are the numbers of pixels in the image Q and I, respectively, falling
within bin x.
tbin

can be 3x256 bins or of 256
3



Coursework Deliverable

This section contains step
-
by
-
step instructions
on how to approach the coursework.

Classification:

1.

Collect two sets of image data, ideally each set should have certain
similarity in terms of their colour content. For example, you can collect 5
images with oranges and 5 images with bananas (or 5 images w
ith grass, 5
images with cloud and sky).

2.

You should create a new class (possibly copying the code in CalcHist.java
or the sample code given below to use as a reference) which is capable of
reading in these images (where the filenames are specified as argum
ents in
NetBeans) and storing them in an array of type BufferedImage

3.

Your class should also be capable of reading in another image (specified
using the NetBeans arguments) which will be a query image (In total you
should be reading 11 image filenames from
the arguments argv[] array).

4.

Your class should be able to produce histogram data for each image
(including the query image). Your class should also be capable of printing
to the command line which classification the query image belongs to, for
example oran
ge or banana / grass or sky.

5.

You will need to calculate the mean of the histograms for each of the
sample images for each class and then measure the distance between the
histogram of the query image and each classification’s mean. The shorter
the distance
, the more similar the input image to the class.


Retrieval

1.

You should modify the class you have created above or create a new class
which is capable of reading in 11 images as previously described (1 query
image and 10 other images), but instead of class
ifying the image your new
class should retrieve a similar image from all the images (i.e. the whole
dataset.) In this case you can have a mixture of various kinds of images in
your dataset.

2.

The similarity measure will be the calculation of the distance be
tween the
query image and every single image in the dataset based on their
histograms. The closer the distance, the more similar. This is an example
of Content
-
based Image Retrieval.



Hints and Example Code


I have provided an example outline of the code
you will be required to produce for the
first deliverable.
Please note that if you cut and paste this code into NetBeans it
will NOT work without modification!

You will need to add extra code (as specified
in the comments) to provide all of the required fu
nctionality.


When this code is complete you should specify appropriate arguments in NetBeans:


e.g. g01.jpg s1.jpg s2.jpg s3.jpg s4.jpg s5.jpg g1.jpg g2.jpg g3.jpg g4.jpg g5.jpg



You can also adapt this code for the Retrieval deliverable.



import java.a
wt.image.BufferedImage;

import java.io.FileWriter;

import com.pearsoneduc.ip.io.*;

import com.pearsoneduc.ip.op.Histogram;

import java.io.*;

import java.awt.image.*;

import java.awt.color.ColorSpace;

import javax.swing.*;

import com.pearsoneduc.ip.io.*;

im
port com.pearsoneduc.ip.gui.*;


public class ClassificationAndRetrieval extends JFrame {




private ImageView[] views; // image display components



//constructor which creates a window and displays the 11 images
//specified as filename
arguments


public ClassificationAndRetrieval(BufferedImage[] imageSequence)
throws IOException, ImageDecoderException {




super("Classification and Retrieval");




views = new ImageView[11];




for (int i=0;i<1
1;i++) {


views[i] = new ImageView(imageSequence[i]);


}




JTabbedPane tabbedPane = new JTabbedPane();


tabbedPane.add(new JScrollPane(views[0]),"Query Image");




for (int i=1;i<11;i++) {


t
abbedPane.add(new JScrollPane(views[i]),"Sample"+i);


}




getContentPane().add(tabbedPane);


addWindowListener(new WindowMonitor());


}




public static void main(String[] argv) {


if (argv.length > 1) {



//c
reate an array of BufferedImages and Histograms here



Histogram[] histogram = new Histogram[11];


BufferedImage[] imageSequence= new BufferedImage[11];






try {


for(in
t i=0;i<imageSequence.length;i++) {


ImageDecoder input =
ImageFile.createImageDecoder(argv[i]);


BufferedImage image =
input.decodeAsBufferedImage();


//create histogram instance for each input imag
e


histogram[i]=new Histogram(image);

//add the image to the BufferedImage array


imageSequence[i] = image;


}


} catch (Exception ex) {


ex.printStackTrace();


}


//here we create 4
integer arrays that will contain our histogram

//data for each classification type. Notice we are using 256*3 bins

//instead of pow(256,3) to limit the amount of computation




int hisSum1[] = new int[256*3];


int hisSum2
[] = new int[256*3];


int average1[] = new int [256*3];


int average2[] = new int [256*3];


//try and understand how the two
-
dimensional histogram data for each
//image is extracted into the one
-
dimensional arrays specified above





for (int b=0;b<3;b++) {


for(int j=0;j<256;j++){


for (int i=1;i<11;i++){


int n=256*b;


if(i<6) {


hisSum1[j+n]+=histogram[i].g
etFrequency(b,j);


} else {


hisSum2[j+n]+=histogram[i].getFrequency(b,j);


}


}


}


}


//here we calculate the histogram average for each

of the two

//classifications and store the result in the

//appropriate place in the average1 or average2 array




for (int i=0;i<256*3;i++) {


average1[i]=Math.round(hisSum1[i]/5);


average2[i]=Math.ro
und(hisSum2[i]/5);


}




int queryHistogram[] = new int[256*3];




for (int b=0;b<3;b++) {


for(int j=0;j<256;j++){


int n=256*b;


queryHistogram[j+
n]=histogram[0].getFrequency(b,
j);


}


}



//here you should implement the calculations that determine which

//classification your query image belongs to


//the following code creates the window and displays it to the

user




JFrame frame = null;




try {


frame = new
ClassificationAndRetrieval(imageSequence);


} catch (ImageDecoderException ex) {


System.out.println("Problem processin
g images");


ex.printStackTrace();


System.exit(0);


} catch (IOException ex) {


System.out.println("Problem loading images");


ex.printStackTrace();


System.exit(0);



}




frame.setSize(900,700);


frame.setVisible(true);




} else {


System.err.println(


"usage: java ClassificationAndRetrieval <Query
image> <sample1> <sample2> <sam
ple3> ........");


System.exit(1);


}


}

}