Automated Categorization of Drosophila Learning and Memory Behaviors

mewlingtincupΛογισμικό & κατασκευή λογ/κού

9 Νοε 2013 (πριν από 3 χρόνια και 5 μήνες)

120 εμφανίσεις

Automated Categorization of Drosophila Learning and Memory Behaviors
using Video Analysis
A Thesis
Submitted to the Faculty
of
Drexel University
by
Md.Alimoor Reza
in partial fulllment of the
requirements for the degree
of
MSc in Computer Science
August 2011
c Copyright August 2011
Md.Alimoor Reza.
ii
Acknowledgements
I would like to sincerely express my gratitude to Allah (SWT) for all the successes along the way
of conducting the thesis.My thankful acknowledgement goes to my advisor,Prof.David E.Breen,
for his constant help and suggestion on the research.I would also like to thank Prof.Ko Nishino,
Prof.Daniel Marenda,and Prof.Aleister Saunders for their valuable guidance and advice.I would
like to acknowledge the contribution of Je Marker who wrote the segmentation module in the
Objective C programming language.Special thanks goes to Manolya Eyiyurekli,Mitch D'Rozario,
and Sheetal for proof reading the thesis.Finally,I owe to my family members:Sheetal,Amma,and
Saz,for their continuous support and encouragement.
iii
Dedications
To my beloved father late Md.Eklimur Reza,my greatest inspiration,whomI lost so unexpected
on January 2011.
iv
Table of Contents
List of Figures..........................................................................................vii
List of Tables...........................................................................................x
Abstract.................................................................................................xiv
1.Introduction.........................................................................................1
1.1 Introduction...................................................................................1
2.Previous Work......................................................................................3
2.1 Behavior Analysis............................................................................3
2.2 Tracking.......................................................................................4
2.3 Contributions.................................................................................6
3.Biological Background.............................................................................8
3.1 Courtship Behavior...........................................................................8
3.1.1 Visual Cues...........................................................................10
3.1.2 Olfactory Cues........................................................................10
3.1.3 Auditory Signals......................................................................10
3.2 Courtship Index..............................................................................11
3.3 Conditioned Courtship Suppression.........................................................11
3.4 Learning Response............................................................................13
3.5 Memory Response............................................................................14
4.Methods.............................................................................................16
4.1 Extract Images from the Video..............................................................18
4.2 Crop Individual Chambers...................................................................18
4.3 Segmentation..................................................................................18
4.3.1 Detect Background...................................................................18
4.3.2 Image Subtraction and Thresholding...............................................19
4.4 Filtering.......................................................................................19
4.4.1 Fill Voids..............................................................................20
4.4.2 Identify the Flies.....................................................................20
4.5 Identify Male/Female Flies..................................................................21
4.5.1 Minimum Distance Criterion........................................................22
4.6 Determine Head Direction of Each Fly.....................................................26
v
4.6.1 Identify the Longest Positive Dot Product Subsequence..........................27
4.6.2 Find the Head Direction of the Start Frame.......................................29
4.6.3 Propagate the Head Direction Vectors..............................................29
4.7 Compute Geometric Measures...............................................................32
4.7.1 Percentage of Frames when One Fly is Looking at the Other....................32
4.7.2 Percentage of Frames Flies are Together...........................................34
4.7.3 Distribution of the Centroid Distance between the Flies..........................36
4.7.4 Distribution of the Head Direction Angle between the Flies......................36
4.7.5 Distribution of Speeds of the Flies:.................................................36
4.8 Create Feature Vector........................................................................36
4.9 Clustering.....................................................................................38
4.9.1 K-means Clustering..................................................................38
4.9.2 Data Preprocessing for K-means Clustering........................................41
4.9.3 Subtraction Step.....................................................................42
4.10 Generate Computed Courtship Index.......................................................43
5.Results..............................................................................................46
5.1 Extract Images from the Video..............................................................46
5.2 Crop Individual Chambers...................................................................46
5.3 Segmentation..................................................................................48
5.4 Filtering.......................................................................................48
5.5 Identify Male/Female Flies..................................................................48
5.6 Determine Head Direction of Each Fly.....................................................52
5.7 Compute Geometric Measures...............................................................52
5.8 Create Feature Vectors.......................................................................52
5.9 Generate the Computed Courtship Index..................................................56
5.9.1 Comparison of CCI and CI for Strong Learning and Strong Memory............60
5.9.2 Comparison of CCI and CI for Moderate Learning and Defective Memory.....63
5.10 Clustering.....................................................................................64
5.10.1 2-means Clustering of Memory Response Behaviors on a Subset of Seventeen
Samples...............................................................................64
5.10.2 2-means Clustering of Learning Response Behaviors on a Subset of Fourteen
Samples...............................................................................69
vi
5.10.3 2-means Clustering of Learning Response Behaviors on a Set of 22 Samples....72
5.10.4 3-means Clustering of Learning Response Behaviors on a Set of 22 Samples....76
5.11 Implementation Details......................................................................79
5.12 Computation Times..........................................................................82
5.13 Problem Encountered........................................................................82
5.13.1 Non-robust Male and Female Identication........................................83
5.13.2 Head Direction Calculation Failure.................................................83
5.13.3 Incorrect Geometric Measurement Due to Male and Female Identication Failure 83
5.13.4 Segmentation Failure in Computing Background..................................84
6.Conclusion..........................................................................................85
7.Future Work........................................................................................86
7.1 Optimization in Implementation............................................................86
7.2 Automated Cropping.........................................................................86
7.3 Modication in Segmentation Algorithm...................................................86
7.4 Multivariate Non-Linear Regression Analysis for the Computed Courtship Index......87
Bibliography............................................................................................88
vii
List of Figures
3.1 Courtship actions (taken from [16]).............................................................9
3.2 Average CI values for a successful learning response..........................................13
3.3 Average CI values for a non-defective memory response......................................15
4.1 The process pipeline..............................................................................17
4.2 Computing the four distances between the white regions in image Y and the white regions
in image X.........................................................................................24
4.3 Determining the minimum of these four distances between the white regions in image Y
and the white regions in image X...............................................................24
4.4 Assigning labels to white regions in image Y from the labels in image X using the mini-
mum distance criterion...........................................................................24
4.5 A sample frame J with the chosen y high-lighted by the blue rectangle....................28
4.6 Computation of Oriented Central Axis OCA
i
J
for a y from its velocity vector V
i
J
and
eigen vector E
i
J
...................................................................................28
4.7 Computation of Oriented Central Axis OCA
i
J
for a y from its velocity vector V
i
J
and
eigen vector E
i
J
...................................................................................28
4.8 A subsequence starting from frame M and ending in frame N.The blue circles denote a
y series in this subsequence.....................................................................30
4.9 The subsequence with OCA
i
j
denoted by the yellow arrows and V
i
j
.........................30
4.10 The longest positive dot product subsequence within the subsequence starting from frame
Mto frame N.The identied longest series starts at frame Q and ends at frame R within
the subsequence...................................................................................30
4.11 Start frame S with head direction denoted by a green arrow.................................31
4.12 Head direction propagation from start frame S in the forward direction.....................31
4.13 A sample frame with a male y (white region on the left) and a female y (white region
on the right).......................................................................................35
4.14 Intersection test using the male y as source object and female y as the destination
object.Finding the start point (left image),which is denoted by a red point.Traversing
along the head-direction line to nd the intersection with the destination y object.......35
4.15 Intersection test using the female as source object and male y as the destination object.
Finding the start point (left image) which is denoted by a red point.Traversing along the
head-direction line in an attempt to nd the intersection with the destination y object..35
4.16 A sample frame where the ies are together....................................................37
viii
4.17 Inter-centroid distance between the ies in a frame............................................37
4.18 Angles between the head directions of the ies in a frame....................................37
4.19 K-means clustering iterations (taken from\Pattern recognition and machine learning"[4]).40
4.20 Sample frame from a video with higher resolution.............................................42
4.21 Sample frame from a video with lower resolution..............................................42
5.1 A sequence of extracted images from the rst 10 minutes video with a set of ve samples.47
5.2 An extracted image from the rst 10 minutes video with a set of ve samples.............47
5.3 Cropped images from Figure 5.2.Samples are denoted from left to right by Set4,Set5,
Set6,Set7,and Set8 respectively using the described numbering convention................47
5.4 The crop stage....................................................................................49
5.5 The detect background step in the segmentation stage........................................49
5.6 The subtract and thresholding step in segmentation stage....................................50
5.7 The ltering stage................................................................................50
5.8 Frame 1............................................................................................51
5.9 Frame 18038.......................................................................................51
5.10 Frame 1 segmented...............................................................................51
5.11 Frame 18038 segmented..........................................................................51
5.12 Frame 1 ltered...................................................................................51
5.13 Frame 18038 ltered..............................................................................51
5.14 The m-th frame is the frame where the two ies are furthest apart in the subsequence
between the s-th and the t-th frame............................................................53
5.15 Frames with head directions drawn as green lines.The yellow circle indicates the male
y.The other y is the female y...............................................................53
5.16 The distribution of the inter-centroid distance between the y pair for the rst 10 minutes
video clip of set 4.................................................................................54
5.17 The distribution of the head direction angle between the y pair for the rst 10 minutes
video clip of set 4.................................................................................54
5.18 The distribution of the speeds of the y pair for the rst 10 minutes video clip of set 4...54
5.19 Comparison of average CI and average CCI for ies with a strong learning response.......61
ix
5.20 Comparison of average CI and average CCI for ies with a strong memory response.......61
5.21 Comparison of average CI and average CCI for the ies with a moderate learning response.64
5.22 Comparison of average CI and average CCI for the ies with a defective memory response.64
5.23 Average CI and average CCI for one cluster (comprised of 7 samples) that exhibits a
strong memory response..........................................................................67
5.24 Average CI and average CCI for the other cluster (comprised of 10 samples) that exhibits
a defective memory response.....................................................................67
5.25 Average CI and average CCI for the cluster (comprised of four samples) containing ies
with a strong learning responses.................................................................70
5.26 Average CI and average CCI for the other cluster (comprised of ten samples) containing
ies with a defective learning response..........................................................70
5.27 Average CI and average CCI for the cluster (comprised of seven samples) containing ies
with a strong learning response..................................................................76
5.28 Average CI and average CCI for the other cluster (comprised of fteen samples) contain-
ing ies with a weak learning response..........................................................76
5.29 Average CI and average CCI for rst cluster in 3-means clustering (comprised of ve
samples) containing ies with a strong learning response.....................................77
5.30 Average CI and average CCI for rst cluster (comprising 5 samples) on the memory
response...........................................................................................77
5.31 Average CI and average CCI for the second cluster (comprised of six samples) containing
ies with a moderate learning response.........................................................77
5.32 Average CI and average CCI for second cluster (comprising 6 samples) on the memory
response...........................................................................................77
5.33 Average CI and average CCI for the third cluster (comprised of eleven samples) containing
ies with a weak learning response..............................................................77
5.34 Average CI and average CCI for third cluster (comprised of 11 samples) on the memory
response...........................................................................................77
x
List of Tables
4.1 Variables in the CCI equation...................................................................45
5.1 The feature vectors for the rst 10 minute samples of the normal-trained ies.............55
5.2 The feature vectors for the last10 minute samples of the normal-trained ies...............55
5.3 The feature vectors for the immediate recall 10 minute samples of normal-trained ies....56
5.4 The feature vectors for the immediate recall 10 minute samples of sham-trained ies......57
5.5 The least squares solution vectors from the ten iterations of the cross validation...........57
5.6 Computed CIs (CCIs) for the rst 10 minutes samples of the normal-trained ies.........58
5.7 Computed CIs (CCIs) for the last 10 minutes samples of the normal-trained ies..........59
5.8 Computed CIs (CCIs) for the immediate recall 10 minutes samples of the normal-trained
ies.................................................................................................59
5.9 Computed CIs (CCIs) for the immediate recall 10 minutes samples of the sham-trained
ies.................................................................................................59
5.10 Selected seven samples with strong learning and strong memory responses and their CIs..60
5.11 Selected ten sham-trained samples and their CIs for the immediate recall 10 minute
responses..........................................................................................61
5.12 Selected seven samples with strong learning and strong memory responses and their CCIs.62
5.13 Selected ten sham-trained samples and their CCIs for the immediate recall 10 minute
responses..........................................................................................62
5.14 Selected ten samples with moderate learning and defective memory and their CIs.........63
5.15 Selected ten samples with moderate learning and defective memory and their CCIs.......64
5.16 The feature vectors produced by the dierence of the immediate recall 10 minute feature
vectors and the average feature vector of the immediate recall 10 minute response of the
sixteen sham-trained ies........................................................................65
5.17 First 10 minute and last 10 minute dierence feature vectors................................65
5.18 2-means clustering on 17 samples to group ies by their memory responses.................66
5.19 First group (7 samples with normal memory reponse) produced by 2-means clustering
on a subset (17) of samples based on the memory response feature vectors,with their CI
values..............................................................................................67
xi
5.20 First group (7 samples with normal memory reponse) produced by 2-means clustering on
a subset (17) of samples based on the memory response feature vectors,with their CCI
values..............................................................................................68
5.21 Second group (10 samples with defective memory response) produced by 2-means clus-
tering on a subset (17) of samples based on the memory response feature vectors,with
their CI values....................................................................................68
5.22 Second group (10 samples with defective memory response) produced by 2-means clus-
tering on a subset (17) of samples based on the memory response feature vectors,with
their CCI values..................................................................................68
5.23 A subset of 14 samples that show strong or weak learning from manual observation.......70
5.24 Members of each group produced by 2-means clustering based on the learning response
feature vectors....................................................................................71
5.25 First group (4 samples with strong learning response) produced by 2-means clustering on
a subset (14) of samples based on learning response feature vectors,with their CI values.71
5.26 First group (4 samples with strong learning response) produced by 2-means clustering on
a subset (14) of samples based on learning response feature vectors,with their CCI values.72
5.27 Second group (10 samples with no learning response) produced by 2-means clustering on
a subset (14) of samples based on learning response feature vectors,with their CI values.72
5.28 Second group (10 samples with no learning response) produced by 2-means clustering on
a subset (14) of samples based on learning response feature vectors,with their CCI values.73
5.29 Members of each group produced by 2-means clustering on all 22 samples based on
learning response feature vectors................................................................73
5.30 First group (7 samples with strong learning response) produced by 2-means clustering
on all the 22 samples based on learning response feature vectors,with their CI values.....74
5.31 First group (7 samples with strong learning response) produced by 2-means clustering
on all the 22 samples based on learning response feature vectors,with their CCI values...74
5.32 Second group (15 samples with no learning response) produced by 2-means clustering on
all the 22 samples based on learning response feature vectors,with their CI values........75
5.33 Second group (15 samples with no learning response) produced by 2-means clustering on
all the 22 samples based on learning response feature vectors,with their CCI values......75
5.34 Members of each group produced by 3-means clustering on all 22 samples based on
learning response feature vectors................................................................76
xii
5.35 First group (5 samples with strong learning response) produced by 3-means clustering
based on learning response feature vectors,with their CI values.............................79
5.36 First group (5 samples with strong learning response) produced by 3-means clustering
based on learning response feature vectors,with their CCI values...........................79
5.37 Second group (6 samples with moderate learning response) produced by 3-means clus-
tering based on learning response feature vectors,with their CI values......................80
5.38 Second group (6 samples with moderate learning response) produced by 3-means clus-
tering based on learning response feature vectors,with their CCI values....................80
5.39 Third group (11 samples with weak learning response) produced by 3-means clustering
based on learning response feature vectors,with their CI values.............................80
5.40 Third group (11 samples with weak learning response) produced by 3-means clustering
based on learning response feature vectors,with their CCI values...........................81
5.41 A typical average processing times for a sample video.........................................82
xiii
There is much to learn from a pair of walking legs.The foot that is forward has no pride and other
foot has no shame.They both know their situation will change.| Unknown
xiv
Abstract
Automated Categorization of Drosophila Learning and Memory Behaviors using Video Analysis
Md.Alimoor Reza
Advisor:David E.Breen,PhD
The ability to study learning and memory behavior in living organisms has signicantly increased
our understanding of what genes aect this behavior,allowing for the rational design of therapeutics
in diseases that aect cognition.The fruit y,Drosophila melanogaster,is a well established model
organism used to study the mechanisms of both learning and memory in vivo.The techniques used
to assess this behavior in ies,while powerful,suer from a lack of speed and quantication.The
technical goal of this project is to create an automated method for characterizing this behavior in fruit
ies by analyzing video of their movements.A method is developed to replace and improve a labor-
intensive,subjective evaluation process with one that is automated,consistent and reproducible;
thus allowing for robust,high-throughput analysis of large quantities of video data.
The method includes identifying individual ies in a video,quantifying their size (which is corre-
lated with their gender),and tracking their motion.Once the ies are identied and tracked,various
geometric measures may be computed,for example distance between ies,their relative orientation,
velocities and percentage of time the ies are in contact with each other.This data is computed
for numerous experimental videos and produces high-dimensional feature vectors that quantify the
behavior of the ies.Clustering techniques,e.g.,k-means clustering,may then be applied to the fea-
ture vectors in order to computationally group each specimen by genotype.Our results show that
we are able to automatically dierentiate between normal and defective ies.We also generated
a Computed Courtship Index (CCI),a computational equivalent of the existing Courtship Index
(CI),and compared CCI with CI.These results demonstrate that our automated analysis provides
a numerical scoring of y behavior that is similar to the scoring produced by human observers.
1
1.Introduction
1.1 Introduction
The fruit y,Drosophila melanogaster,is a well-established model organism used to study the
mechanisms of both learning and memory.The ability to study learning and memory behavior in
Drosophila has signicantly increased our understanding of the underlying molecular mechanism
of this behavior.Understanding what aects this behavior allows for the rational design of ther-
apeutics in diseases that aect cognition.The existing techniques used to assess this behavior in
ies is powerful.The techniques,however,are time-consuming,laborious,and subjective to the
evaluator.An automated method based on video analysis of y movements will replace and improve
a labor-intensive,subjective evaluation process.Automating the process will provide a consistent
and reproducible analysis of fruit y courtship behavior.Additionally,it also enables quantication
of many aspects of the behaviors that are not easily perceived by a human observer.Moreover,the
automation promises the possibility of robust,high-throughput analysis of large quantities of video
data.
A variety of molecular tools and behavioral assays are available for Drosophila melanogaster
learning and memory research [32].In addition to the extensive genetic tools available for the
analysis of y neural circuitry associated with learning and memory [5] [26] [44] [45],both adult
and larval ies exhibit a number of behaviors that can be altered with training [32].One of these
behaviors that is well established in the eld is courtship conditioning.
Courting behavior by males in Drosophila follows a linear,stereotyped,and well documented set
of behaviors [2],and this behavior is modied by previous sexual experience [34] [18].Courtship
conditioning is a formof associative learning in Drosophila,where male courtship behavior is modied
by exposure to a previously mated female that is unreceptive to courting [34] [35].Thus,after 1
hour of courting a mated female,males suppress their courtship behavior even towards subsequent
receptive virgin females for 1-3 hours [34] [24].This courtship suppression is measured by the
Courtship Index (CI),which is calculated by dividing the total amount of time each male y spends
courting by the total duration of a testing period [34] [35].A decrease in CI in ies after such
training is indicative of behavioral plasticity in the form of memory [29].
Courtship conditioning is a powerful tool for learning and memory research.The manual com-
2
putation of courtship data is time-consuming.Courtship behavior is video-taped and the courtship
actions (some courtship-specic y behavior,e.g., y orientation,wing extension,wing vibration)
are manually quantied.Automating the process can signicantly reduce the amount of time a
human observer spends quantifying these behaviors.The quantication process is subjective to the
human observer too.A sample video clip can have multiple CI scores depending on the varying
assessments of multiple scorers.Besides this subjective evaluation,many aspects of quantication
are infeasible using manual methods.For example,it would be a dicult and tedious task for an
observer to nd out how many times a male y stays within 5 millimeters of a female y.These
aspects of quantication are computationally precise and reproducible.Moreover,automated clas-
sication of y specimens (in terms of their learning response and memory response) oer a novel
approach to investigating courtship behavior besides the approach based on average CI comparisons.
Development of a reliable computational method that is capable of analyzing courtship conditioning
would signicantly enhance our ability to utilize this behavior when testing learning and memory in
large-scale genetic screens and disease modeling.
The computational method includes identifying individual ies in the video,quantifying their size
(which is correlated with their gender),and tracking their motion.Besides tracking,we also compute
the head directions of the ies.Head direction computation is central to some of our geometric
computations.Geometric measures may be computed,for example distance between ies,relative
orientation,velocities,percentage of time the ies are in contact with each other,and percentage of
time one y looks at another.This data is computed for numerous experimental videos and produces
high-dimensional feature vectors that quantify the behavior of the ies.Clustering techniques,
e.g.,k-means clustering,may then be applied to the feature vectors in order to computationally
group each specimen by genotype.Our results show that we are able to automatically dierentiate
between normal and memory defective ies,and between normal and learning defective ies.We
also formulated a computational equivalent (Computational Courtship Index) of the existing CI and
compared it with CI.Our formulation of the CCI for a video clip is a linear function of the eight
numbers that are generated from the geometric computations for that video clip.
The remainder of the thesis is organized as follows.Chapter 2 presents related previous work
in automated behavioral analysis.It also discusses some of the existing computational tracking
methods.Chapter 3 brie y covers some biological background of the thesis.In Chapter 4 we
describe in detail our methods for the automation.Chapter 5 explains the results of the analysis.
Chapter 6 concludes with some remarks and future direction of the work.
3
2.Previous Work
In the following sections,we discuss related work in behavior analysis and relevant research in
object tracking.
2.1 Behavior Analysis
We created a tracking system to analyze courtship behavior of a pair of ies for use in learning
and memory studies.Several previous studies have been done on behavior modeling and analysis.
Some of the related work is discussed below.
Dankert et al.describe an automated social behavior (aggression and courtship) measurement
of Drosophila [10].They try to understand the connection between genomes and aggression within
courtship behavior.They developed a vision-based tracking system.A self-calibrated camera was
mounted on top of the target arena.The system monitors a pair of interacting ies and computes
their location,orientation,and wing posture.The detection and tracking phase computes the y
silhouette and the position of the head and wings.The tracking assumes that the female y is
larger than the male y.For a male-male assay one y is labeled for tracking.Fly actions such as
lunging,tussling,wing threat,wing extension and circling,chasing,and copulation are measured
from the tracked y.The new process speeds up behavior analysis with an accuracy that in turn
allows for large-scale genetic screens with high throughput.The computed features also allow for
the construction of an ethogram which accounts for the frequency of each action,as well as the
frequency of one action followed by the same or another action.After detection of the actions they
study genetic and environmental in uences on social behavior.Though the system has a courtship
analysis component,the utilization of this technology for assessing courtship suppression in learning
and memory,specically how this analysis compares to manual courtship suppression analysis for
leaning and memory,is unknown.
Branson et al.present a method for high-throughput computer-vision-based tracking that quan-
ties individual and social behaviors of fruit ies [7].The tracking component can track multiple
(up to 50) ies.The tracking consists of a detection and an identity-assignment components.De-
tection was done based on a background subtraction method,and identity assignment is predicted
using the trajectory history.The system automatically detects various behaviors such as walk,stop,
4
chase,crabwalk,backup,sharp turn,jump,and touch using machine learning.It creates ethograms
from behaviors.They showed that these ethogram-based behavior quantications can dierentiate
between gender and genotype.They also found that relative positions of ies vary according to
gender,genotype and social environment during social interaction.
Valente et al.developed a system for analyzing the trajectory of Drosophila melanogaster in
a circular open eld arena [40].They characterize the locomotor behavior of the fruit y using
a video tracking system that tracks a single y.They used a spatial and temporal probabilistic
measure to characterize the y locomotor behavior.The measure includes prior and joint probability
distributions of speed,position,segment duration,path curvature,and reorientation angle.Using
these measures they characterize the behavior of a single y and analyze its interaction with the
environment.
2.2 Tracking
Signicant research has been conducted on tracking objects in video frames [28].It ranges from
tracking human objects in complex scenes to tracking animals and insects (single or in groups)
in experiments.Our automated courtship behavior characterization system also has a tracking
component.The system is developed to track two ies in a circular chamber.The developed
tracking component enables behavior analysis and processes data oine.Relevant work on tracking
is discussed below.
Israd et al.developed a system,BraMBle (A Bayesian framework for multiple blob tracker) that
combines a blob tracking and particle ltering approach to develop a Bayesian multiple-blob tracker
[22].They experimented with the developed system to track people.It uses a single static camera
to track variable numbers of people entering and leaving the scene.The main contribution of the
research is a novel observation model based on the theory of Bayesian correlation called the multi-
blob likelihood function.The observation model is developed using individual statistically-learned
background patches and pooled foreground patches.The available observation model is used for
a particle lter to nd the posterior probability distribution over the number and conguration of
objects.The system shows a robust and ecient implementation of tracking on a small number of
objects in real time,but fails to distinguish when one object passes in front of another by switching
the identication assignments.They suggested an improved tracking algorithm may be possible by
assigning an individual foreground model to each object.
5
Khan et al.identify objects entering and leaving a tracking arena and keep track of the trail of
the targets [25].The targets in their study were insects in an enclosed arena.Their study leverages a
motion model that is able to depict the target behavior during interaction of the targets.They use a
particle lter which can handle complex measurement models.They incorporate a Markov Random
Field (MRF) as a motion prior which can handle\hijacking"during target interaction.Inclusion
of an MRF motion prior works as an additional interaction factor to the importance weights in a
particle lter.For large numbers of targets it is infeasible in terms of computational complexity
to implement the joint particle lter.To overcome the exponential complexity of the MRF they
used Markov Chain Monte Carlo (MCMC) sampling.The inecient importance sampling step of
the particle lter implementation is replaced by an MCMC sampling step which also improves the
tracking results.They also extended the particle lter to track variable numbers of targets by using
a reversible jump MCMC (RJMCMC) sampling step.In their study,they show that a traditional
joint particle lter is incapable of tracking a large number of targets resulting in high numbers of
errors.Introduction of the MRF motion prior with MCMC-based sampling improves the tracking
quality,especially during target interaction.The RJMCMC-based particle lter performs well in
tracking a variable number of targets.
Branson et al.address tracking multiple targets through occlusion [6].They track three brown
mice from a video capturing the side view of the arena (unlike most tracking which is done from the
top).The tracking leverages automated behavior analysis of individual mice,helping to improve au-
tomated data collection for animal behavior research.It breaks the tracking into several submodules.
In the background/foreground labeling module,each pixel in each frame is classied as background
or foreground.Having each pixel in a frame classied as background and foreground,it then assigns
the foreground pixel membership to one of the mice.In a frame where the mice are separated this
assignment of mice to pixel locations is equivalent to tting a mixture of k Gaussians to the loca-
tions of the foreground pixels.Detection of occlusion events is done using the Gaussian Mixture
Model (GMM) parameters computed using Expectation Maximization (EM).After detecting the
occluded frames,tracking through occlusion is based on optical ow estimation using multiple ane
models in conjunction with a prior.High amount of occlusion and lack of features on the targets
make the optical ow cue inadequate for accurate motion estimation.The optical ow cue combined
with a depth ordering heuristic (matching up the front mouse to the start of an occlusion with the
front mouse at the end of the occlusion) yields a robust identication of mice during the occlusion
event.The algorithm combines numerous cues for tracking thereby failure of one module does not
6
undermine the overall tracking result.
Tao et al.present a CONDENSATION-like sampling algorithm [21] for tracking multiple objects
and its variants [38].The proposed algorithm explicitly supports counting,addition,deletion,and
occlusion of objects.They also address the potential high-dimensional sampling space of the cong-
uration with a hierarchical algorithm.The algorithms have been tested on real video sequences to
track multiple people.
Comaniciu et al.focus on tracking based on target representation and localization [9].Target
representation focuses on the target model and target candidate.The kernel-based tracking tech-
nique used a feature-histogram-based target representation.The target localization problem was
formulated using the basin of attraction of local maxima.A derivation of Bhattacharyya coecients
[3] was used for a similarity measure.The proposed algorithm can track despite the presence of
camera motion,partial occlusion,clutter,and target scale variations.They also described three ex-
tensions of the basic algorithm.The rst extension incorporates the background information in the
tracking.Another variation of the algorithm integrates a Kalman Filtering [23] framework.Finally
the kernel-based tracking was used as a face tracker.
2.3 Contributions
Many aspects of y behaviors have been studied previously [10],[7],[40].To the best of our
knowledge,no work has been done to individually categorize y specimens from their courtship
behavior.The novelty of our work is a technique for the automated categorization of individual
y specimens in terms of the y's learning and memory responses.Additionally,we developed a
tracking system utilizing geometric computations on the extracted images of a video tape.The
contributions of our work are listed below:
1.Automated y tracker.In order to calculate the novel metrics for the quantication of
the courtship behavior,a new tracking algorithm has been proposed.The tracking algorithm
identies the male and female ies separately.The identied male y is denoted by a round
circle in the processed video le.
2.Identication of the head direction of ies.Aside from tracking the ies,their head
directions are also computed.The direction of the head is displayed by a green line from the
estimated centroid of the y.
7
3.Novel metrics.Some novel metrics have been dened for the quantication of the courtship
behavior.The metrics are:1) percentage of frames when one y looks at the other,2)
percentage of time the ies are together,3) distribution of the inter-centroid distances of
the ies,4) distribution of the head direction angles of the ies,and 5) distribution of the
speeds of the ies.
4.Automated grouping.Ak-means clustering algorithmhas been used to automatically group
the ies by their learning and memory responses.In doing so,each of the specimens (10 min
duration of y video) has been represented by a feature vector of 8 numbers.These 8 numbers
are computed from the novel metrics formulated for the analysis.
5.Formulation of a computational equivalent of existing Courtship Index.Biologists
use the Courtship Index (CI) score to express a quantitative measure of the courtship activity
of a y pair.We formulated a computational equivalent of this CI.We compared our new
Computed CIs with the manually obtained CIs.Our formulation of CI for a video sample is
a linear functions of the eight numbers computed from the geometric measures for that video
sample.
8
3.Biological Background
Flies of dierent genotypes used in the analysis were raised in the Marenda Laboratory of the
Department of Biology,Drexel University.The genotypes include:1) wild type,2) ies expressing
the human Alzheimers disease proteins APP and BACE (Elav-Gal4;UAS:APP;UAS:BACE,as
described in [8]),and 3) outcrossed controls of this genotype (Elav-Gal4;+;+).Individual ies
from each of these genotypes showed both normal and abnormal learning and memory.However,
ies expressing UAS:APP and UAS:BACE showed abnormal memory as a group [8].In the follow-
ing sections,we discuss some concepts related to the behavior analysis of the fruit y,Drosophila
melanogaster.
3.1 Courtship Behavior
Courtship behavior is the most common innate trait that is seen in the behaviors of fruit ies.
A naive male y that has been separated at birth and has never encountered a female y will tend
to court the rst female y it encounters in its adulthood.Genetic regulation has been shown to
change courtship behavior[16].
Specic Mate Recognition System (SMRS) is a species-specic control of all the sensory infor-
mation (visual,acoustic,and olfactory) that are required for a successful courtship between a male
and a female y [31][16].Courtship in Drosophila melanogaster follows a series of sequential actions
that can culminate with copulation [16][34].The series of actions involve:
1.Orientation:The male orients towards the potential partner,
2.Tapping:He taps her with his frontal leg,
3.Wing vibration:Flies vibrate their wings to create a\song",
4.Licking:If the partner is stationary the male will circle around her and lick her genitalia with
his proboscis,
5.Attempted copulation:He will attempt to copulate with her by bending his abdomen and by
thrusting his genitalia towards her.
The actions are depicted in Figure 3.1 [16].
The ies exchange various signals in the course of courtship.The signals are comprised of:
9
Figure 3.1:Courtship actions (taken from [16]).
10
1.Visual Cues,
2.Olfactory Cues,
3.Auditory Signals.
The absence of one or more signals has a signicant impact on the change of courtship behavior
[16].The male y follows various visual,olfactory,and auditory cues to initiate courtship with his
potential court mate [11] [12].
Greenspan et al.discuss these signals and their association in Drosophila courtship to a large
extent [15].A brief description of these signals are discussed in the following sections.
3.1.1 Visual Cues
Flies use various visual cues during courtship,which range from dynamic visual cues (e.g.,wing
display,locomotion activity,motion) to static visual cues (e.g.,color and shapes).The responses
to visual cues are also species-specic.Some species tend to court even in the dark,where as
other species show depressed courtship behavior in the absence of light [30] [36].The courting male
identies the conspecic female prior to courtship.The male orients towards the courting partner
and taps her with his foreleg.The locomotor activity promotes the inception of courtship.Upon
increased reception of courtship,the female shows decreased locomotion activity,which allows the
male to perform the licking activity [39].
3.1.2 Olfactory Cues
The detection of volatile pheromones,stimulatory substances that in uence courtship,are de-
tected at a distance before the male and female ies come into their rst contact.It has been
found that males that are presented with increased volatile female pheromones exhibit an increased
courtship activity [41].Mature ies detect chemical substances during the tapping and licking ac-
tions.Drosophila melanogaster has separate receptors for the stimulatory pheromone and inhibitory
pheromone.Pheromones also play a positive role in attracting ies of dierent species to the same
food.
3.1.3 Auditory Signals
Drosophila males sing a species-specic\love song"in the form of acoustic signals.This\love
song"is important for species recognition [33].The\love song"is comprised of humming and a
11
series of pulses.The song stimulates the female y to be more receptive.The auditory signals have
several other variations.Mature males ick their wings toward the courtship partner.On the other
hand,mature females buzz with their wings.Some species of Drosophila perform scissoring aside
from icking,while some male ies send a rejection song towards a courting male in a male-male
assay.
3.2 Courtship Index
Courtship Index,in short CI,is a standard quantitative measure in the biology community
that expresses the courtship activity between a pair of ies.This is dened as a fraction of the
observation period during which any courtship-specic activity is exhibited by the interacting y
pair.The accounted courtship activity includes orientation of the y,wing extension,and wing
icks.The CI ranges from 0.0 to 1.0,0.0 representing the absence of courtship,and 1.0 expressing
comprehensive courtship.Any fraction in between represents equivalent partial courtship behavior.
3.3 Conditioned Courtship Suppression
Siegel et al.conducted a detailed assay on the conditioned responses in the courtship behavior
of normal and mutant Drosophila [34].The ndings of this analysis lay the foundation for the
behavioral paradigm called\Conditioned Courtship Suppression"or\Courtship Conditioning"[17].
The courtship behavior of the male is modied depending on its exposure to any prior courtship
experience.
Their rst experiment observed that naive male Drosophila court active virgin females more
vigorously than active mated females.They conducted an hour long experiment on a number of
naive males.The experiment showed that with time,the naive males that were paired with active
mated females showed decreased courtship activity.On the contrary,the naive males paired with
active virgin females exhibited no signicant decrease in courtship throughout the duration of the
exposure.This depressed behavior in the presence of active mated females does not depend on the
activity of the female or any visual cues received by the male.
This independence of visual cues is supported by another experiment,where blind ies were
exposed to the same set of two females (active mated female and active virgin female).The blind
male ies also courted active virgin female more vigorously than the active mated female.
Another observation consisting of exposure of a naive male to an immobilized female also showed
12
depressed behavior of the naive male towards the immobilized mated female.This substantiates the
claim that decline in courtship activity towards mated females does not depend on the activity of
the female.
The experience of courting a mated female in uences the future courtship activity of the naive
male.The experiment showed that the amount of time a male courted a mated female has inverse
in uence on his courtship towards any subsequent virgin females.The naive male that had been
exposed for a longer period of time to a mated female tended to court less vigorously with any
subsequent virgin female.
When the males were kept alone in a courtship chamber for 30 minutes and then led to court
with virgin females they expressed a normal CI.This experimentation corroborates the claim that
depressed activity is due solely to the prior experience with a mated female.
The supposition is that a naive male that has actively courted a mated female without success
becomes exhausted and shows a lower CI.They showed that this claim is not strong,because a
naive male shows no exhaustion while actively courting a virgin female without success,producing
a normal CI.
Siegel et al.conducted their nal experiments on an amnesiac,a memory defective y.Wild-
type ies have retentive memory.They tend to retain more than half of their learned behavior
one hour after the learned behavior.On the other hand,learning behavior decays relatively fast
in amnesiac ies,though they have no defect in learning (they show more vigorous courting with
virgin females than with fertilized females).Prior experience with mated females does not change
their subsequent courtship behavior because of the decay of the learned behavior.Wild-type naive
ies showed depressed courtship behavior because of their prior courtship experience with mated
females.
The data for our study comes from experiments which employ the conditioned courtship sup-
pression behavioral model on various genotypes of Drosophila in a laboratory setting.The learn-
ing and memory responses of various genotypes (wild-type,Alzheimer Disease (AD)) of Drosophila
melanogaster ies are assayed.The following sections discuss the specic procedures of the biological
assay.
13
Figure 3.2:Average CI values for a successful learning response.
3.4 Learning Response
To determine the learning response of a male Drosophila,the male y (wild-type or AD) and a
mated or fertilized female y are placed in a courtship chamber for 60 minutes.The CI is quantied
by human observation for the rst 10 minutes of the assay duration.A similar quantication of the
CI measure from human observation is carried out for the nal 10 minutes of the assay duration.
If the observed CI in the last 10 minutes drops signicantly compared to the CI observed in rst
10 minutes,then it can be said that the male has exhibited a normal courtship learning response.
If there is no signicant dierence between the CI of the rst 10 minutes and the CI of the last 10
minutes,then it is an indication of a defective learning response in courtship.Figure 3.2 shows a
normal or successful learning response for a set of samples.The bar chart in the gure represents
the average CI of the set's rst 10 minute learning responses on the left.The average CI of the last
10 minute responses of the same set is on the right.The higher average CI of the rst 10 minute
responses signies a successful learning scenario.
The duration (i.e.,60 min of observation with a mated female) for obtaining the learning response
is called the training phase of a male Drosophila.This training phase is important considering the
assay's progression towards the memory response observation.
14
3.5 Memory Response
The memory response of a male Drosophila can be obtained using a similar courtship assay.The
same male Drosophila that is used in the training phase is assayed to nd its memory response.Five
phases of memory are dened in the Drosophila courtship study [15].They are the following:
1.Immediate Recall Memory (phase that persists for 0-2 minutes after the training),
2.Short-term Memory (phase that persists out to 1 hour after the training phase),
3.Medium-term Memory (phase that persists out to 6 hours after the training phase),
4.Anesthesia-resistant Memory (phase that persists out to 2 days after the training phase),
5.Long-term Memory (phase that persists out to 9 days after the training phase).
In our assay we conducted immediate recall memory experiments for 22 pairs of ies.To obtain the
immediate recall memory response of a male y (Wild-type or Alzheimer Disease(AD)),the male
y is transferred into a separate courtship chamber within 2 minutes after its training phase.In
the new courtship chamber the male was tested with a virgin female unlike the training phase,in
which the male courts a fertilized female.The new pair was assayed for 10 minutes.During this
10 minutes assay the CI of the male y is analyzed by a human observer.This CI is compared
with a sham-trained male y of the same species (e.g.,melanogaster).Sham training is a training
phase with the same duration (i.e.,60 minutes) as the usual training,however the male y is not
paired with a female y.If the CI of the immediate recall memory phase of a male y is signicantly
less compared to the CI of the immediate recall memory phase of a sham-trained male y of the
same species,this scenario demonstrates that the male y is capable of remembering its suppressed
courtship.In other words,this situation indicates that the subject male y is not defective in its
immediate recall memory response.Contrary to this event,if the CI of the immediate recall memory
phase of a male y has no signicant dierence from the CI of the immediate recall memory phase of
a sham-trained male y of the same species (e.g.,melanogaster),then it indicates that the subject
male y is defective in its immediate recall memory response.Figure 3.3 shows a successful memory
response for a set of samples.Let us denote this set as set A.The left bar of the chart is the average
CI of set A's immediate recall 10 minute memory responses.The right bar represents the average
CI of the immediate recall 10 minute responses for a set of sham-trained ies of the same species
(e.g.,Drosophila melanogaster).Let us denote this set as set B.A lower average CI value for the
15
Figure 3.3:Average CI values for a non-defective memory response.
ies in set A compared to the average CI value for set B re ects a successful memory response for
set A.
16
4.Methods
We developed a computational pipeline to characterize the memory and learning behavior of the
fruit y based on the analysis of video of courting ies.A sample video of 10-minute duration is
processed in the pipeline.Images are extracted from the video.This sequence of images is used
in segmentation,ltering,identifying the ies for tracking,and computing geometric measures.
Feature vectors are generated for a set of samples using the computed geometric measures.A
clustering method,i.e.,k-means clustering,is used to group the samples according to their behaviors.
Additionally,the feature vectors are used for generating a Computed Courtship Index (CCI),an
equivalent to the existing Courtship Index (CI).The stages of the pipeline are the following:
1.extract images from the video,
2.crop individual chambers,
3.segmentation,
4.ltering,
5.identify male/female ies,
6.determine head direction of each y,
7.compute geometric measures,
8.create feature vectors,
9.clustering,and
10.generate computed courtship index.
The pipeline is depicted in Figure 4.1.An arrow from one stage to another in the pipeline indicates
the ow of the processing and data.The input and output of each stage are also labeled over the
arrows.Results produced from each pipeline stage are discussed in Chapter 5 (Results).Each of
the stages is discussed in the following sections.
17
Figure 4.1:The process pipeline.
18
4.1 Extract Images from the Video
The goal of this stage is to extract individual frames from an input video.A y pair is video-
taped while keeping the pair inside a circular chamber.An input video may contain 3 to 6 of
these chambers.Individual frames are extracted from the video at a rate of 30 frames-per-second.
QuickTime [19] player is used to extract the images.These image frames are used in the subsequent
stage for cropping out individual assay chambers.
4.2 Crop Individual Chambers
Chambers are isolated from the cropped images extracted from the video.The rectangular
coordinates surrounding a chamber are manually identied.The coordinates are specied for the
rst frame.Each cell is separated out using the same coordinates from the subsequent frames.The
sequence of all the cropped images are used in the segmentation stage for detecting the background,
ltering out noise,lling in the voids inside the identied white regions,and identifying y objects.
The pseudo code in Algorithm 1 explains the Crop Algorithm.
Algorithm 1 Crop Algorithm(ImageArray,x,y,w,h)
for each image in img ImageArray do
Crop img of size w pixels wide and h pixels long from the start position at pixel location (x,y)
end for
4.3 Segmentation
A binary image of each cropped out frame is found using a background subtraction algorithm.
The rst step for this stage is to identify the background.The rst 1000 frames are used to compute
the background image.The white regions,representing the ies,are detected by subtracting each
frame from the computed background image,then classifying pixels above a certain threshold.The
two steps in the segmentation stage are described in the following subsections.
4.3.1 Detect Background
This component considers a set of N images to nd the most frequent color that appears in each
pixel location.The derived background is a color image,where each pixel is the most common color
19
fromthe set of N images.In general,using the rst 1000 cropped images was found to be empirically
sucient to detect the background accurately.The pseudo code in Algorithm 2 explains the Detect
Background Algorithm.
Algorithm 2 Detect Background Algorithm(ImageArray,N)
for each pixel location (x,y) do
Initialize maxCount to zero
for each img from the rst N contiguous images in ImgArray do
Pick the color c at pixel location (x,y) in image Img
Keep count of the color c for pixel location (x,y)
if count of the color c is greater than maxCount then
freqC = c
maxCount = current count for the color c
end if
end for
Assign the color freqC to be the background pixel color at pixel location (x,y)
end for
4.3.2 Image Subtraction and Thresholding
This component processes each of the (approximately 18,000) frames sequentially.The back-
ground image,an individual image frame,and a threshold value are used at each step of the itera-
tion.The result is a binary image,where pixels in a dierence image that exceed the threshold are
set to white color.The rest of the pixels are set to black.The threshold is a number between 0.0
and 1.0.If the brightness dierence (Hue,Saturation,Brightness (HSB) color model) between the
background and foreground pixel exceeds the threshold,the binary pixel is set to white.Empirically,
a threshold value of 0.15 is found to be sucient for most of the sample videos to identify the ies.
The pseudo code in Algorithm 3,the Subtraction and Thresholding Algorithm,explains how the
binary image is computed.
4.4 Filtering
The ltering stage removes noise from the binary segmented images,and identies the white-
region representation(s) of the y pair.The segmented binary image contains voids inside the white
regions.A\ll voids"step lls these holes inside white regions.Then,an\identify the ies"step
selects one or two white regions from the void-lled binary image.These steps are discussed in the
20
Algorithm 3 Subtraction and Thresholding Algorithm(BgImg,FgImg,Threshold)
for each pixel location (x,y) do
Initialize Image ResImg as a BLACK image of the same dimension as the FgImg
bgColor is the color at pixel location (x,y) from the image BgImg
fgColor is the color at pixel location (x,y) from the image FgImg
Find bgBrightness,the Brightness component in HSB color,from bgColor
Find fgBrightness,the Brightness component in HSB color,from fgColor
if (bgBrightness fgBrightness)  Threshold then
Set the color at the pixel location (x,y) of ResImg to a WHITE color
end if
end for
Image ResImg is the segmented binary image for Image FgImg
following subsections.
4.4.1 Fill Voids
The white-region representation(s) of the y pair may have voids inside of them.The rst step in
the ltering stage lls voids inside the white-region representation(s) of the y pair in a segmented
image.The voids inside the white regions are removed using an iterative ood-lling process,starting
from the top left corner pixel.The 4-connected black region is found.The white regions are dened
as the complement of the ood-lled black region;thus removing interior voids in the white regions.
The pseudo code in Algorithm 4 explains the Fill Void Algorithm [43].
4.4.2 Identify the Flies
We identify the white-region representation(s) of the ies in this step.White regions are identied
in the void-lled binary images.The area of each white region is calculated.Finally,a number of
white regions are selected from the current image depending on the following conditions:
1.If the ratio between the second largest white region and largest white region is less than or
equal to a given ratio,e.g.,
1
15
,then the largest white region is identied as a single blob.This
single blob signies that the two ies are in contact with each other.
2.If the current image has only one white region,then the region is selected as the y object.It
indicates that the ltered image is a single blob image,and once again the ies are together.
3.Otherwise,the largest two white regions are selected to represent the two ies.
21
Algorithm 4 Fill Void Algorithm(Img,x,y)
Initialize a new image fImg to BLACK color with the same dimension as Img
Initialize an empty queue S
if The color at pixel location (x,y) is not BLACK then
return
end if
Add the pixel location (x,y) into queue S
for each element s in queue S do
if The color of s is BLACK then
Initialize two new locations l and r to S
Move l to the left until the color of the position to left of l is not BLACK
Move r to the right until the color of the position of the right of r is not BLACK
Set the color of all the positions in between l and r in fImg to WHITE
for each position m between l and r do
if The color of the pixel location to the upward direction of m is BLACK then
Add that pixel location to S
end if
if The color of the pixel location in the downward direction of m is BLACK then
Add that pixel location to S
end if
end for
end if
end for
return the complement of fImg
The ltered image,which contains one or two white regions,is used in the subsequent stage for
tracking the male/female ies.
4.5 Identify Male/Female Flies
The ies are tracked according to their gender.The size of the y is correlated to its gender.
In general,female ies are larger in size than male ies.From this assumption,two series of y
objects are tracked in the images produced in the previous stage.A sequence of approximately
18,000 ltered images is divided into smaller subsequences.Male and female ies are identied for
each subsequence where the ies are separated.
During an experiment,the male y follows the female y frequently.In doing so, ies come
into physical contact.In this situation,the binary blobs of the ies merge together as detected by
the segmentation stage.Thus,the two connected ies appear as a single large white region and
subsequently selected as a single-blob image in the ltering stage.A single-blob image is not used in
the male/female object identication algorithm.Detection of the single-blob image,however,allows
us to measure the total number of images (during the whole analysis) where the ies are together.
22
The detection of single-blob images is done by counting the number of white regions in each ltered
image.
A subsequence is a contiguous series of images where each image contains two y objects (white
regions).The male/female object-identication algorithm is applied to these subsequences.A sub-
sequence with fewer than 15 (i.e.,half of the extraction frame-rate) images is not processed.
Let us denote a subsequence starting from image number M to image number N (in increasing
order of extraction).The images at the start and end of this subsequence are denoted by I
M
and
I
N
respectively.The male/female identication algorithm considers all the images in the subse-
quence I
M
to I
N
for assigning the male/female correspondence.The Euclidean distance between
the centroids of the two white regions in each image is computed.The frame with the maximum
inter-centroid distance is identied as a starting image for this subsequence.Let us assume that the
starting image is at image number S and the image is denoted by I
S
.From this starting image,
I
S
,a frame-to-frame correspondence for each white region is attributed to the rest of the images
in the subsequence.In other words,each white region in image S is associated with a y and this
association is propagated from image S in the forward and backward directions in the range of the
subsequence.A minimum distance criterion is used for establishing the y object correspondence.
The following subsection explains the minimum distance criterion.
4.5.1 Minimum Distance Criterion
Given two images,X and Y,each containing two white regions,a minimum distance criterion
assigns the label of a white region in image X to one of the two white regions in image Y.This
assumes that the labels of the white regions in image X are known.This correspondence is done by
nding the shortest Euclidean distance between the centroids of the white regions in image Y and
image X.Let us assume that the rst and second white regions of image X are labeled as A and
B.The order (i.e.,rst and second) of the white regions is dened to be the order in which these
regions were selected in the ltered image.The four-way distances between the white regions in
image Y to the white regions in image X are calculated (Figure 4.2).The minimum of these four
distances is determined,which gives the starting white region for labeling the unknown white regions
in image Y.More specically,the following method is used to assign the correspondence from the
known white-region labels of image X to the unknown white-region labels in image Y.The minimum
Euclidean distance from the rst white region in image Y to one of the two white regions in image X
is found.Let us denote this minimumdistance as Min
p
.Similarly,the minimumEuclidean distance
23
from the second white region in image Y to one of the two white regions in image X is found,and
this minimum distance is denoted by Min
q
.Then the labels of the white regions in image Y are
assigned from the following conditions.
1.If Min
p
is less than Min
q
,and Min
p
is the distance to white region labeled A then the rst
white region in image Y is assigned label A.The second white region in image Y is assigned
label B.
2.If Min
p
is less than Min
q
,and Min
p
is the distance to white region labeled B then the rst
white region in image Y is assigned label B.The second white region in image Y is assigned
label A.
3.If Min
p
is greater than or equal to Min
q
,and Min
q
is the distance to white region labeled
B,the second white region in image Y is assigned label B,and the rst white region in Y is
assigned label A.
4.If Min
p
is greater than or equal to Min
q
,and Min
q
is the distance to label A,the second
white region in image Y is assigned label A,and the rst white region in Y is assigned label
B.
As shown in Figure 4.2,the four-way distances fromthe white regions in image Yto the white regions
in image X are computed.The blue lines in Figure 4.2 show the distances from the top white region
in image Y to the white regions in image X.Similarly,the red lines in Figure 4.2 show the distances
from the bottom white region in image Y to the white regions in image X.The minimum of these
four distances is shown in Figure 4.3.The distance from the top white region in image Y to the top
white region of image X is found to be the minimum,and the label of the top white region in image
Y is assigned to the label of the top white region in image X,i.e.,label A.Subsequently,the label of
the bottom white region in image Y is assigned to label B.The nal labels are shown in Figure 4.4.
Given the minimum distance criterion,the frame-to-frame correspondences of the white regions
are assigned from I
S
to its adjacent images in the subsequence both forward and backward in
time.The labels for the rst and second white regions of image I
S
are initialized to be A and B
respectively.The immediately adjacent images,I
S1
and I
S+1
,are labeled from the labels of I
S
using the minimum distance criterion.Labels (A and B) for the white regions of each image in the
subsequence are then assigned.
24
Figure 4.2:Computing the four distances between the white regions in image Y and the white
regions in image X.
Figure 4.3:Determining the minimum of these four distances between the white regions in image Y
and the white regions in image X.
Figure 4.4:Assigning labels to white regions in image Y from the labels in image X using the
minimum distance criterion.
25
Once the white regions are labeled over all the images in the subsequence,two series of white
regions are identied.One series of white regions is labeled A,and other series is labeled B.The
male/female identication process nds the average area of the white regions in each of the two
series in the subsequence.The series with the larger average area is designated to be the series
that corresponds to the female y.The other series corresponds to the male y.We assume that
the series of white regions having the larger average area will correspond to the female y over the
subsequence of images.The pseudo code in Algorithm 5 explains the y identication algorithm.
Algorithm 5 Fly Identication Algorithm(s)
Find the frame S in the subsequence s where the distance between the two white regions is
maximum
Create two series of white regions by assigning labels to the white regions in all images in s starting
from S using a minimum distance criterion
Calculate the average area of each series
Label the series with the larger average area as the female y,and label the other series as the
male y
Given a series of white regions in the subsequence labeled as a male y and the other series labeled
as a female y,we compute the head directions of each series using the head direction algorithm in
the next stage.
26
4.6 Determine Head Direction of Each Fly
The head direction algorithm computes the head direction of each y object in the subsequences
of the ltered images.Before describing the steps of this algorithm let us dene some notations.
Velocity Vector of a fly
i
in frame J is denoted by V
i
J
.The centroid of a fly
i
in frame J
is denoted by C
i
J
,a 2-dimensional Euclidean point.The velocity vector V
i
J
of frame J for fly
i
is
calculated using the following equation.
V
i
J
=
C
i
J+1
C
i
J
J +1 J
pixels=frame:
The denominator in the velocity vector equation is 1,since we compute the velocity vectors from
two consecutive frames.The velocity vectors fV
i
M
;V
i
M+1
;;V
i
N
g of all the frames in the subsequence
are calculated for the fly
i
series.
Eigen Vector of a frame J for fly
i
is denoted by E
i
J
.E
i
J
is the eigen vector that corresponds
to the largest eigen value of the covariance matrix generated from the pixel locations in the 2-
dimensional region of fly
i
.The white-region representation of fly
i
is identied in an earlier stage.
Given the 2-dimensional pixel coordinates of a fly
i
in frame J,the Compute Eigen Vector Algorithm
in Algorithm 6 computes E
i
J
.
Algorithm 6 Compute Eigen Vector Algorithm(XArray,YArray)
Initialize xx,yy,and xy to zero
Compute the centroid (x
c
,y
c
) from XArray and YArray
for each element x in XArray and element y in YArray do
a = (x-x
c
)(x-x
c
)
b = (y-y
c
)(y-y
c
)
c = (x-x
c
)(y-y
c
)
xx = (xx+a)
yy = (yy+b)
xy = (xy+c)
end for
Construct the covariance matrix C using the quantities xx,yy,and xy
Determine the eigen vector corresponding to the largest eigen value of the matrix C
Oriented Central Axis OCA
i
J
in the direction of the velocity vector V
i
J
of fly
i
is determined
from the sign of the dot product between V
i
J
and E
i
J
.We already dened how V
i
J
and E
i
J
are
calculated.Given V
i
J
and E
i
J
,OCA
i
J
is calculated from the following conditions:
27
OCA
i
J
=
8
>
>
>
>
<
>
>
>
>
:
E
i
J
if (V
i
J
 E
i
J
) > 0
E
i
J
if (V
i
J
 E
i
J
) < 0
0 if (V
i
J
 E
i
J
) = 0
(4.1)
0 is a 2-dimensional zero vector,and\"represents the dot product of two vectors.In other
words,OCA
i
J
is fly
i
's eigen vector E
i
J
that is in the same direction as V
i
J
.OCA
i
J
for all the Js in
the y subsequence are calculated.
Figure 4.5 shows a sample frame J.The selected y is high-lighted by the blue rectangle.The
oriented central axis for the selected y may be positioned in two opposite directions depending on
the angular relationship of V
i
J
and E
i
J
.Figure 4.6 and Figure 4.7 show the directional variations of
OCA
i
J
.
Head Direction Vector of fly
i
in frame J is denoted by HD
i
J
.The head direction vector
HD
i
J
of frame J for fly
i
is either the eigen vector E
i
J
or the opposite direction of the eigen vector,
i.e.,E
i
J
.The head direction vector of the starting frame,HD
i
S
,in the subsequence is calculated
rst.Head direction vectors for the rest of the frames in the subsequence for fly
i
are calculated
from HD
i
S
.
Given these notations,the steps to determine head directions of the ies in the subsequence are
described in the following subsections.
4.6.1 Identify the Longest Positive Dot Product Subsequence
In a courtship assay,a y moves in the forward head direction for most of the time in the assay
chamber.This forward movement includes:1) the male y chasing the female y,and 2) random
circulation around the chamber.Our assumption in computing the longest positive dot product
series of the OCA
i
j
s is motivated by this observation.The longest positive dot product series of
OCA
i
j
vectors in a subsequence captures the longest forward movement of the ies.This leads to
the computation of the head direction of the y in the subsequence.Let us consider a subsequence
extending from frame M to frame N.Figure 4.8 shows the subsequence.The blue circles denote
the series of fly
i
for which we are computing the head directions.To compute the longest positive
dot product series we need to calculate the OCA
i
j
s for all of the frames in the subsequence.We
already described how to compute the OCA
i
j
in a frame j for fly
i
.The OCA
i
j
-annotated y series
is shown in Figure 4.9.The blue arrow indicates the velocity vector V
i
j
in frame j.Oriented central
axis OCA
i
j
s are denoted by yellow arrows.The example shows that the OCA
i
j
s are oriented in
28
Figure 4.5:A sample frame J with the chosen y high-lighted by the blue rectangle.
Figure 4.6:Computation of Oriented Central Axis OCA
i
J
for a y from its velocity vector V
i
J
and
eigen vector E
i
J
.
Figure 4.7:Computation of Oriented Central Axis OCA
i
J
for a y from its velocity vector V
i
J
and
eigen vector E
i
J
.
29
dierent directions in the subsequence depending on the velocity vectors in each of the frames.The
computation of a longest series of consecutive positive dot products between the adjacent OCA
i
j
s will
give us the longest number of frames where the angle between each of the adjacent pairs of OCA
i
j
s is
less than 90 degrees.In other words,the adjacent pairs of OCA
i
j
s are oriented in the same direction.
To calculate the longest positive dot product series we compute the pairwise dot product between
each of the adjacent OCA
i
j
.We select the longest series of dot products in which all the products are
positive.Figure 4.10 shows identied longest series of positive dot products.The identied longest
positive dot product starts at frame Q and ends at frame R within the subsequence.
4.6.2 Find the Head Direction of the Start Frame
Given the series of longest positive dot products of the OCA
i
j
s in the subsequence,we choose
the frame with the maximum speed in the series as a start frame.Let us denote this start frame
to be frame number S.The head direction vector HD
i
S
with respect to fly
i
of frame number S is
OCA
i
S
,since it is the central axis vector pointing towards its velocity vector V
i
S
.Figure 4.11 shows
the computed head direction of frame S by a green arrow.The selected start frame S is shown in
the middle in Figure 4.10.
4.6.3 Propagate the Head Direction Vectors
The computation of the unknown head direction vector HD
i
X
for fly
i
of frame number X imme-
diately adjacent to frame number Y,whose head direction vector HD
i
Y
for fly
i
is known,is done
by the following equation.
HD
i
X
=
8
>
<
>
:
E
i
X
if (HD
i
Y
 E
i
X
)  0
E
i
X
Otherwise
The head direction vector HD
i
X
is the Eigen vector of fly
i
that is in the same direction as HD
i
Y
.
Since the head direction vector HD
i
S
for fly
i
of start frame S is known,the head directions of the
frames immediately adjacent to frame S are computed from the above equation.The remaining
frames in the subsequence are propagated in the same way.Figure 4.12 shows the head direction
propagation in the forward direction from start frame S.
The pseudo code for determining a y's head direction is in Algorithm 7.The pseudo code for
calculating the starting frame in a subsequence is described in Algorithm 8.
30
Figure 4.8:A subsequence starting from frame M and ending in frame N.The blue circles denote a
y series in this subsequence.
Figure 4.9:The subsequence with OCA
i
j
denoted by the yellow arrows and V
i
j
denoted by the blue arrow.
Figure 4.10:The longest positive dot product subsequence within the subsequence starting from
frame M to frame N.The identied longest series starts at frame Q and ends at frame R within the
subsequence.
31
Figure 4.11:Start frame S with head direction denoted by a green arrow.
Figure 4.12:Head direction propagation from start frame S in the forward direction.
32
Algorithm 7 Head Direction Algorithm(s)
for each series of frames t,in the subsequence s do
Find the starting frame S using Start Frame Algorithm 8
Find the head direction vector HD
S
of frame S
Propagate head direction vector HD to rest of the frames in the series t
end for
Algorithm 8 Start Frame Algorithm(t)
for each frame j in t do
Find the central axis vector OCA
j
using conditions 4.1
Find the longest series of positive dot products of consecutive OCA
j
s in t
Label the frame S in t with maximum speed as the starting frame
end for
4.7 Compute Geometric Measures
Five geometric measures are computed from a processed video sequence.They are the following:
1.percentage of frames when one y is looking at the other,
2.percentage of frames when ies are together,
3.distribution of the distances between the y centroids,
4.distribution of the head direction angles between the ies,and
5.distribution of the ies'speeds.
4.7.1 Percentage of Frames when One Fly is Looking at the Other
Each subsequence contains a y pair.The ies'head directions are computed using the head
direction algorithm described in the previous section.Given the head directions,we detect when
the head direction of one y intersects with the other y's body.If one head direction intersects the
other y,we say that the y is looking at the other.We count the total number of frames when
one y is looking at the other.Once we have a total number of\looking-at"frames,we calculate a
percentage that is the total number of\looking-at"frames divided by the total number of frames
where the ies are separated.Our assumption is that the female y does not court the male y.
The male y,however,pursues the female y.Therefore,the total count of\looking-at"frames
adequately captures the male courting behavior.The y looking-at algorithm is described below.
33
We run two iterations of this algorithm.The rst iteration assumes that the male y is the source
y object and the female y is the destination y object.This gives us the number of times the
male y is looking at the female y in a given subsequence.The second iteration of the algorithm
swaps the source and the destination y objects.This gives us the number of times a female y is
looking at the male y.The union of these two intersection tests produces one\looking-at"event
for a frame.If the union of the two intersections results is\true",we increase the total number
of\looking-at"frames by one.On the other hand,if the union of the two intersections results is
\false",the total count of\looking-at"remains the same.We perform two iterations of the looking-
at detection algorithm because the male/female detection algorithm is not robust.We encountered
a small number of subsequences of short length where the male/female detections are incorrect in
some 10-minutes video samples.Conducting two intersection tests,followed by a union operation
resolves the inconsistency in counting the total\looking-at"events.
The y looking-at detection algorithm marches along the head direction fromthe source y object
until the destination y object is reached or an image boundary pixel is hit.If the destination y
object is hit during the traversal,the algorithm indicates that the head direction of the source y
object intersects with the destination y object.Otherwise,if an image boundary pixel is reached,
the algorithm indicates that the source y's head direction does not intersect with the destination
y object.The steps of the algorithm are discussed in the following subsections.
Find the Start Point along the Head Direction in the Source Fly Object
We dene the rst black white-region boundary pixel along the head direction in the source y
object as the start point for the next step.In order to nd the start point we draw only the source
y in an image with the same dimensions as the original ltered image.We use this image in this
step to nd the black pixel at the boundary of the source y.We also have the head direction at
our disposal.One end point of the line representing the head direction is the centroid.We dene
the other end point of this line to be suciently away from the source y object.Let us denote the
head direction vector HD
i
J
and the Centroid C
i
J
of the source fly
i
in the frame J.The other end
point of the line representing the head direction of the y is calculated as follows.
HE
i
J
= C
i
J
+CONSTANT  HD
i
J
:(4.2)
34
where CONSTANT = Diagonal length of a frame ensures that HE
i
J
is outside the image.We start
traversing from point HE
i
J
,along the straight line towards centroid C
i
J
using the Bresenham Line
Drawing algorithm[14].Traversal terminates when it reaches a white pixel or the centroid of the
source y object.The rst white pixel that is hit is the neighbor of the boundary black pixel of the
white region representing the y.Let us say this black boundary pixel of the source y object is
BB
i
J
.We use this point in the next step.If the line does not hit any white pixel,then at the end
of the traversal we reach the centroid of the source y object.Let us also denote this pixel as BB
i
J
.
In this case,we use this point as the start point in the next step.Figure 4.13 shows a sample frame
with two ies.Figure 4.14 shows the red point which is found to be the start point BB
i
J
.
Find the Destination Object
A start point BB
i
J
is found from which we can start traversing along the head direction until we
hit the destination y object or we hit the image boundary.During this traversal we use the binary
ltered image for the current frame that contains both ies.During the traversal if a white pixel is
encountered then the pixel is on the destination y object;because in the binary ltered image there
are only source and destination ies.The traversal starts at the boundary black pixel of the source
y and advances in the head direction away from the source.Finding a white pixel indicates the
source y is looking at the destination y object.If the line reaches an image boundary pixel then
the source y object is not looking at the destination y object.Figure 4.14 shows when traversal
from the source object intersects the destination y object.Figure 4.15 shows the second iteration
of the intersection test.In Figure 4.15 the source y does not intersect the destination object before
reaching an image boundary pixel.The union of the two intersection test in Figure 4.14 and Figure
4.15 results in a true.Therefore,the total\looking-at"count is increased by one.
4.7.2 Percentage of Frames Flies are Together
We keep track of the number of times the ies are together.This situation is detected when
there is only one white region in the ltered image.We calculate the percentage of times over all
the frames when the two ies are together.In other words,we divide the frame count where the ies
are together by the total number of frames.Figure 4.16 shows a frame where the ies are together.
35
Figure 4.13:A sample frame with a male y (white region on the left) and a female y (white region
on the right).
Figure 4.14:Intersection test using the male y as source object and female y as the destination
object.Finding the start point (left image),which is denoted by a red point.Traversing along the
head-direction line to nd the intersection with the destination y object.
Figure 4.15:Intersection test using the female as source object and male y as the destination
object.Finding the start point (left image) which is denoted by a red point.Traversing along the
head-direction line in an attempt to nd the intersection with the destination y object.