breezebongAI and Robotics

Nov 6, 2013 (3 years and 7 months ago)



J. Haywood, W. Merril, T. Ladewski

Visteon Automotive Systems

J. W. V. Miller and M. Shridhar

The University of Michigan


An important safety issue in automobiles is the shatter
characteristics of windshield glass. There are specific North
American and European standards that all windshield glass panels
have to meet before being installed on an automobile. Statistical
sampling is used to verify that windshield glass meets the
ated standards. In such tests, an adhesive plastic film is
applied to a sample windshield panel. Subsequently, the panel is
placed over print paper and a metal center punch is used to shatter
it. Ten seconds after shattering, a high
intensity light briefly

exposes the print paper to record the shatter pattern. The print is
then developed and analyzed to check if the standards are being
met. The number of complete glass particles in any two square
inch regixon of the panel as well as the shape of these parti
cles is
then evaluated. Until recently, this process was carried out
manually resulting in inaccurate counting as well as subjective
shape analysis. In this paper the authors describe a low
automated analysis system that has been developed to count gl
particles in accordance with the ECE fragmentation test. The
region to be counted is manually marked on the print image and
scanned into a PC using a motorized hand scanner. Image
processing is performed on the scanned image to determine the
location o
f the marked region, compensate for uneven illumination,
reduce noise and determine crack boundaries. Prior to counting, a
special restoration algorithm is used to fill in missing parts of crack
boundaries. The system identifies and counts the number of
rticles inside the region and on the boundary. The results of the
processing are displayed on the PC and can be compared to the
original image


A problem commonly encountered in machine vision
applications is the analysis of cellular or in
terconnected patterns in
which the features of interest are either the cells or the boundaries of
the cells. In either case, an important part of the analysis is the
isolation and identification of the boundaries using a segmentation
algorithm or boundary

detection scheme. Applications that require
this type of processing include analysis of biological cells and crystal
grain studies
. The analysis of shattered glass, which has a similar
appearance, is the topic of this paper.

If the cell boundaries ar
e incomplete or have areas with poor
contrast, analysis becomes much more difficult. Missing boundaries
will cause adjacent regions to merge during connectivity resulting in
fewer "cells" with larger areas and highly irregular shapes. For this
reason it
is necessary to detect the broken boundaries, and
reconstruct them before meaningful analysis can be performed. A
strategy for accomplishing this is presented here with the emphasis
on the analysis of images of shattered automotive safety glass.

Due to l
egal requirements concerning specifications on tempered
safety glass, destructive testing on a given number of glass panels
taken from production is routinely performed. An adhesive
transparent film is applied to the sample to maintain the basic

of the panel during testing. The sample is placed over a
sheet of photosensitive print paper and shattered by a sharp impact
from a center punch. Ten seconds later, the print paper is exposed
briefly with a high
intensity light source and developed. Timi
ng is
critical since the breakage pattern changes significantly for several
minutes follow
ing the impact. A typical region from a print showing
a typical breakage pattern is given in Fig. 1. Manua
l identification of
two 5cm x 5cm regions, one with a high p
article density and the
other with a low particle density, is performed along with manual
counting of the particles within these regions. Manual measurements
are also done to obtain statistical shape information. This process is
labor intensive, error pro
ne and does not provide timely feedback. If
a production problem occurs, a significant number of finished glass
panels will have to be destroyed if the problem is not detected

A suitable machine vision system, however, would be able to
perform t
he above tasks much faster with more consistent results and
at a lower cost. Such a system is also compatible with automatic
report generation for documenting the glass characteristics and
providing permanent records of tested samples.

Image acquisition
is one of the more challenging aspects of this
application. Several different approaches were considered including
front and back lighting. One system, described in [3], used front
illumination. However, given the high resolution needed to resolve
the crac
ks and large area that needed to be scanned, a linear camera
was the only cost
effective approach for directly imaging the
breakage pattern. However, because rapid image acquisition is
required for this application, multiple cameras and a large amount of
speed memory would be needed. In addition, processing this
quantity of data is a challenging undertaking.

Because of the costs that would be incurred, it was decided to
continue using print paper for image acquisition. In addition, the
practice of ma
nually identifying high

and low
count regions,
required as part of
ECE Regulation No. 43
, was also retained
In principle, the entire print could be scanned and software used to
identify these regions. However, as noted previously, the large
amount of

data to be processed is daunting. Manually identifying
these regions is relatively easy and takes very little time. As will be
described in detail later, a hand scanner was used to capture images
of the manually marked regions as shown in

Fig. 2.

There a
re four main tasks to automate particle counting: 1)
identify the manually marked regions for analysis and minimizing
errors introduced by the marking, 2) binarize the image to preserve
fragment boundaries as completely as possible, 3) restore missing
incomplete portions of boundaries and 4) identify and
eliminate the effects of heater (defroster) grid lines (wires) that are
present in some panels.

The isolation and identification of boundaries using a suitable
segmentation algorithm or boundary detect
ion scheme are common
image processing problems and can be applied to both the heater
grid lines and crack boundaries. This problem is commonly
encountered in machine vision applications, for example in the
analysis of biological cells and crystal grain s
tudies [

When the particle boundaries are incomplete or have areas with
poor contrast, the analysis is much more difficult. Missing
boundaries will cause adjacent regions to merge during
connectivity resulting in fewer particles with larger areas and
highly irregular shapes. For this reason it is necessary to detect the
broken boundaries, and reconstruct them before meaningful
analysis can be performed.

The remainder of this paper describes the image processing
techniques used and the results of the
automated counting. The
next section describes the image processing used to enhance the
particle boundaries and identify heater grid lines. The boundary
reconstruction algorithm is then described. Next, the user interface
is discussed and, finally, the
results of comparisons between
automatic and manual counts and conclusions are presented.


Image processing is used to identify the bounding boxes for the

and low
count regions and recover as much of the breakage
attern as possible. As noted previously, extracting the breakage
pattern is a daunting task given that the pattern is never completely
resolved. The bounding boxes, however, are somewhat easier to
extract because color processing is used. To delineate the
regions, a red pen or marker is used to provide high contrast
against the blue hues of the print paper.

2.1 Image Acquisition

A motorized color scanner is typically used to scan the print.
It is aligned with the marked region so that this region
approximately centered in the scanned image and the scanned
image borders are relatively parallel to the borders o
f the marked
region as illustrated in Fig. 2. As will be discussed l
ater, precise
orientation of the scanner relative to the marked borders

is not
required for acceptable results as long as the entire marked area is
captured in the scan.

2.2 Detection of bounding boxes

Before analyzing the breakage pattern, it is necessary to
identify the marked regions on the print. Since the shape of the
border is highly constrained, it is not necessary to extract all points
along the border and simple processing based on the red and blue
components of the scanned is sufficient.

2.2.1 Contrast enhancement of the bounding box

When scanning is completed, re
d and blue components are
extracted. A typical example is shown in the top two images of Fig.
2. To enhance the contrast of the red border, the red image is
subtracted from the blue image by calculating the following at each
pixel location:

D[i, j] =
R[i, j]

B[i, j] +128

Here D[i, j] represents the difference gray
scale value for the pixel
at row i and column j and R[i, j] and B[i, j] are the red and blue
scale values. After this operation, the red border will appear
bright against a darker ba
ckground as shown in the lower left
image of Fig. 3.

It may be noted that more optimum relationships can be used to
enhance the red border based on spectral characteristics of the print
paper and marking pen ink. However, such an approach would
likely req
uire frequent recalibration given that developed print
paper can vary substantially in color. Also, some variation in
marker color may occur.

2.2.2 Thresholding

Once the contrast
enhanced image of the red outline is
available, thresholding can be applied t
o extract the red boundary.
The use of manually adjusted thresholds is an obvious solution but
incorrect settings are likely to cause the border to be incorrectly
identified. For this reason, it was desirable to employ an automatic
technique such as Otsu’s

. Thresholds derived from this
approach were not satisfactory because other features beside the
border had sufficient contrast to be detected.

It was observed that the red boundary was consistently brighter
than other areas but the differences

were relatively small. In
addition, the area of the marked region was relatively constant
from print to print. This suggested deriving a threshold based on a
percentage of pixels above a certain value. By calculating the
histogram of the image and summing

bins until the targeted
percentage of pixels is exceeded provided an effective way to
obtain the threshold. A typical result, shown in the lower
image of Fig. 3, demonstrates the result of calculating a threshold
that will leave .2% of the pixels on
. While the border is not
recovered completely, further processing will provide a good
estimate of it.

2.2.3 Estimating the border

During image acquisition, the scanner is oriented so that it is
parallel to the bottom of the bounding box. Since the scan
ner is
manually positioned, there is some error but it is relatively small.
Linear regression can be used on each side to estimate the location
and slope of each edge but only the points associated with a given
edge must be identified before regression can

be applied.

In essence, it is necessary to segment the bounding box into
four straight
line segments using the “on” pixels from the lower
right image in Fig. 3. Since the sides of the box will be relatively
parallel to the edges of the image, counting a
ll “on” pixels in each
horizontal line of the image will provide pixel counts as a function
of image line number. Two maxima occur corresponding to the
upper and lower sides of the border. Using only pixel locations
near the horizontal lines will provide a

good set of points for
estimating the two lines. In the same manner, pixel locations for
the two vertical lines can also be found by finding pixel counts
along the x

This technique works reasonably well but there will still be
some pixels included
that are not part of the line being fitted. High
contrast areas of the breakage pattern may cause erroneous “on”
pixels and pixels from orthogonal lines near the corners will also
be included. To minimize errors from these sources, regression is
applied it
eratively such that pixels greater than a set distance from
the fitted line are eliminated. This process continues until all pixels
are within a given distance from the fitted lines. Once the four lines
have been found, intersections are calculated to prov
ide the corners
of the bounding box.

2.3 Image Segmentation Issues

As stated above, a major difficulty in automated counting is
that the image typically does not have distinct boundaries for all
particles. A magnified image of a typical breakage pattern
shown in Fig. 4 in which a number of particle boundaries have
poor contrast or missing segments. The presence of noise is also
apparent. Due to intensity variations across the image, many of the
cracks are lost with simple thresholding, but this is min
imized with
more effective image processing. A more severe problem occurs
when a crack is perpendicular to the plane of the light source when
exposing the print. Such a crack is very narrow and does not block
a sufficient amount of light to be detected.
As a result, there are
missing boundary segments that cause the merging of two or more
pieces into one larger piece. This decreases the accuracy of the
particle counts. Such regions must be reconstructed before
accurate automated particle analysis can be


Initially, morphological processing was used to identify
particles. With the dark particle boundaries and bright particles,
successive erosion operation were performed until it either was
segmented into smaller pieces, or was reduced to an in
size. This method assumes that all particles are basically convex in
shape. While this is true most of the time, some particles,
especially those that are spline
like, are not convex. If a particle is
concave, repeated erosion operations woul
d incorrectly split the
piece at its thinnest point. Because of this problem, techniques to
restore missing boundary segments were developed.

The first step in preparing the image for analysis is to perform a
logarithmic transform to reduce the multiplic
ative effects of
illumination between areas of high and low intensity. Next, a
background estimate is made on the transformed image by
performing a gray
scale morphological closing operation with a
structuring element large enough to span the cracks. The

background image is subtracted from the transformed image to
create an inverted image with the particle boundaries appearing
bright against a dark background.

2.4 Heater Grid Line Detection

Next, if necessary, heater grid lines are identified.

Since the
grid lines are typically wider than the particle boundaries, they can
be enhanced relative to the boundary using a gray scale
morphological closing operation with a structuring element
somewhat smaller than the width of the heater grid lines. A

modified Hough transform is then used to estimate the slope and
intercept of the gird lines. Next, points associated with the grid
line are determined using the estimated slope and intercept and a
polynomial least squares fit done to determine the best
epresentation of the grid lines. Pixels associated with the fitted
grid lines are set equal to zero.

2.5 Contrast Enhancement Filter

A special filter is used to brighten pixels that are part of the
particle boundaries and darken others. The brightest p
ixels in a
boundary segment can be likened to the peak of a “ridge” that
results if the local region of the boundary is viewed as a three
dimensional surface as illustrated in Fig. 5. A ridge point is
characterized by having points of similar intensity in

one direction
and points with significantly lower intensity in a perpendicular
direction. For example, a point on a vertical ridge would have
pixels of similar brightness above and below it and darker pixels to
its right and left. The neighborhood aroun
d the pixel being
processed needs to be larger than the ridge width so that some of
the points examined are not on it. The new pixel value is obtained
by calculating half the maximum of the sum of opposite edge
pixels minus the sum of the edge points perp
endicular to these
points for each point on the edge. This means that



is the value at pixel


is the size of the neighborhood
to examine and


The resulting image is
then binarized as illustrated in Fig. 6.
The user sets the threshold value by observing the resulting image
interactively. The objective is to recover as much of the particle
boundaries as possible without detecting other unrelated areas. The
presence of
these spurious areas must be avoided since they will
cause excessive segmentation.

A morphological closing operation is applied to the binary image
to fill in small gaps in particle boundaries. The image is then
thinned preserving 4
connectivity for the fo
reground. At this point,

boundary restoration can be applied to the resulting image given in
Fig. 7. Note that this image has been reversed (along with
subsequent thinned images) to provide better viewing.

3. Reconstruction algorithm

Following the above
processing, not all particle boundaries are
complete. Typically, two cases are encountered. In the first case,
there is a section of a border missing as illustrated in the right
image of Fig. 8. Here, reconstruction involves connecting the two
ches. In the second case, a section of border is missing with
no apparent matching segment. This is illustrated in the left
image of Fig. 8, in which there is only one branch that must be
extended to complete the boundary pattern.

3.1 Reconstruction


An algorithm to perform the reconstruction is outlined below
and later described in more detail:


Identify all points with only one neighboring “on” pixel.


Eliminate all "branches" that are less than several pixels long
since these are likely caus
ed by noise.


Locate all base points, the first point along a branch that has
more than two neighbors.


Determine direction of all segments.


Connect points with only one neighbor.

3.2 Identification of broken segments

Since the image is 4
connected and thin
ned, every point has at
most four neighbors. To identify points with only one neighbor,
the pixel value (represented internally as an 8
bit character) is
replaced with the results of the following binary classifier, where
the asterisk represents the curre
nt pixel.

[0 4 0]

[1 * 2]

[0 8 0]

The resulting value for a pixel with an adjacent pixel to the
north, and another to the west would be 5, and so forth. The
resulting pixel values now represent the connectivity and spatial
relations of each point in the
image. Identifying dead end branches
is simply a matter of finding pixels with only 1 neighbor, namely
those with values of 1, 2, 4, or 8. Once such points have been
found, and recorded as end
points, their accompanying base point
must be found. The base

point is the point at which a branch either
connects to the boundary pattern, or in the case of a totally isolated
branch, simply ends. The algorithm makes use of the directional
information produced by the classifier to follow the branch from its
nt to the base point, which is then recorded for future use.

3.3 Restoring missing boundary segments

The equation of a line (l) from the base point passing through
the endpoint of the segment is determined as illustrated in Fig. 9.
Next, the point along t
he branch that maximizes the distance
between the branch and the line l is found and is designated as
point A. Point B, which is halfway from point A to the end point,
is then determined. The line starting at point B and passing
through the end point est
imates the direction of the branch.

Note that the direction does not necessarily reflect the direction
(derivative) at the end point. The original estimate is correct if the
segment is itself linear, but as the curvature increases, the estimate
becomes wo
rse. To improve the estimate, the direction is modified
by an amount proportional to the curvature of the segment.

First, a measure of the amount and direction of curvature is
made. If the angle from the horizontal to line l is less than the
angle from
the horizontal to line BA, the segment is defined to
have negative curvature, since it reduces the slope of the estimated
branch direction as illustrated in Fig. 10. Similarly, if the angle to
line l is greater than the angle to line BA, the segment is de
fined to
positive curvature. The amount of curvature C is proportional to
the ratio of the chord height

to the chord length

with the sign
determined using the above definition of positive and negative.
This is a heuristic function that has been experi
determined to provide satisfactory results. The direction of the
segment is then updated using an empirically determined weighted
average of the original estimate and the curvature.

Using this estimate for the direction of the segment, a search
conducted along a linear path starting from the end point of the
segment, about its direction, for connection to other candidate
segments. All vectors starting at the end point and in directions
from (


) to (


) are examined to see if they conta
in any
points with only one neighbor corresponding to the end point of
another segment. The search along each vector is terminated when
either a candidate is found, or the vector reaches a boundary. Note
that here a candidate means either an end point of

another segment
or a vertex. Typically,

should be between 30 and 45 degrees for
best results.

The algorithm identifies all regions where a particle boundary
is incomplete, and locates any possible end points, which can be
joined either to other end po
ints or vertices. The next step is to
examine this data and make the appropriate connections. This is
accomplished by applying the following connection rules in


Rule 1:

Connect all segments that are mutual candidates for
each other where at lea
st one of the segments has only one
candidate, and mark both segments as connected. Such
connections are typically made in regions where a short section
of the boundary pattern is missing in a long shard.


Rule 2:

Connect all remaining segments with only o
candidate, and mark both segments as connected. Typically
such connections are made when a short section of the
boundary is missing, and one of the segments to be joined is a
complex curve, usually having a 'hook' at the very end.


Rule 3:

Connect all r
emaining segments that are mutual
candidates for each other to the candidate closest to the
direction of the current segment, and mark both segments as


Rule 4:

Connect all remaining segments to the candidate closest
to the direction of the curre
nt segment, and mark both segments
as connected if there are any candidates.


Rule 5:

Connect all remaining segments to the vertex closest to
the direction of the current segment, and mark the segment as
connected. Note that any connections made at this po
int are
made to segments for which no other segments were found as
candidates for connection.


Rule 6:

Connect all remaining segments to the boundary pattern
by extending them until they connect with the boundary.

It should be pointed out that the above rul
es only allow one
connection to be made

each segment, but multiple

each segment are allowed. Hence, a segment can

still be a candidate for another connection even if it has already
been connected by a previous rule. It is this mechanism

provides reconstruction of missing vertices.

As will be described later, these rules have been extensively tested
to verify that they provide an accurate restoration of missing
boundary segments. Even when a large portion of a boundary
segment is mi
ssing, reasonable results are obtained. This can be
seen when reconstructing boundary segments that have been
eliminated due to removal of the heater grid lines. Failure to
remove the grid lines results in substantial errors due to excessive

3.4 Heater line correction

This can be seen in Fig. 11 that shows what happens when the
heater grid lines are not removed. Fig. 12 shows the same image
after removal of the heater grid lines. An examination of Fig. 12
clearly reveals that every segment
has been completely

4. User interface

The automatic counting program operates under the Windows
OS as a win32 executable. The user can select to scan a new
image or process an image that has been saved. There is also an
option for pan
els with heater grid lines so that previously described
processing can be performed on them. The program first
determines where the red box is and extracts the data in the region
of the box from the initial image. After compensating for uneven
on, heater grid lines are found (if requested) and the
image filtered to enhance the particle boundaries. The user then
selects the threshold at which to binarize the image. While
adjusting the threshold, the user is free to toggle between the initial
age and the binary image for the current threshold. This allows
the user to select the threshold that achieves the best compromise
between reducing noise and maintaining particle boundaries.

Once the threshold has been selected, the binarized image is
ocessed to restore any missing boundaries and the particles
counted. The program reports the number of particles wholly
within the box, the number of particles on the edge of the box
divided by two and the total number of particles. The image with
the re
stored particle boundaries is displayed and particles wholly
within the box are colored blue, particles, which are on the edge of
the box, are colored yellow and particles which are outside the box
are colored green. At this point, the user can toggle bet
ween any
two of the three images (i.e. the original image, the thresholded
image and the processed image). If the user is unsatisfied with the
processed image, the threshold can be adjusted and the counting
process repeated.

5. Results and conclusions

o studies were conducted to compare the automatic counting
system to manual counting. In both studies, some of the regions
counted had heater grid lines. The feature to eliminate heater grid
lines was not used in either study.

The accuracy of the automa
ted counting system was compared to
manual counting using thirty blueprints. These prints covered a
variety of glass thicknesses and type (tint, solar tint and reflective)
and the prints varied in contrast and overall exposure level. For each
print, high

and low count regions were identified and marked using a
red pen. The number of particles in each region was counted by 5
people and by the automated counting system.

An ANOVA was run comparing the manual counts to the
automatic counts. The F
Ratio of
this test was 0.067, which
represents a significance level of 0.7992. This means that there is no
statistically significant difference between the manual counts and the
automatic counts (this would be indicated by a significance level less
than 0.05). Th
e average difference between the automatic counts
and the average of the manual counts was 0.7%, which corresponds
to an average difference of 2 counts for the average high count of
270 particles.

A similar analysis was run for the low count regions. The

Ratio was 0.695, which corresponds to significance level of 0.4144.
Again, this is not a statistically significant difference. The average
difference between the automatic counts and the average of the
manual counts was 2.4%, which corresponds to an a
difference of 2 counts for the average low count of 94 particles.

The repeatability and reproducibility (R&R) of the automatic
counting system was compared to that for manual counting.
Repeatability refers to the variation one user has when repeat
ing a
measurement on a collection of objects, and reproducibly refers to
the variation between users when examining the same collection of
objects. Two people using both the automatic counting system
examined a set of ten blueprints and manual counting th
e high and
low count regions. The results of the R&R study are shown in the
following table.

Table I. Automatic and Manual Counting Repeatability and Reproducibility
High Count
Low Count
Based on total tolerance

For the high
count regions, both the repeatability and
reproducibility are significantly lower for the automatic counting
with the total

R&R being 28% better than for manual counting.
The R&R for the automatic counter in the low count regions is
25% worse than the manual counting. Most of this difference is in
the repeatability. Some of this difference in the repeatability may
be due to

the operators varying the thresholds more while
attempting to minimize the affect of the heater grid lines which are
in some of the images.

Efforts are continuing to improve the performance of the
system and its ease of use. These include, development o
techniques to automate the threshold selection, provide further
feedback to the operator about the quality of the scanned image,
improve the handling of heater grid lines to prevent them from
being processed as particle boundaries and determine the feasi
of scanning entire prints and automatically identifying the high and
low count regions.


1. Beucher, S (1990) Proc. SPIE Vol. 1350, p. 70
84, Image
Algebra and Morphological Image Processing, PD Gader; Ed.

2. E/ECE (1988)
Agreement concern
ing the Adoption of Uniform
Conditions of Approval and Reciprocal Recognition of
Approval for Motor Vehicle Equipment and Parts. Addendum
42, Regulation No. 43: Uniform Provisions Concerning the
Approval of Safety Glazing and Glazing Material, Annex 5
ruary 1988.

3. Gordon, G (1996) Proc. SPIE Vol. 2665, p. 244
252, Machine
Vision Applications in Industrial Inspection IV, A. R Rao; N
Chang; Eds.

4. Otsu, N (1979) A Threshold Selection Method from Gray
IEEE Transactions on Systems, M
an, and

vol. 9, no. 1, pp. 62
66, 1979

5. Serra, J (1982) Image Analysis and Mathematical Morphology.
Academic Press


Fig. 1. Typical image of shattered glass with heater grid lines

Fig. 2. Hand scanner placed on print adjacent to mar
ked region

Fig. 3. Border extraction to separate the red border from the
predominantly blue print paper. The red component of the color
image is given in the upper left and the blue component is shown
in the upper right image. Subtracting the blue image fr
om the red
image results in the image in the lower
left image. The lower
image is obtained by thresholding the difference image.

Fig. 4. Magnified region of print that illustrates the poor contrast of
some particle boundaries. Noise and shading effec
ts are also

Fig. 5. Small region of scanned print after background subtraction
is on the left. Right image is 3
D plot of image illustrating particle
boundaries or “ridges” that result because of high grayscale values.

Fig. 6. Image after heater
grid removal

Fig. 7. Image after thinning operation

Fig. 8. Two possible situations for boundary restoration. In the
image on the left, a boundary segment is restored by extending it in
the same general direction until it reaches a boundary. For the case
f two discontinuous segments near each other as shown on the
right, the endpoints are simply connected together.

Figure 9. Technique for estimating direction of line segment.

Fig.10. Definition of Negative Curvature.

Fig. 11. Failure to remove heater grid
lines results in obvious
oversegmentation of breakage pattern.

Fig. 12. Restoration of particle boundaries using image from Fig. 7.