Machine vision on fishing vessels

builderanthologyAI and Robotics

Oct 19, 2013 (4 years and 6 months ago)


Machine vision on fishing vessels
for the identification of fish
species (Digital Observer)

Eric Davis


Dr. Roth (Chair)

Dr. Nance

Dr. Bueler


What is the Digital Observer?

What is my part in the Digital Observer?

What happened over the summer?

Let it end! When will my part be done?


What is the Digital Observer?

What is an Observer?

Records data about what the boat is
catching(species, length, weight).

Records data if the boat or its personnel
are doing anything wrong.

Writes lots and lots of reports.

Difficulty with observers

They take up space on the boat.

They must be paid for by the boat.

Try having someone watching you work
and making notes 24 hours a day.

Every fishing vessel has got a story
about a bad observer.

High turnover in the industry.

Digital Observer to the rescue!

Won’t take up much room on the boat.
Ideally, it will just be a box and a couple
of cameras that are out of the crew’s

Easier to reproduce than humans.

Completely objective data.

Creates the reports automatically.

Only records necessary data.

The Equipment

A Camera

A Computer




Vessel Monitoring Systems

Canada has a VMS system already
created by Archipelago Marine
Research, Ltd.

Record all the fishing activities, as well
as location, to videotape.

The tapes are later audited by humans
to see if the crews report was accurate.

Fish identification abroad

A cannery system has been created
that works at identifying fish.

Takes one fish at a time into a box,
takes pictures of it, and identifies the

How the Digital Observer is

No humans involved except for
identifying problem fish.

Provides a census of all fish, not just a
statistical sample.

Has a distinct lack of environment
control on the boat.

Can provide real
time reporting.

The parts of the Digital Observer.

Image Capture

Camera and computer
work together to get an
image of the fish


Computer determines
which segments of
picture are part of a
fish and which are not.


The fish segments are
measured for
characteristics such as
length, width, and color


The measurements are
sent to a neural
network to identify the
species of the fish.

Image Capture

Written in C, adjustable in number of
frames / second.


The area that I have been researching.

Current product by SciFish is written in
Matlab and works pretty well for chute
images in controlled lighting.

Segmentation example


Gathers data on the segmented fish


Various widths at different points along the fish

Overall Hue

Eccentricity of ellipse that fits the fish.


Neural Network

Takes the morphometric data for each
image and tries to make a best guess
on the best picture.

Works pretty well on a limited sample
test set(about 80% correct).

An example output

Current Group Status

A prototype form of each component
has already been constructed.

Works at a little more than 80%
accuracy for limited in
chute test
data in the lab.

New worries

Variable lighting

Needs to work reliably
in variable conditions

cloudy, sunny,
even at night.

Over the side data(Loss of all
environment control).

Both worries have to do with

Over the side data

Improves counts of fish actually caught in the
lines instead of those hauled and kept on the

Lose any control of picture elements

choice of background along with little light
control. A whole extra level of difficulty.

Need a breakthrough in segmentation to
make this possible(and possibly improve
current software).

What was my part in the Digital

My goals

Evaluation of LEGION.

Implement and evaluate the
segmentation algorithm for the Digital

Evaluation documentation
(segmentation and project).


An oscillator based neural network that has
shown some amazing results in difficult
segmentation problems.

Consists of a smoothing algorithm followed
by the oscillator network algorithm.

Had to evaluate its feasibility for this particular


Implemented against the chute data to
gain familiarity with the project and see
basic feasibility of LEGION.

If successful, would try LEGION to
approach the over
side problem.

LEGION results

Original image

Segmented image


current results.

Has really good segmentation on both
example files and on fish in the chute.

The algorithm had two problems with it
that indicate it won’t work for this

Too slow(by a couple of orders of

Not robust enough in parameters.

Fish Segmentation

4 minutes


No smoothing

Too much smoothing

Smoothing just right

Fish segmentation examples

What I did on my summer

The Cruise

Went on the Kariel for a month to do a test of
the current system created by SciFish.

Also had the objective of gathering data for
where the project should go in the future
(especially over
side data).

While collecting data, I was to evaluate the
process and the equipment being used for
the project (cameras, lighting control, camera
positions, etc.).

The Cruise results

Found no satisfactory solution for
lighting control: daytime, night
time, or

The system didn’t work quite as well as
in the lab tests. Under the best
conditions, we were probably getting
about 50% accuracy.

Gathered hours upon hours of data from
every conceivable angle for future work.

More cruise results

After the cruises I continued in my evaluation
of the equipment used to see if a change of
equipment might afford us better results.

From watching the process of gathering fish
and talking to the crew, it was realized that no
chute camera system would ever replace

Bruises. Lots of bruises.

Say that again? A chute system
won’t work?

In normal fishing
operations, the only
thing that goes through
the chute is the target

To gather information
on what is really
caught, we have to
process some sort of
side data.

Three kinds of over the side data

Stabilizer Camera

Underwater Camera

Rail Camera

Over the side data

All three vantage points have their
advantages as well as disadvantages. The
segmentation is either extremely complex, or
a large amount of control needs to be exerted
on the physical conditions.

This is the angle that is going to be looked at
in Phase II of the Digital Observer project
(beyond the scope of my project).

What I still need to accomplish

More evaluation and documentation of
the segmentation algorithm used.

More evaluation and documentation of
the project as a whole and on future
promising avenues of research for
problems of this type.

Hopefully will be done by mid