MS Word tutorial - Clemson University

unclesamnorweiganΤεχνίτη Νοημοσύνη και Ρομποτική

18 Οκτ 2013 (πριν από 3 χρόνια και 2 μήνες)

75 εμφανίσεις

ECE 495 Fall 2010

Clemson University








Ninad Pradhan

Overview of Image Processing Software Options


Two
Options

The requirement is to read an image from the camera on the laptop. This image is processed, game
logic is calculated, and a decision is communicated to the xPC target using UDP communication. It is
possible to do this using two combinations of software
,
on
ly
one of which needs to be selected

by
groups for their implementation
:



Option 1
:
OpenCV (in C++)
-
> xPC Simulink program.



Option 2
:
MATLAB Image Acquisition and processing
-
> xPC Simulink program.

Option 1
-

OpenCV/xPC System















In

this setup, image processing and game logic
are

carried out in a Visual C++ Solution, which uses the
OpenCV library. OpenCV

is an extensive library of image processing and computer vision functions
with an interface that can easily be used in C++. We use UDP functions written for C++
(author: Peng
Xu) for communicating with the target PC.

Runtime Procedure:



Build the xPC progr
am inside Simulink.



Connect to the Target from
xpcexplr
.



Run the C++ Program using
F7

(Compile) and
Ctrl+F5

(Execute).



Run the xPC program using
xpcexplr
.




Came
ra

Image
process
ing using
OpenCV/C++


Game logic
C++

Laptop

C++ UDP

TCP/IP for
xpcexplr

xpcexplr


xPC Target PC:
MATLAB

Simulink

Robot
Controller with
Q4 Interface


Q4

Robot

Laptop C++
program

Target
PC
Simulink model


Option 2 :
MATLAB Image Processing
/xPC System

















In

the MATLAB
Image Acquisition

setup, image processing and game logic is carried out in MATLAB
Simulink program which runs in parallel with the xPC Target program on the host PC. MATLAB’s Image
Acquisition Toolbox is used to get images from the camera. The laptop progr
am also uses a toolbox
called the Real
-
Time Blockset, which is free to download and use. It is used to approximately
synchronize the laptop and the xPC target Simulink models. UDP communication in MATLAB is used to
send game commands to the xPC target.

Ru
ntime Procedure:



Build the xPC program inside Simulink.



Connect to the Target from
xpcexplr
.



Run the host (laptop) Simulink model.



Run the xPC model using
xpcexplr
.

Comparing Image Processing in MATLAB and Simulink

OpenCV and MATLAB image processing funct
ions are closely related because they were written for
the purpose of providing easy access to computer vision algorithms. Both are equally well
-
accepted in
industry and academia. MATLAB image processing is slower and is preferred more as a prototyping
sof
tware than as the final product. OpenCV/C++ can do real
-
time image processing depending on the
functions being called

and is more widely used in applications
.

Here are some examples to compare
functions across the two platforms:

Functionality

MATLAB

Functi
on

OpenCV

Function

Create variables to store images

Stores to a matrix as default.

cvCreateImage

Convert RGB to grayscale image

rgb2gray

cvCvtColor

Get RGB Channels of image

Matrix operations

cvCvtPixToPlane

Threshold the image

im2bw

cvThreshold

Morphological Operations

imerode, imdilate

cvErode, cvDilate

Image Display

imshow

cvShowImage

Add two images

Matrix addition

cvAdd

Region
-
based processing

The regionprops structure

The cvBlob library

Came
ra

MATLAB Image

processing


Game logic

Laptop

xPC
UDP

TCP/IP for
xpcexplr

xpcexplr


xPC Target PC:
MATLAB

Simulink

Robot
Controller with
Q4 Interface


Q4

Robot

Host PC
Simulink model


Target
PC
Simulink model




Choosing between C++ and MATLAB Image Processing



MATLAB route requires learning to use user
-
defined Simulink blocks called S
-
Functions.



Using OpenCV requires implementing game logic using C++ code and debugging in the C++
environment.



Both have similar image processing capabilities. However, MATLAB ha
s a readily available
region processing function, which will take an extra step or two to accomplish in OpenCV.



Both programming options are equally easy to use. An important factor determining your final
choice could be your personal preference for C++ or

MATLAB programming.


Next Section: Image Processing in OpenCV




Image Processing in OpenCV


Software

The following software tools used to communicate camera data to the xPC Target might be new to
ECE 495 Groups:



OpenCV



Open Source Image Processing/Computer Vision Library for C++.



UDP Communication using C++



xPC UDP Communication Blocks


Overview

of the System

The camera is interfaced to the laptop using image acquisition functions in C++. OpenCV is an
extensive image
processing library which enables image processing. The output of the image
processing functions is used to calculate the next strategic move for the robot, in the form of a control
command. This command is sent to the xPC Target PC over the UDP communicati
on channel. The
Target PC program received this control command and the position control system sends a control
signal out to the robot via the Q4.
Look at the last section ‘
Region processing in OpenCV
’ to know
about identifying the game state.


OpenCV

Download and Installation


Step 1


If

you haven’t done so already,

d
ow
nload and install Visual C++ 2008/2010 Express from
Microsoft:


http://www.microsoft.com/express/downloads/

Step 2


Download

OpenCV 2.1 Installer from the project’s SourceForge page.


http://sourceforge.net/projects/opencvlibrary/files/opencv
-
win/2.1/OpenCV
-
2
.1.0
-
win32
-
vs2008.exe/download

Step 3



Run the

.exe file to install the library to your laptop. Again, it is recommended that you install to
C:
\
OpenCV2.1

for provided examples to work without modification.

Step 4


OpenCV

will work with Visual C++ 2008 and 2010. Follow the instructions at the following link
exactly

to avoid Compiler/Linker errors. Create the Hello World program as explained in this link. Note
that, before the
Configure Project Directories

(second) step,
co
mpile

your program once (F7).


http://opencv.willowgarage.com/wiki/VisualC%2B%2B_VS2010

Step 5


This step may not be applicable to those who did not get a Linker error related to
lib
cd.lib

in
Step 4.
Libcd.lib


is not a required library for OpenCV projects but returned Linker errors when I tried
to compile a program (more complex than the Hello World above) on my machine.



Go to Project > “projectname” Properties and select
All Config
urations
at the top.



In the left panel, select Configuration Properties > Linker > Input and look at ‘
Ignore Specific
Default Libraries
’.



Add the name
libcd.lib;
to the list.


Some OpenCV jargon for reference



c
v
_____ : OpenCV

functions are defined starting with ‘cv’. For example, image thresholding is
done using the command
cvThreshold
.



Cv____ : OpenCV’s customized variables are defined starting with ‘Cv’. For example, the
variable which contains camera information for grabbin
g frames off it is called
CvCapture
.



IplImage * : This is OpenCV’s image structure. Just as MATLAB requires a matrix to store
images for processing, OpenCV requires the use of IplImage*.


Setting up the Webcam using OpenCV

Use the provided C++ Solution
(SLN file) as a reference for understanding the steps explained here.
Use the Solution corresponding to your version of Visual C++ Express. Some of the commands used in
that Solution belong to the UDP communication setup and will be explained later.

Captu
ring frames from the webcam

Where: process_camera_frame.cpp

Capturing frames from the camera is a two step process in OpenCV, analogous to MATLAB.



The first step is to identify whether a device (webcam) is connected to the laptop. This is done
by calling

the
cvCaptureFromCAM(0)
function, which identifies the camera connected to the
laptop and connects to it.



The second step is to get the latest frame (or image) from that camera. This is done by calling
frame =
cvQueryFrame(capture),

which captures a frame and stores it to the IplImage
variable
frame
.

Image capture can be used as defined in the example. No modifications are needed to the above
steps.

Modification needed: No.


Calibrating the camera

Where: process_camera_frame.cpp
, pro
cess_camera_frame.h

The function
calibrateCamera()
called i
n

the CPP file is defined

in the .H file.



First, we define a function called
calibrateClicks()

which allows the program to process
mouse clicks.



The first LEFT mouse button click corresponds to
the TOP
-
LEFT corner of the board.



The first RIGHT mouse button click corresponds to the BOTTOM
-
RIGHT corner of the board.



These values are stored in the variables
top_left
and
bottom_right
.



Use the function
calibrateCamera()

to set grid dimensions based on

the mouse click
inputs.


Modification needed:
Yes, to calibrate individual cells in the grid
.



Test Run 1

Run a test to see whether the Solution compiles successfully.



Connect the webcam.



F7 to compile the program.



Ctrl+F5 to run the program.



In the
window which shows an image, LEFT click to get the top
-
left corner. Hit Enter.



In the same window, RIGHT click to get bottom
-
right corner. Hit Enter.



The program continues to run in a manner explained in the following section. Stop
execution by closing th
e console window.



Implementing Image Processing Logic using OpenCV

Now

that the webcam has been set up correctly, various image processing operations can be carried
out on the image. Here’s a short list of image processing operations implemented in the e
xample and
their MATLAB analogs.

Detailed references for these functions can be obtained at:

http://opencv.willowgarage.com/documentation/cv._image_processing_and_computer_vision.html

Create images

Where: process_camera_frame.cpp
, process_camera_frame.h

cvCreateImage



This function is used to define the size and type of an image. Size is set to th
e same as the
webcam frame in our example, and type is set to grayscale (
IPL_DEPTH_8U
).


In MATLAB
:
none, since MATLAB assigns images to matrices internally.


Convert RGB to Grayscale

Where:

process_camera_frame.h > imageProcessing()

cv
CvtColor



Converts the color of an image from RGB to grayscale.



The specifier CV_BGR2GRAY specifies the kind of format conversion desired.


In MATLAB
:

rgb2gray


Get RGB Channels of a Color Image

Where:

process_camera_frame.h > imageProcessing()

cv
CvtPixToPlane



In a single command, individual channels of an RGB image are isolated.


In MATLAB
:

red =
rgb_image(:,:,1); green = rgb_image(:,:,2); blue = rgb_image(:,:,1);




Threshold the Grayscale Image

Where:

process_camera_frame.h > imageProcessing()

cvThreshold



If the threshold is user defined, then the specifier
CV_THRESH_BINARY

should be specified.



If the threshold is automatically calculated, then
CV_THRESH_
OTSU should be specified.


In MATLAB
:

im2bw or graythresh + im2bw (for automatic thresholding).


Morphological Operations

Where:

process_camera_frame.h > imageProcessing()

cvErode



Erosion and dilation (
cvDilate
) can be carried out using a standard structuring element (by
using NULL instead of
strEl
).



To specify a structuring element, the function
cvCreateStructuringElementEx

is used.


In MATLAB
:

imerode or strel+imerode to specify structuring element.


Display Results

Where:

process_camera_frame.cpp

cvShowImage



The window in which an image should be displayed is created using
cvNamedWindow.



The function
cvShowImage

is called to display an image to that window.


In MATLAB
:

figure; imshow();


Test Run 2

Run a test to see whether the Solution performs specified image processing functions.



Connect the webcam.



F7 to compile the program.



Ctrl+F5

to run the program.



Calibrate the camera as per Test Run 1.



See that the left window displays the unfiltered webcam input and the right windows shows
a modified binary image.


UDP
Communication using xPC

The provided simulations may be used as a template for the final project. For details, please see the
User’s Guide at
http://www.mathworks.com/help/toolbox/xpc/xpc_product_page
.html

In general, UDP is a data communication protocol which does not require handshaking between two
computers for data to be shared, and xPC provides an interface to it. The following things need to be
noted when using UDP:



Sample time on UDP Send/Receiv
e blocks should match the xPC target program Fixed Step
Time.



Output port width needs to be specified in the UDP Receive block. This is in terms of number
of bytes and should match the number of bytes coming into the corresponding UDP Send
block.



Port dime
nsions on corresponding Pack and Unpack blocks should match.



IP addresses to which data is being sent and from where it is being received should be set
correctly in the UDP blocks.

All these settings have been tested with the simulations provided on the we
bsite. However, a
different number of inputs will require some settings to be changed based on the above
requirements.


Test Run 3

Run a test to see whether the complete software works as expected.



Connect a spare webcam. Do not connect the game camera
if you intend to move it for
testing.



Compile target model in Simulink.



Compile and Run the camera program as in Test 2.



Play target program using xPC Explorer.



Move the camera so that it points to different regions of the room. The detected intensity
thre
shold should change and the change should reflect in the xPC Scope.


Region Processing in OpenCV

We use a sophisticated OpenCV
-
based library called
cvBlob

to perform region based image processing
in OpenCV
, which can be used to get the current board state. For users familiar with MATLAB image
processing, this library performs the same function as
regionprops
, with more functions for region
filtering.

To use this library

(already done in example SLN)
:



Copy and paste all the files from the folder
specific to your version of Visual C++ Express

to
your project folder where other .cpp and .h files are located
.



Add
cvblob.cpp

and
cvblob.h

to your C++ Solution (by opening the SLN inside Visual C++
Express).




Include the
cvblob.h

file and the
cvBlob namespace

(as shown in process_camera_frame.h).

The provided documentation files (in the
cvBlob doc

folder) list various functions contained within the
cvBlob library. Look at
process_camera_frame.h > imageProcessin
g()
to see the usage of
one of these functions:
cvFilterByArea()
.

To see more examples of usage, visit the cvBlob webpage and
download the Source

(_src) ZIP file. This
contains some
sample

and
test

code in folders with the same name.


Next Section: Image Processing in
MATLAB



MATLAB Tools

The following tools used to communicate camera data to the xPC Target might be new to ECE 495
Groups:



MATLAB Image Acquisition Toolbox



xPC

UDP Communication Blocks



Real
-
Time Blockset 7.1 for Simulink



MATLAB S
-
Functions


Overview
of the System

In

the MATLAB Image Acquisition setup, image processing and game logic is carried out in MATLAB
Simulink program which runs in parallel with the xPC

Target program on the host PC. MATLAB’s Image
Acquisition Toolbox is used to get images from the camera. The laptop program also uses a toolbox
called the Real
-
Time Blockset, which is free to download and use. It is used to approximately
synchronize the l
aptop and the xPC target Simulink models. UDP communication in MATLAB is used to
send game commands to the xPC target.

Two Simulations

In brief
: Groups will be running two simulations in parallel. On the laptop, the Image Acquisition
Toolbox will be used

to grab camera frames, process them, and give a control set point as the output.
This host simulation will communicate over the xPC UDP channel with the simulation running on the
target PC. The target simulation accepts set point commands from the host, r
uns a controller (e.g. PID)
to process them and controls the shooting mechanism via the Q4 card.


In Detail:
A complete system will require two Simulink programs (block diagrams) executing in
parallel on two different computers.
On the laptop (host PC)
, a

Simulink program will be built that
contains the RTBlockset (see description below) and most of your game logic. The RTBlockset will try
to force the execution of your Windows
-
based program at a fixed sample rate; however, since this
sample frequency can
not be fully guaranteed this is a
“soft real
-
time”

execution of your code. On the
xPC Target
, which we know implements
“hard real
-
time
” execution of your code, you will implement
functions that require real
-
time execution or access to the Quanser Q4 board
.

You have to decide which functions should be executed on the
“soft real
-
time”

Host PC and
which on the
“hard real
-
time
” xPC Target machine. Generally functions such as game logic (where it
would be fine to make a decision in seconds rather than millise
conds) can be executed on the Host PC.
Examples include a user interface or a game strategy algorithm. For example, you would use the
Image Acquisition Toolbox to grab camera frames, process them, and give a control set point as the
output. Note that som
e Simulink functions cannot be compiled for the xPC Target and so you could
use these functions on the Host version of Simulink and then send the results to the XPC Target
program. Operations that are time critical, such as implementing closed
-
loop contro
l of a motor will
be run on the Target PC. The only way to access the Quanser Q4 board inputs and outputs is through
the Target PC program.

In order to send data between the executing
“soft real
-
time”

Host PC program and the executing
“hard real
-
time”

xPC
Target program you need to establish a communications channel. The hardware
channel is through Ethernet cable and network cards linking the two computers. The UDP
communications protocol can be used to send data between the two programs over this hardware

channel. For example, the target simulation could accept set
-
point commands from the host, run a
controller (e.g. PID) to process them, and control the shooting mechanism via the Q4 card. Note that
on this same hardware channel, a built
-
in function of
xPC is using
TCP/IP
communications to exchange
data between the executing xPC Target program and the target viewer (
xpcexplr
.exe) running under
Windows on you Host PC.


Runtime P
rocedure



Compile

target simulation using
Tools > Real Time Workshop > Build
Model
.



Open
xpcexplr

and connect to target.



Play the host simulation using the Play button or
Simulation > Start
.



Play the target simulation from xPC Explorer.




S
-
Functions

In the supporting MDL files provided with this document, note that the laptop (host) simulation has a
block called a Level 2 S
-
Function block. S
-
Functions are Simulink blocks which contain user
-
written
code with a customizable number of inputs, outputs, pa
rameters. The example simulation has an S
-
function which processes a camera frame at every rising edge of a pulse train, calculates and outputs a
dummy control set point, and holds the previous set point value until a new one needs to be
calculated.

Refer
the MATLAB website (
http://www.mathworks.com/help/toolbox/simulink/sfg/f7
-
67622.html
)
for details on writing S
-
functions. Also, open the Simulink library inside MATLAB and go
to

Simulink
> User Defined Functions > S
-
Function Examples > M
-
File S
-
Functions > Level
2 M
-
File S
-
Functions
to see examples.

Note that S
-
functions can be added to the target, but a special, considerably complex procedure
needs to be carried out to compile

S
-
functions with xPC. Including a S
-
function in the xPC target
simulation is
not recommended
.


MATLAB Image Acquisition Toolbox

In the S
-
Function
process_camera_frame.m
on the host simulation, the Image Acquisition Toolbox
is used to capture a camera
frame. Beyond this, image processing commands from MATLAB can be
used. Note that:



vid = videoinput(
'winvideo'
, 1);

cr
eates a handle to the camera connected to
your laptop.



getsnapshot(vid);
grabs the current frame from the webcam.



delete(vid);clear
vid
;
are necessary to prevent memory leaks or else the simulation
will generate an error after a while.

More about the toolbox here:
http://www.mathworks.com/help/toolbox/imaq/


xPC UDP blocks

The provided simulations may be used as a template for the final project. For details, please see the
User’s Guide at
http://www.mathworks.com/help/toolbox/xpc/xpc_product_page
.html

In general, UDP is a data communication protocol which does not require handshaking between two
computers for data to be shared, and xPC provides an interface to it. The following things need to be
noted when using UDP:



Sample time on UDP Send/Receiv
e blocks should match the xPC target simulation Fixed Step
Time.



Output port width needs to be specified in the UDP Receive block. This is in terms of number
of bytes and should match the number of bytes coming into the corresponding UDP Send
block.



Port d
imensions on corresponding Pack and Unpack blocks should match.



IP addresses to which data is being sent and from where it is being received should be set
correctly in the UDP blocks.

All these settings have been tested with the simulations provided on the

website. However, a
different number of inputs will require some settings to be changed based on the above
requirements.


Real
-
Time Blockset

RTBlockset is a toolbox written to assign high priority to the host simulation to give it soft real
-
time
character
istics (real
-
time with a margin of error). Download the toolbox from
http://www.mathworks.com/matlabcentral/fileexchange/3175
.

Unzip it to either the local folder (where your MDL file
s are stored) or to the MATLAB ‘toolboxes’
folder located in the installation folder (e.g.
C > Program Files > MATLAB
). If you do the latter,
add the Blockset to your MATLAB path (using
MATLAB > File > Set Path > Add Folder …
).
This will allow you to use
the toolbox in any simulation located anywhere on your laptop.

RTBlock
, the Simulink block for this toolbox, needs to be part of the laptop simulation. The
timestep

for the block needs to match your xPC target simulation fixed step. The
priority

should
be set to REAL
-
TIME.


Downloads



Host simulation



Target simulation



Real
-
Time Blockset.