Image based operation for human-robot interaction - IEEE Control ...

fencinghuddleΤεχνίτη Νοημοσύνη και Ρομποτική

14 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

80 εμφανίσεις

Image Based Operation
for
Human-Robot
Interaction
Tomoyuki Hamada, Kohji Kamejima, and lkuo Takeuchi
A major problem in applying robots to
intelligent manufacturing is in the operation
of the robot system in an uncertain environ-
ment. For example, in construction or main-
tenance applications, the environment is
ill-defined because object locations can be
uncertain, and they can change dynamically.
In these situations, it may be better for a
human operator to interactively specify the
robot action at every major step of a task.
This paper describes a prototype system for
human-robot interaction based on the "mental
image" of the task
[
1,2]. The mental image is
a visual model which reflects a real world
situation, such as object placement in the work
space, and simulates the effects of robot action
in this situation. A human operator estimates
and plans the task using this mental image.
This paper presents the system architecture
used to produce the mental image for the
operator.
Information Structure for Robot
Manipulation
The robot motion is determined by an ob-
ject specification in the work space and by an
action specification which gives the effect on
objects.
Object specification is accomplished using
a visual image of the work space created by
the system. To identify the object in the visual
image requires correspondence with the ac-
tual object in the work space, and this is done
by preparing a model of the work space with
an object location estimation process [3].
On the other hand, the action specification
is represented using motion verbs such as
"grasp," "attach" or "put." Although a variety
of actions are required for execution of a
single task, each action is clearly represented
by a sequence of these verbs. Therefore, the
action
is
generated using a procedural se-
Presented at the 1989 IEEE International Con-
ference on Industrial Electronics, Philadelphia, PA,
Nov
6-10,1989. The authors are with the Mechani-
cal Engineering Research Laboratory, Hitachi, Ltd.,
Tsuchiura, Ibaraki
300,
Japan.
quence of primitive action modules cor-
responding to these words. However, the mo-
tion trajectory for the action module depends
upon target object features such as shape and
weight,
so
an adaptive mechanism is needed
to generate a motion appropriate to the target
object. Hence, the model for action is made up
of the procedural sequence of action modules
and an adaptive mechanism of the action
module
[4].
To satisfy both spatial and procedural
demands. the basic information architecture
for the system has two categories; spatial in-
formation and procedural information. These
categories lead to the work space model and
the action model, respectively.
Work Space Model
The work space model design is based on
a frame system where each instance frame
represents an object in the work space. In-
stance frames are structured with three basic
slots: "a-kind-of' "connection" and "loca-
tion."
grasp point GP
Motion Data Perception Part
Execution Part
Space
move
110.50.300.90.0,O
move
110,50,100.90,0,0
~ ~ ~ Q r l ~ 4 0
1
r k l
Model
I
Modification Part
move
110.50,150,90.0.0
a-kind-of
Ca
location
[x,y,z] a-kind-of cap
ig.
1.
Structure
of
motion schema and its interaction with
work
space model.
24
0272-1708/90/1000-0024 $01
.OO
0
1990 IEEE
/€€E
Confro/ Systems Magazine
Fig.
2. Operation
system
overview.
The "a-kind-of' slot indicates object class
which links the frame
to
highly organized
knowledge of the object such as three dimen-
sional shape definition and feature point
definition written in a style of symbolic
description. The feature point represents how
to handle the object in motions such as grasp-
ing or attaching.
Both the "location" slot and the "connec-
tion" slot indicate the spatial relationship
of
the object, but their semantic levels differ.
While the "location" slot determines geomet-
ric location of objects by a relative translation
and rotation matrix, the "connection" slot
determines meaning
of
the spatial relationship
by a symbolic label called the connection
type. Examples of connection types are "on,"
"attached" or "fixed."
A dynamic model matching process is used
to update object location and orientation. Ob-
ject location is estimated by matching the
stored contour image with the corresponding
object contour edge pattern from the camera.
The two dimensional pattern matching
process utilizes the pattern kinetics matching
method
[5],
which recursively calculates the
disparity direction and rotation
(F,r,
F,,
N)
between contour image and camera edge pat-
tern. By shifting and rotating the contour
image in this disparity direction, it can be
matched with the camera edge pattern. One
calculation cycle can be executed in
1/60
second using image processing hardware
developed previously
[3].
Three independent values,
FA, F,
and
N,
are
provided by the two dimensional pattern
matching and indicate the disparity direction
between the contour image and the camera
edge pattern. It is interpreted as a three dimen-
sional object location using the symbolic level
description of the work space model.
For
ex-
ample, the work space description
"box
is on
floor"
produces a constraint which allows
dis-
panty only in certain directions. This sym-
bolic level description of the work space,
called work space context, provides enough
information to extract three dimensional loca-
tion data from a single camera image.
Action
Model
The action model is composed of the action
network which gives the procedural sequence
of action modules and the motion schema
which is the adaptive mechanism of the primi-
tive action module.
For example, the action network translates
an operator message, such as "carry
(cap,
path, body),"
which means "carry the
cap
through the
path
to the
body,"
to a list of
motion schemata. The action network works
like a rewrite rule and expands the operator
message to
a
list of more primitive motions,
until each element of the list reaches the mo-
tion schema in the terminal symbol.
An example of the motion schema is shown
in Fig.
1.
The adaptive mechanism is realized
by describing a motion subset with three in-
ternal subparts: the "perception part," "execu-
tion part" and the "modification part." The
perception part extracts data required for mo-
tion generation from the work space model.
The execution part assembles the data into a
motion sequence. The modification part
modifies the work space model according to
expected change caused by motion effect. The
example in Fig.
1 is
a motion schema for
"grasp." In
this example, required data are
grasp point
GP,
approach point
AP
and grasp
width
W
of the target object
"cap."
The per-
ception part finds descriptions of
GP, AP,
and
W
from the knowledge data about the
"cap."
The execution part calculates coordinates of
GP
and
AP
according to the descriptions and
generates numerical motion data, such as
"move
[10,20,30],
gripper open
..."
Execution
of motion changes the object location and
connection relationship. For the example in
Fig. 1, the modification part of the motion
schema "grasp" modifies the connection slot
value from
"floor,
on"
to
"gripper, fixed,"
and rewrites the location slot value
of "cap."
Prototyping
A prototype of the image based robot
operation system was developed to validate
this method. A system overview is shown in
Fig.
2.
illustrating the two monitor screens.
The screen on the left displays the computer
graphics image of the work spacemodel while
the screen
on
the right displays the camera
image of the work space for interactive map-
ping. The camera itself is shown at the upper
left in Fig.
2.
Because of the work space model and the
dynamic matching, the process of construct-
ing the environment model is greatly
simplified. After specification
of
the object
class name and connection type, the system
can extract object location data from the
camera image. After viewing the camera
image, the operator selects the object class
name from a menu
on
the right side and then
specifies the connection type. At present the
matching process has a problem in local min-
imum matching,
so
the operator must also
specify the approximate location of the object
on the screen image. With this information,
the system automatically extracts actual ob-
ject location data. Objects registered in this
way are displayed with wire-frame
on
the
interactive mapping screen. Every object
registered in the work space model can be
monitored by the system, and when the object
in the real world is moved, the location of the
object image
on
the computer graphic screen
changes automatically.
The operator can make a robot motion plan
based
on
the graphic image and then input the
name of the action plan. The system will
display some reference points such as grasp-
ing points or attachment points concerned
with the action. The operator specifies the
object by selecting one of these reference
points to complete the action message. During
October 1990
25
these instructions, the work space model is
temporarily disconnected from the real world
and the operator can simulate the proposed
plan using the screen image. After this simula-
tion, the system generates the robot motion
data, according to the action message, and
then drives the robot mechanism.
Experiments
Experiments on valve casing assembling
were made to validate the developed
prototype system. The assembling procedure
consists of two steps which are attaching a cap
to the casing body and fixing it with a bolt.
This procedure typically appears in main-
tenance tasks of power plants.
In the experiment, the cap and the bolt
location were initially unknown
to
the system.
They were arbitrarily placed on the floor, and
their locations were registered to the work
space model using the interactive mapping.
Consequently, the robot successfully ex-
ecuted the assembling task. The following are
summaries of the experimental results.
1 ) Several people tried to operate the sys-
tem. The average operation time was about
15
seconds per step, and this did not vary with
the operator who used this system for the first
time. The visual interface with the work space
and action model considerably enhances the
operability of the robot system.
2) Without entering the robot work space,
an operator handled all the steps of the assem-
bling task including the work space model
construction. The system architecture is also
useful for tele-operation for such
as
the space
applications.
3) Using
a
camera one meter from objects,
the dynamic model matching method detects
object location to within
a
one millimeter
error. This is precise enough for the robot to
grasp objects without collision. In the experi-
ment, the 1-mm error was reduced to 0.1 mm
using
a
force sensor attached to the robot
We have demonstrated the feasibility of
using a
model
based approach for the ill-
defined robot tasks, but unknown object han-
dling is
a
remaining problem. In order to
generate motion data, the system needs the
object feature points and these can be deter-
mined from the object shape. Therefore, we
are now trying to integrate
a
shape recognition
process into the system to handle unknown
objects.
gripper.
Conclusions
A prototype. system has been developed for
human-robot interaction based on the "mental
image." By introducing
a
multimodal infor-
mation structure into
a
coherent architecture,
consistency between the system model and
the real world is successfully maintained. Ac-
tion applied to the system model causes the
same effect on the real world, and the change
of the real world is immediately updated by
the camera image. This creates auseful mental
image of the robot task for the human
operator. Through prototyping and
a
series of
experiments, the effectiveness of the method
has been verified.
References
[ I ]
T. Hamada and
K.
Kamejima, "Image based
operation
-
Robot
arm
trajectory generation from
summarized motion planning," in Proc. 3rd SICE
Symp. Human Interface, Oct. 1987, pp. 149-152.
[2] K. Kamejima,
I.
Takeuchi,
T.
Hamada and
Y. C.
Watanabe, "A direct mental image manipulation
approach to interactive robot operation," in Proc.
IEEE Int. Workshop on Intelligent Robots and Sys-
tems (IROS
'88).
Oct. 1988,
pp.
645-650.
[3]
T. Hamada,
K.
Kamejima and
I.
Takeuchi,
"Dynamic work space model matching
for
interac-
tive robot operation," in Proc. IEEE Int. Workshop
on
Industrial Application
of
Machine Intelligence
and Vision (MIV'89), Apr. 1989, pp. 82-87.
[4] T. Hamada, K. Kamejima and
I.
Takeuchi,
"Knowledge representation
for
image based
robot operation," in Proc. IEEElnt. Workshop
on Artificial Intelligence for Industrial
Applications, May 1988,
pp.
417-422.
[SI K. Kamejima,
Y. C.
Ogawa, and Y. Nakano,
"A
fast algorithm
for
approximating 2D diffusion
equation with application to pattern detection in
random image fields," in Proc. IMACSllFAC Int.
Symp. on Modeling and Simulation
of
Distributed
Parameter Systems, Oct. 1987, pp. 149-
156.
1990 IECON
The sixteenth Annual Conference of the
IEEE Industrial Electronics Society will be
held November 2730, 1990, at the Asilomar
Conference Center in Pacific Grove, Califor-
nia. Asilomar is in a beautiful setting by the
sea, adjacent to Monterey. Nearby attractions
include an aquarium,
17
mile drive, Hearst
Castle, and golf courses.
IECON
'90
is an international conference
sponsored by the IEEE Industrial Electronics
Society. The objectives of the conference are
to attract high-quality papers on industrial
electronics and
to
promote professional inter-
actions for the advancement of science, tech-
nology and fellowship. The conference
focuses on industrial applications of electron-
ics, with particular emphasis on the use of
electronic and computer technologies in the
industrial environment. IECON '90 will have
four
conference themes: 1) PowerElectronics,
2) Signal Processing and System Control, 3)
Factory Automation, and
4)
Emerging Tech-
nologies. For additional information, contact
any of the following:
AT&T Bell Laboratories
P.O. Box
400, Room HOH R-222
Holmdel, NJ 07733
(201) 888-7264 (Office)
(201) 888-7074 (Fax)
General Chairman
Mr. Robert Begun
23609 Skyview Terrace
Los
Gatos, CA 95030- 1560
(408) 353-1560
Registration Chairman:
G.J.
Qua
Now is
also
the time
to
prepare for the
1991 conference: IECON '91, October 28-
November
1,
1991, Kobe, Japan. Contact:
Professor Hiro Haneda, Department of
Electronics Engineering, Kobe University,
Rokko-dai, Nada-Ku, Kobe City, Hyogo 657,
Japan; Telephone: 81-78-881-1212, Fax: 81-
78-861-7679.
26
/€€E
Control Systems Magazine