Sign Language Translation
System
Group ID: 13
-
008
Introduction
There
are
about
72
million
deaf
people
who
use
sign
language
as
their
first
language
or
mother
tongue
.
It
is
estimated
that
more
than
80
%
of
these
72
million
live
in
developing
countries,
where
authorities
are
rarely
familiar
with
their
needs
or
desires
.
Recent
figures
from
the
British
Deaf
Association
suggest
that
on
any
day
up
to
250
,
000
people
use
some
BSL
.
Our
Concern
:
is
to
interpret
the
BSL,
British
two
-
handed
sign
language
alphabet
and
number
gestures
(plus
others
for
essential
additional
keywords)
Drawbacks of existing systems:
All systems designed for ASL
Capture only one hand
Mostly accessories like wrist bands
and glows are used
Costly equipments like kinect is used
Works under background constraints
Scope of the system:
•
A combination of four components and
convinces two approaches
•
Sign to Text/Voice
•
Image capturing, Enhancement and
gesture recognition
•
Image segmentation and feature
extraction
•
Database and shot segmentation
•
Text/Voice to Sign
•
Convert text to visual sign 3D
FUNCTIONS
Background Subtraction
Used features
Skin color area.
Motion of the hand.
Edges of the background.
Skin color area
HSV color space is being used to detect
the skin color.
Motion of the hand
•
Assumption : Hands are the only objects
which are moving in the interested area.
•
Difference of two consecutive frames is
used to track the motion.
Difference image : foreground objects are
eliminated.
Edges of the background
Laplacian operator is being used to detect
the edges.
Steps Briefly
Skin area is extracted
Take the difference of two consecutive
frames.
Threshold the difference image.
Take the difference of skin area and the
threshold frame.(a)
Take the edges from laplacian operator and
dialate the output.
And the dialated edges with threshold.(b)
And output a and b and then erode it.
This gives the final output of the edge of
the moving hand.
Shot Segmentation
Separation between words using
speed variation
Feature Extraction
Hand
region
recognition
and
finger
tip
detection
•
Sequence
of
convex
points
.
•
ConvexityDefects
defect
points
•
Depth
d
of
the
fingers
•
Calculates
mid
of
the
palm
•
Radius
of
the
palm
•
Angles
between
fingers
using
palm
center,
convex
and
defect
points
Hu moments
Use moments() function in opencv
Returns three types of moments,
Spatial moments,
Central Moments and
Central Normalized Moments
Calculated from Central Moments which are
invariant to size, position and orientation
Circularity
Edge Histogram Descriptor
Edge: important low level feature
Essential for content based image
analysis
16 segments of the image
Describes 5 edge types
Mpeg7 executable
•
Homogeneous Texture Descriptor (HTD)
•
Edge Histogram Descriptor (EHD)
Data Classification using SVM(Support vector
machines)
Becoming more and more popular: for its
easiness
Data format is an issue
The goal of SVM
To produce a model (based on the training
data) which predicts the target values of the
test data
Procedure followed:
Transformed data to the format of an SVM package
Conducted simple scaling on the data
in the range of [0,1]
(The original data maybe too huge or small in range, thus we
can rescale them to the proper range
so that training and
predicting will be faster.)
Used the RBF kernel K(x; y)
Did cross
-
validation to find the best parameter C and
ϒ
using
grid.py
Used the best parameter C and
ϒ
to train the whole
training set
Testing was done
Current situation
Trained SVM for 4 letters in the alphabet A,B,C,D
Alphabet was initially classified as single hand and
both hands in order to increase accuracy.
Ex: Letter c uses only one hand
A,B,C letters are identified
Problems still occur in identifying letter D
Future progress :
Adding up of calculated EHD features to the feature
set
Test to Sign Converter
Development Process
Design 3D model
Rigging and Skinning 3D model
Mapping IK[1] and FK[2]
Animation Design
Programming to join text and animations
1.
Designing 3D models
(Open Source )
2.
Design 3D models
Design 3D models
(Open Source )
Rigging and Skinning 3D model
The best way to animate a complex mesh object
like a character is through the use of Armatures
armature acts like a skeleton: you actually move
the bones of the armature and those bones drive
the animation of the character mesh
The process of building an armature is called
"rigging," and the process of attaching the
armature to a mesh is called "skinning."
Rigging and Skinning 3D model
Test 01
Create simple animation for few bones
Export into different file format
*.fbx
*.x
*.3cd
*.fbx
Implementing 3D model
Microsoft XNA
Microsoft XNA
Test 02
Features
-
3D Modeling
-
3D Animation Designing
-
3D Programming
-
Python
-
MaxScript
3D Max
Disadvantages
-
Have to animate all animation clips in same layer. So
many confused because large time track and have to
remember start and end frame key number of each
animations
Test 03
MonoDeveloper
◦
Open
sourse
-
Java script
-
C#
-
Boo
Unity3D
Powerful Game engine
Possibility of building a executable files for
multiplatform like Android,
iOS
, Web, Windows just
in few second
Macanim
Tools
Generic
Legacy
Available Rigging and
animation technique
Humanoid
Final Implementation
Motion Builder
+ Humanoid Rig
+
Legacy
3. Mapping IK and FK
4. Animation Design
Motion Builder
Very powerful rigging and animation
designing software specially designed
for human character animation
Today widely use in Film industry
Final Output
–
(up
-
to
-
now)
Motion Builder
+
Macanim
+
Legacy
Challenges
Character customization features
Give more Facial expression to the 3D
character
THANK YOU!!!
Q&A
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Comments 0
Log in to post a comment