Sign Language Translation

Sign Language Translation
System

Group ID: 13
-
008

Introduction



There

are

about

72

million

deaf

people

who

use

sign

language

as

their

first

language

or

mother

tongue
.



It

is

estimated

that

more

than

80
%

of

these

72

million

live

in

developing

countries,

where

authorities

are

rarely

familiar

with

their

needs

or

desires
.



Recent

figures

from

the

British

Deaf

Association

suggest

that

on

any

day

up

to

250
,
000

people

use

some

BSL
.



Our

Concern
:

is

to

interpret

the

BSL,

British

two
-
handed

sign

language

alphabet

and

number

gestures

(plus

others

for

essential

additional

keywords)

Drawbacks of existing systems:



All systems designed for ASL


Capture only one hand


Mostly accessories like wrist bands
and glows are used


Costly equipments like kinect is used


Works under background constraints

Scope of the system:


•
A combination of four components and
convinces two approaches

•

Sign to Text/Voice

•

Image capturing, Enhancement and
gesture recognition

•

Image segmentation and feature
extraction

•

Database and shot segmentation

•

Text/Voice to Sign

•
Convert text to visual sign 3D

FUNCTIONS

Background Subtraction


Used features


Skin color area.


Motion of the hand.


Edges of the background.

Skin color area


HSV color space is being used to detect
the skin color.



Motion of the hand


•
Assumption : Hands are the only objects
which are moving in the interested area.

•
Difference of two consecutive frames is
used to track the motion.



Difference image : foreground objects are
eliminated.






Edges of the background





Laplacian operator is being used to detect
the edges.

Steps Briefly


Skin area is extracted


Take the difference of two consecutive
frames.


Threshold the difference image.



Take the difference of skin area and the
threshold frame.(a)



Take the edges from laplacian operator and
dialate the output.



And the dialated edges with threshold.(b)



And output a and b and then erode it.
This gives the final output of the edge of
the moving hand.


Shot Segmentation


Separation between words using
speed variation

Feature Extraction


Hand

region

recognition

and

finger

tip

detection



•
Sequence

of

convex

points
.

•
ConvexityDefects



defect

points

•
Depth

d

of

the

fingers

•
Calculates

mid

of

the

palm

•
Radius

of

the

palm

•
Angles

between

fingers

using

palm

center,

convex

and

defect

points

Hu moments



Use moments() function in opencv


Returns three types of moments,


Spatial moments,


Central Moments and


Central Normalized Moments


Calculated from Central Moments which are
invariant to size, position and orientation





Circularity



Edge Histogram Descriptor



Edge: important low level feature


Essential for content based image
analysis


16 segments of the image


Describes 5 edge types


Mpeg7 executable

•
Homogeneous Texture Descriptor (HTD)

•
Edge Histogram Descriptor (EHD)




Data Classification using SVM(Support vector
machines)


Becoming more and more popular: for its
easiness

Data format is an issue


The goal of SVM

To produce a model (based on the training
data) which predicts the target values of the
test data







Procedure followed:



Transformed data to the format of an SVM package



Conducted simple scaling on the data
in the range of [0,1]
(The original data maybe too huge or small in range, thus we
can rescale them to the proper range

so that training and
predicting will be faster.)



Used the RBF kernel K(x; y)



Did cross
-
validation to find the best parameter C and
ϒ

using
grid.py


Used the best parameter C and
ϒ

to train the whole
training set


Testing was done

Current situation

Trained SVM for 4 letters in the alphabet A,B,C,D


Alphabet was initially classified as single hand and
both hands in order to increase accuracy.

Ex: Letter c uses only one hand


A,B,C letters are identified

Problems still occur in identifying letter D


Future progress :

Adding up of calculated EHD features to the feature
set



Test to Sign Converter

Development Process


Design 3D model


Rigging and Skinning 3D model


Mapping IK[1] and FK[2]


Animation Design


Programming to join text and animations


1.
Designing 3D models

(Open Source )

2.
Design 3D models


Design 3D models

(Open Source )


Rigging and Skinning 3D model



The best way to animate a complex mesh object
like a character is through the use of Armatures



armature acts like a skeleton: you actually move
the bones of the armature and those bones drive
the animation of the character mesh



The process of building an armature is called
"rigging," and the process of attaching the
armature to a mesh is called "skinning."

Rigging and Skinning 3D model


Test 01


Create simple animation for few bones


Export into different file format

*.fbx

*.x

*.3cd

*.fbx

Implementing 3D model


Microsoft XNA

Microsoft XNA

Test 02

Features

-
3D Modeling

-
3D Animation Designing

-
3D Programming


-
Python


-
MaxScript


3D Max

Disadvantages


-
Have to animate all animation clips in same layer. So
many confused because large time track and have to
remember start and end frame key number of each
animations

Test 03


MonoDeveloper

◦
Open
sourse


-
Java script


-
C#


-
Boo



Unity3D



Powerful Game engine



Possibility of building a executable files for


multiplatform like Android,
iOS
, Web, Windows just
in few second



Macanim

Tools

Generic

Legacy

Available Rigging and


animation technique

Humanoid

Final Implementation

Motion Builder

+ Humanoid Rig

+
Legacy

3. Mapping IK and FK

4. Animation Design


Motion Builder

Very powerful rigging and animation

designing software specially designed

for human character animation


Today widely use in Film industry





Final Output
–

(up
-
to
-
now)

Motion Builder

+
Macanim

+
Legacy

Challenges


Character customization features


Give more Facial expression to the 3D
character

THANK YOU!!!



Q&A