Sign Language Translation

minedesertSoftware and s/w Development

Oct 31, 2013 (3 years and 1 month ago)

60 views

Sign Language Translation
System

Group ID: 13
-
008

Introduction



There

are

about

72

million

deaf

people

who

use

sign

language

as

their

first

language

or

mother

tongue
.



It

is

estimated

that

more

than

80
%

of

these

72

million

live

in

developing

countries,

where

authorities

are

rarely

familiar

with

their

needs

or

desires
.



Recent

figures

from

the

British

Deaf

Association

suggest

that

on

any

day

up

to

250
,
000

people

use

some

BSL
.



Our

Concern
:

is

to

interpret

the

BSL,

British

two
-
handed

sign

language

alphabet

and

number

gestures

(plus

others

for

essential

additional

keywords)

Drawbacks of existing systems:



All systems designed for ASL


Capture only one hand


Mostly accessories like wrist bands
and glows are used


Costly equipments like kinect is used


Works under background constraints

Scope of the system:



A combination of four components and
convinces two approaches



Sign to Text/Voice



Image capturing, Enhancement and
gesture recognition



Image segmentation and feature
extraction



Database and shot segmentation



Text/Voice to Sign


Convert text to visual sign 3D

FUNCTIONS

Background Subtraction


Used features


Skin color area.


Motion of the hand.


Edges of the background.

Skin color area


HSV color space is being used to detect
the skin color.



Motion of the hand



Assumption : Hands are the only objects
which are moving in the interested area.


Difference of two consecutive frames is
used to track the motion.



Difference image : foreground objects are
eliminated.






Edges of the background





Laplacian operator is being used to detect
the edges.

Steps Briefly


Skin area is extracted


Take the difference of two consecutive
frames.


Threshold the difference image.



Take the difference of skin area and the
threshold frame.(a)



Take the edges from laplacian operator and
dialate the output.



And the dialated edges with threshold.(b)



And output a and b and then erode it.
This gives the final output of the edge of
the moving hand.


Shot Segmentation


Separation between words using
speed variation

Feature Extraction


Hand

region

recognition

and

finger

tip

detection




Sequence

of

convex

points
.


ConvexityDefects



defect

points


Depth

d

of

the

fingers


Calculates

mid

of

the

palm


Radius

of

the

palm


Angles

between

fingers

using

palm

center,

convex

and

defect

points

Hu moments



Use moments() function in opencv


Returns three types of moments,


Spatial moments,


Central Moments and


Central Normalized Moments


Calculated from Central Moments which are
invariant to size, position and orientation





Circularity



Edge Histogram Descriptor



Edge: important low level feature


Essential for content based image
analysis


16 segments of the image


Describes 5 edge types


Mpeg7 executable


Homogeneous Texture Descriptor (HTD)


Edge Histogram Descriptor (EHD)




Data Classification using SVM(Support vector
machines)


Becoming more and more popular: for its
easiness

Data format is an issue


The goal of SVM

To produce a model (based on the training
data) which predicts the target values of the
test data







Procedure followed:



Transformed data to the format of an SVM package



Conducted simple scaling on the data
in the range of [0,1]
(The original data maybe too huge or small in range, thus we
can rescale them to the proper range

so that training and
predicting will be faster.)



Used the RBF kernel K(x; y)



Did cross
-
validation to find the best parameter C and
ϒ

using
grid.py


Used the best parameter C and
ϒ

to train the whole
training set


Testing was done

Current situation

Trained SVM for 4 letters in the alphabet A,B,C,D


Alphabet was initially classified as single hand and
both hands in order to increase accuracy.

Ex: Letter c uses only one hand


A,B,C letters are identified

Problems still occur in identifying letter D


Future progress :

Adding up of calculated EHD features to the feature
set



Test to Sign Converter

Development Process


Design 3D model


Rigging and Skinning 3D model


Mapping IK[1] and FK[2]


Animation Design


Programming to join text and animations


1.
Designing 3D models

(Open Source )

2.
Design 3D models


Design 3D models

(Open Source )


Rigging and Skinning 3D model



The best way to animate a complex mesh object
like a character is through the use of Armatures



armature acts like a skeleton: you actually move
the bones of the armature and those bones drive
the animation of the character mesh



The process of building an armature is called
"rigging," and the process of attaching the
armature to a mesh is called "skinning."

Rigging and Skinning 3D model


Test 01


Create simple animation for few bones


Export into different file format

*.fbx

*.x

*.3cd

*.fbx

Implementing 3D model


Microsoft XNA

Microsoft XNA

Test 02

Features

-
3D Modeling

-
3D Animation Designing

-
3D Programming


-
Python


-
MaxScript


3D Max

Disadvantages


-
Have to animate all animation clips in same layer. So
many confused because large time track and have to
remember start and end frame key number of each
animations

Test 03


MonoDeveloper


Open
sourse


-
Java script


-
C#


-
Boo



Unity3D



Powerful Game engine



Possibility of building a executable files for


multiplatform like Android,
iOS
, Web, Windows just
in few second



Macanim

Tools

Generic

Legacy

Available Rigging and


animation technique

Humanoid

Final Implementation

Motion Builder

+ Humanoid Rig

+
Legacy

3. Mapping IK and FK

4. Animation Design


Motion Builder

Very powerful rigging and animation

designing software specially designed

for human character animation


Today widely use in Film industry





Final Output


(up
-
to
-
now)

Motion Builder

+
Macanim

+
Legacy

Challenges


Character customization features


Give more Facial expression to the 3D
character

THANK YOU!!!



Q&A