ck
dsd.sztaki.hu
DSD
MTA SZTAKI
Department of Distributed Systems
An assistive interpreter tool using glove-based hand
gesture recognition
Péter Mátételki
Máté Pataki
Sándor Turbucz
László Kovács
ck
dsd.sztaki.hu
DSD MTA SZTAKI DSD
MTA founded in 1827
SZTAKI founded in 1968
DSD founded in 1994
pioneers of web
technology in Hungary
main fields
collaboration systems
digital libraries
web accessibility
IoT
crisis management
plagiarism detection
ck
dsd.sztaki.hu
DSD Helen Keller
„Blindness separates us from things, but
deafness separates us from people.”
ck
dsd.sztaki.hu
DSD
Target groups
hearing and speech impaired people
deaf people (native language: sign language) Social problems
problematic social integration
isolated group
To overcome the language barriers
human sign language interpreter
or: a good assistive tool
Challenges
ck
dsd.sztaki.hu
5
DSD
Silent Speech Translation (interACT)
ck
dsd.sztaki.hu
6
DSD
Eyes-Free Keypad Input
ck
dsd.sztaki.hu
7
DSD
Keyboard Glove – University of Alabama
ck
dsd.sztaki.hu
DSD
Our assistive tool
is controlled by sign language (intuitive)
talks for the deaf
everyday use Features
sign language to speech
real time
seamless
Talking Hands
ck
dsd.sztaki.hu
DSD TalkingHands
ck
dsd.sztaki.hu
DSD How does it work?
sign language
custom gestures and text
motion capture with the glove
gesture descriptor stream
signal processing
language processing
text to speech
ck
dsd.sztaki.hu
DSD Glove: gesture capturing
9DOF sensors
signal fusion
absolute position
relative angles
virtual hand
gesture descriptors
30 Hagdil descriptor/sec
Bluetooth
ck
dsd.sztaki.hu
DSD Mobile application: processing
Segmentation algorithm
simple similarity
repetition
unkown descriptors
sliding window
kinetics based algorithm Context-sensitive auto-correction
modified Levenshtein
n-gram (1- & 2-grams)
confusion matrix speech
raw text descriptors
text
ck
dsd.sztaki.hu
DSD Scenarios, situations
ck
dsd.sztaki.hu
DSD Scenarios, situations
work
banking
shopping
public services
healthcare
education
free time
…
ck
dsd.sztaki.hu
DSD Future plans
TalkingHands as a product
two-handed gesture recognition
capture motion dynamics
research in assistive technologies and IoT
robotics
remote manipulation
ck
dsd.sztaki.hu
DSD The project
ck
dsd.sztaki.hu
DSD
Péter Mátételki MTA SZTAKI DSD matetelki@sztaki.hu
http://dsd.sztaki.hu/projects/TolmacsKesztyu
ck
dsd.sztaki.hu
DSD
www.youtube.com/watch?v=NhTUeZ16ZTw
ck
dsd.sztaki.hu
DSD Dactyl fingerspelling alphabet
ck
dsd.sztaki.hu
DSD Hagdil gesture descriptor
ck
dsd.sztaki.hu
