EASYTALK : Sri Lankan Sign Language Translator

Research Scope

This section contains domain details of EasyTalk

LITERATURE SURVEY

Researches related to sign languages are done around the world. Chinese sign language, American sign language, and Sinhala are some of them. These sign language detections have two approaches. One is image or vision-based and the other one is device-based. In general, the identification of signs includes pre-processing, extraction of features, and classification in a simple phase. Pre-processing is done in the sense to remove an unwanted element from the context. From the Region of Interest (ROI), image feature will be extracted, and it will be classified using a certain method.

When talking about device-based sign recognition, a system is proposed by Gibran & team for Mexican sign language using Kinect sensor. The system will store the colour, depth and the skeleton tracking information using the RGB-D camera. Dynamic Time Wrapping (DTW) algorithm had been used in this system to interpret gestures.

Haar Cascade Classifier has shown 92.68% accuracy in Kanchan & Surekha’s work. They have interpreted Indian sign language using webcam images. However, some images are not classified because of high motion gestures. For the same sign language, Probabilistic Neural Network (PNN) classifier has shown more accuracy than K-Nearest Neighbor (KNN) classifier.

RESEARCH GAP

The first reason is Sri Lanka still listed as a developing country. The tools & technologies which were used to build such systems by other countries are very expensive nowadays. Sri Lankan government cannot be able to spend that amount of money for only one main purpose. The second reason is that even we buy one of those system, people cannot get the benefits of the system as each country has their own sign language.

Another reason is that, the system which already implemented for a specific language such as Sinhala. Due to this reason, if any person who doesn’t have a proper practice in that language won’t understand the system. Therefore,by-today the ordinary people who are willing to communicate with hearing-impaired & mute people have challenged with a huge communicating gap in their daily lives.

RESEARCH PROBLEM

Sri Lanka relies on tourism and education for its growth.
Reliable communication between every citizen is key for country’s growth
The verbal and hearing impaired need a reliable and an easy go-to solution.
Such a translating solution should increase such person’s value

OBJECTIVES

MAIN OBJECTIVE

To improve the communication bond between the ordinary people & people with difficulties in hearing & speaking. To successfully fulfil the above idea, this system should be able to convert SSL into text formator audio format & converting English text format into SSL using low-resolution web application.

SPECIFIC OBJECTIVES

To capture the input images using low-resolution web cameras in real-time.
To use ML algorithms effectively for detected sign classification
To display the detected hand sign meaning in textual or voice format in real-time
To translate user-entered text into relevant SSL hand sign

METHODOLOGY

Data Acquisition:

Creating Dataset using 500 SSL hand images
Building a model using Faster RCNN
Detecting user-images on a live camera stream

Sign Language Classification:

Building a model using CNN to interpret signs
Checking the accuracy of the user's hand image
Creating an API to fetch the identified letters

Text-Audio Output Generation:

Converting output into a textual format
Checking the spelling mistakes in each collection
Converting output into audio-format

Text to Sign Language Convertion:

Detecting user-inserted text using NLP
Building a model to identify relevant signs for a text
Creating a GIF images using hand signs

TECHNOLOGIES USED

Research Milestone

This section has details about EASY TALK project

2020-01-29

12%Project Proposal

Presentation: 6% | Proposal Report: 6%
The explaination how individual components are developing within the year to achieve the final research product.

2020-07-14

Progress Presentation I15%

A live demonstration with a presentation to justify the 50% completeness of the research product.

2020-09-25

18%Progress Presentation II

Presentation: 12% | Code Evaluation: 6%
A live demonstration with a presentation to justify the 90% completeness of the research product.

2020

DemoView Demo

The video-captured demonstration to present the final view of the completed research product.

2020

29%Final Assessment

Presentation: 10% | Final Reports: 19%
The final detailed demonstration of the product which was done as the last stage of the research.

2020

Viva37%

PP I: 15% | PP II: 12% | Final Presentation: 10%
The questionaire session with the research panel based on the flow of the research, EasyTalk.

Research Documents

This section includes already-produced documents and the documents which are pending to produce.

Project Charter Document

The formal documentation which used to share the understanding of the project’s scope, development, and objectives, while defining the roles and responsibilities of each member involved.

Proposal Document

A detailed documentation which gives an idea to the audience of what problems EasyTalk intends to address and the methods through which intend to achieve the goal. This describes desired product's results and several quality constraints.

Research Log Book

An online record book which used to record daily meetings conducted with supervisor and co-supervisor regarding the research's works. The daily updates in the GitLab accounts were also reported with the intension of tracking the improvements in the future.

Status Document I

Status Document-I was maintained individually while containing all the details about the current progress of each individual component. To prove the progresses the Gantt charts, WBSs, components' milestones and the details of the meetings with the supervisor were presented.

Status Document II

Status Document-II also maintained individually and contains same cotents as Status documet I. Additionally, this documents contains the acceptance letters got from the conferences.

Final Reports

Five detailed documentations contain information about the overall product and it's 4 main components respectively. Each report evaluates how the research was executed, being honest and objective.