ASSISTIVE VOCALIZER USING OPENCV FOR SIGN LANGUAGE RECOGNITION AND TRANSLATION

Project Domain / Category

Image Processing / Artificial Intelligence / Web App/Deep Learning

Abstract / Introduction

The Assistive Vocalizer is created as a two-way communication system, which integrates computer vision and speech recognition to assist speech and hearing impaired people. The system works with a camera built into the web interface that is based on Django to record real-time hand gestures and work with them via OpenCV to identify, segment and extract landmarks. This is followed by the classification of these features of gestures with the help of a TensorFlow/Keras deep learning model CNN to map them to alphabets, words, or phrases. The identified motions are presented on the web front-end and converted further to audible speech by the text-to-speech integration, which enables the smooth communication between the sign language and the non-sign language users. The project is made to be accessible because the logic is embedded on Django views and templates to provide a user-friendly and clean interface that can be deployed on the web. The system also has voice recognition to facilitate two way communication where words spoken are translated into sign language gestures shown in the interface. Such advanced tools as sentiment analysis improve the natural course of communication by understanding the emotional tone and Urdu translation makes the application inclusive to local users. The system uses image processing, deep learning models, and natural language processing together on the same interactive platform, thereby integrating the front-end and back-end communication with Django. This integrative design will grant the speech-impaired community with an effective and scalable assistive tool that can be utilized in various gadgets.

Functional Requirements:

        The system will record real-time hand gestures with the help of a camera and be processed with OpenCV to detect gestures and identify landmarks.

        The system will identify identified gestures as alphabets, words or phrases through a trained TensorFlow/Keras deep learning model.

        The system will process accepted gestures into text and consequently an audio speech with the text-to-speech software.

        The system will also be voice recognition to be able to translate voice input into the sign language gesture that is displayed on the interface.

        The system will incorporate sentiment analysis to comprehend the emotion of conversation.

        It will be inclusive with the system offering multilingual assistance, such as English and Urdu translation.

Non-functional Requirements:

        Act promptly on gestures with low latency levels so that there is free interaction.

        Learn to work under various lighting, background and hand positions.

        Notify to personal users, languages and custom gestures.

        Grow effortlessly to facilitate more gestures and words.

Page 27 of 167