Introduction
Communication between deaf individuals who rely on sign language and those who do not understand sign language remains a significant challenge. Globally, an estimated 466 million people have disabling hearing loss and rely on visual languages like American Sign Language (ASL) as their primary means of communication. Without an interpreter, deaf persons often face barriers in everyday interactions such as education, healthcare, and customer service. AI-powered sign language translation offers a promising solution by automatically translating sign language into spoken/written language and vice versa, thereby closing the divide in communication. Recent advances in computer vision and deep learning enable robust recognition of hand gestures and facial expressions, while NLP and speech technologies can generate fluent sign language or speech output. The objective of this research is to design a two-way translation system that: (1) recognizes sign language from webcam video and converts it to text and audible speech in real time, and (2) converts spoken language (voice) into accurate sign language, presented via an animated avatar. By facilitating bidirectional communication, such a system can greatly enhance the independence and social integration of deaf and hard-of-hearing individuals. In the following sections, we discuss background and related work in sign language recognition and synthesis, detail our methodology including the AI models and system architecture, present experimental results, and examine the impact on the deaf community along with future research directions.
Related Work
Early approaches to automated sign language translation involved instrumented gloves or heuristic computer-vision techniques. For example, instrumented glove devices with sensors have been used to capture hand motions, but these solutions can be intrusive and limited to specific vocabularies. With the rise of computer vision, focus shifted to camera-based sign language recognition. Traditional vision methods employed techniques like skin-color segmentation and handcrafted features (e.g., Haar-like features or optical flow) to detect hand gestures, but often struggled with variability in lighting and sign execution.