Contrary to common belief, sign languages are distinct across different communities and cultures, evolving organically through interactions among deaf people, rather than being based on spoken... Show moreContrary to common belief, sign languages are distinct across different communities and cultures, evolving organically through interactions among deaf people, rather than being based on spoken languages. Each sign language has its own grammar, vocabulary, and cultural nuances, with variations even within a single country, showcasing the diverse communication methods within the deaf community. Deaf individuals often face encouragement to use spoken language techniques like lipreading or text communication, highlighting a bias towards spoken languages. This is compounded by the lack of sign languages in linguistic technologies, emphasizing the need for more inclusive research and development. This dissertation aims to address this gap using machine and deep learning to improve sign language processing and recognition. It covers six chapters, introducing methods for video-based sign annotation, webcam-based sign language dictionary search, and ranking systems for sign suggestions. It also explores tools for visualizing and comparing sign language variation, contributing valuable resources to linguistic research. Show less
Signs in sign languages have been mainly analyzed as composed of three formational elements: hand configuration, location, and move- ment. Researchers compare and contrast lexical differences and... Show moreSigns in sign languages have been mainly analyzed as composed of three formational elements: hand configuration, location, and move- ment. Researchers compare and contrast lexical differences and simi- larities among different signs and languages based on these formal elements. Such measurement requires extensive manual annotation of each feature based on a predefined process and can be time con- suming because it is based on abstract representations that usually do not take into account the individual traits of different signers. This study showcases a newly developed tool named DistSign, used here to measure and visualize variation based on the wrist trajectory in the lexica of two sign languages, namely American Sign Language (ASL) and Ghanaian Sign Language (GSL), which are assumed to be historically related (Edward 2014). The tool utilizes the pretrained pose estimation framework OpenPose to track the body joints of different signers. Subsequently, the Dynamic Time Warping (DTW) algorithm, which measures the similarity between two temporal sequences, is used to quantify variation in the paths of the dominant hand’s wrist across signs. This enables one to efficiently identify cognates across languages, as well as false cognates. The results show that the DistSign tool can recognize cognates with a 60 percent accuracy, using a semiautomated method that utilizes the Levenshtein distance metric as a baseline. Show less
Sign language lexica are a useful resource for researchers and people learning sign languages. Current implementations allow a user to search a sign either by its gloss or by selecting its primary... Show moreSign language lexica are a useful resource for researchers and people learning sign languages. Current implementations allow a user to search a sign either by its gloss or by selecting its primary features such as handshape and location. This study focuses on exploring a reverse search functionality where a user can sign a query sign in front of a webcam and retrieve a set of matching signs. By extracting different body joints combinations (upper body, dominant hand's arm and wrist) using the pose estimation framework OpenPose, we compare four techniques (PCA, UMAP, DTW and Euclidean distance) as distance metrics between 20 query signs, each performed by eight participants on a 1200 sign lexicon. The results show that UMAP and DTW can predict a matching sign with an 80\% and 71\% accuracy respectively at the top-20 retrieved signs using the movement of the dominant hand arm. Using DTW and adding more sign instances from other participants in the lexicon, the accuracy can be raised to 90\% at the top-10 ranking. Our results suggest that our methodology can be used with no training in any sign language lexicon regardless of its size. Show less
Fragkiadakis, M.; Nyst, V.A.S.; Putten, P.W.H. van der 2021
The annotation process of sign language corpora in terms of glosses, is a highly labor-intensive task, but a condition for a reliable quantitative analysis. During the annotation process the... Show moreThe annotation process of sign language corpora in terms of glosses, is a highly labor-intensive task, but a condition for a reliable quantitative analysis. During the annotation process the researcher typically defines the precise time slot in which a sign occurs and then enters the appropriate gloss for the sign. The aim of this project is to develop a set of tools to assist the annotation of the signs and their formal features in a video irrespectively of its content and quality. Recent advances in the field of deep learning have led to the development of accurate and fast pose estimation frameworks. In this study, such a framework (namely OpenPose) has been used to develop three different methods and tools to facilitate the annotation process. The first tool estimates the span of a sign sequence and creates empty slots in an annotation file. The second tool detects whether a sign is one- or two-handed. The last tool recognizes the different handshapes presented in a video sample. All tools can be easily re-trained to fit the needs of the researcher. Show less
Fragkiadakis, M.; Nyst, V.A.S.; Putten, P.W.H. van der 2020
This study presents a new methodology to search sign language lexica, using a full sign as input for a query. Thus, a dictionary user can look up information about a sign by signing the sign to a... Show moreThis study presents a new methodology to search sign language lexica, using a full sign as input for a query. Thus, a dictionary user can look up information about a sign by signing the sign to a webcam. The recorded sign is then compared to potential matching signs in the lexicon. As such, it provides a new way of searching sign language dictionaries to complement existing methods based on (spoken language) glosses or phonological features, like handshape or location. The method utilizes OpenPose to extract the body and finger joint positions. Dynamic Time Warping (DTW) is used to quantify the variation of the trajectory of the dominant hand and the average trajectories of the fingers. Ten people with various degrees of sign language proficiency have participated in this study. Each subject viewed a set of 20 signs from the newly compiled Ghanaian sign language lexicon and was asked to replicate the signs. The results show that DTW can predict the matching sign with 87% and 74% accuracy at the Top-10 and Top-5 ranking level respectively by using only the trajectory of the dominant hand. Additionally, more proficient signers obtain 90% accuracy at the Top-10 ranking. The methodology has the potential to be used also as a variation measurement tool to quantify the difference in signing between different signers or sign languages in general. Show less
