Menu

Menu controller icon

Virtual reality and the future of sign language learning

Posted: 07-09-2021

Author: Ryan McCloskey

Tags:

    As of 2019, there were over 70 million hard-of-hearing people worldwide using sign language as their primary means of communication. Sign language allows them to access education, to work, access public services and much more, in a manner which would otherwise pose an even greater challenge.

    However, there is an undeniable difficulty in learning languages that require an altogether different form of communication from traditional speech and text. Common forms of learning such as books struggle to convey the required information as descriptions and diagrams are not suited for teaching abstract gestures.


    Irish Sign Language Alphabet – (Source: Irish Deaf Society)

    The digital era has seen a trend towards online education, which offers the benefits of video content which can more accurately assist in learning sign language. However, this is still not without its drawbacks, as technical difficulties such as network issues or teaching through a two-dimensional screen can cause a disconnect between teacher and student.

    This means that, to date, the most effective process of learning sign language is through in-person tuition, which for any number of reasons, particularly in the current climate, can prove challenging and necessitate a new solution.

    The potential for virtual reality as an educational tool has been slowly coming to realisation in recent years, with technological advancements and a reduced cost barrier to entry proving pivotal in growing the wide appeal of the technology.

    Mondly VR

    An interesting area of virtual reality education is that of language learning in VR. The concept has been around for a few years, with services such as Mondly gaining traction by offering language learning in a virtual environment designed to place students in everyday situations and communicate using the language they wish to learn. Such a practice is based around the concept that immersing oneself in a cultural environment is the most effective way of learning a new language.

    More recent development in the VR space, specifically the introduction of accurate hand tracking to mass-market VR headsets such as the Oculus Quest, have given rise to the possibility of gesture-based language education through virtual reality.

    The major advantage to the use of VR for learning sign language stems from the fully-3D immersive environment offered by virtual reality, and its relatively accurate hand tracking model can be used for representation of users’ own hands. This, combined with software-based gesture recognition can generate instant feedback and even translation based on hand poses and movements.

    Current drawbacks are primarily around the accuracy of device hand tracking. The existing hand tracking system on the Oculus Quest, for example, struggles with fingers overlapping or in very close proximity which is common to some letters of the ISL alphabet and can prove to cause issues. However, as the technology is improved over time these issues will naturally be diminished and facilitate more fine-grained gestures crucial to sign language.

    Even with the limitations of existing technology, there is an exciting potential for the use of virtual reality to teach sign language, and with more research and technological improvements, it is very feasible that virtual reality will become a crucial element for those learning sign language in the foreseeable future and help to bring about a more inclusive society.