UNT Professor and Students Create an App for People Unable to Speak and with Motor Disabilities

DENTON (UNT), Texas – A University of North Texas Professor and his students are working to help people who are unable to speak and who have severe motor disabilities communicate more efficiently.

The project started back in 2018 when Department of Computer Science and Engineering Assistant Professor Mark V. Albert worked at another university and met student Hannah Thompson.

Thompson is diagnosed with cerebral palsy, which affects the way she communicates, as she cannot speak and has difficulties making gestures. Out of necessity, she uses a tablet that has preprogrammed phrases. All she has to do is click certain buttons and the tablet will speak those phrases.

But it is not as easy as it seems. Thompson struggles to accurately touch the buttons she wants. She even has a key guard that keeps her fingers from sliding off the screen. As a result, it becomes time consuming as she looks down to type and concentrates on which buttons to click.

“In a social context, long delays and visual distractions are very limiting. If she can simply sign custom movements, and then naturally tie them together and use that in conversation, it’s a game changer,” Albert said.

That is when Professor Albert and his Ph.D. student Riyad Bin Rafiq, had the idea of creating an application that helps people like Hannah communicate better. So far, they made a prototype of the app which will be called TalkMotion.

The app works on a device like your phone or smart watch which will have the ability to preprogram common phrases, just like Thompson’s tablet. But the difference is that the preprogrammed phrases will match the movements of the device.

The user can then use the device in conversation without having to type messages on a tablet. The goal of the TalkMotion app is for the user to train the system by making their own set of movements and meanings.

You can find the application prototype now on the Google Play app store. As of now, Albert and his students are working on refining it. There is not a completion date at the moment, as it takes years for approval and sponsorships. But at the end of the day, this will make a huge difference and improve communication for the disabled.