Computerized lip reading - tight lips?

UEA researchers are working in collaboration with experts at the Centre for Vision, Speech and Signal Processing at Surrey University, who have already developed advanced face and lip motion tracking systems. Both teams are currently collecting data, such as videos, which will be used in an analysis designed to determine the exact lip movements and facial expressions associated with specific letter combinations in the most accurate way. The scientists say that since very little is known about exactly what kind of visual information is required for effective lip-reading, one of their main challenges is the task of building a precise and sufficient database of photographs and videos to form a reliable basis for the lip-reading software. Their ultimate goal is to build machines that will be able to automatically convert lip-motion videos into text. According to the researchers, they also plan to extend the silent speech-recognition system to additional languages.

“To be effective the systems must accurately track the head over a variety of poses, extract numbers or features that describe the lips, and then learn what features correspond to what text.” – said Dr. Harvey. “To tackle the problem we will need to use information collected from audio speech, so this project will also investigate how to use the extensive information known about audio speech to recognise visual speech. The work will be highly experimental. We hope to produce a system that will demonstrate the ability to lip-read in more general situations than we have done so far.”

Dr. Richard Harvey (Credit: UEA)

TFOT previously covered MIT’s Lecture Browser software, which allows users to search video content using keywords. You can also read about NASA’s sub-vocal speech-recognition research, aimed to enable silent communication and speech augmentation in extremely noisy environments.

More information can be found on the UEA website (PDF).