Notre Dame alums create glasses that project subtitles for the deaf
Subscriber Benefit
As a subscriber you can listen to articles at work, in the car, or while you work out. Subscribe Now
A pair of traditional looking glasses that project subtitles for a deaf or hard of hearing person is the bold vision of two recent University of Notre Dame graduates. The concept may seem far-fetched, but the duo has its first prototype and a startup that quickly raised $300,000 in funding to support its futuristic vision: to reimagine assistive hearing technology with “wearable subtitles.”
HeARsight co-founder and Chief Technology Officer Danny Fritz had his lightbulb moment during the pandemic when his friend, who is hard of hearing (HOH), could no longer read lips due to face masks.
“Movies and TV shows all have subtitling,” says Fritz, “so why can’t that be done in a real-time fashion with smart glasses?”
People who are deaf or HOH sometimes use cell phones for speech-to-text assistance, but it requires the person to constantly hold their phone and look down at the display, which Fritz says furthers the stigma that deaf and HOH people already feel. South Bend-based HeARsight says its smart glasses would provide a “heads up” display with captions projected in the wearer’s field of vision.
Fritz and HeARsight co-founder and CEO Riley Ellingsen were classmates in Notre Dame’s ESTEEM Master of Science program. The roommates nurtured the idea throughout the entrepreneurial-focused program and cooked up a plan for HeARsight “just talking in our kitchen,” says Fritz.
HeARsight’s prototype uses an embedded microphone to pick up audio from the person the wearer is having a conversation with. An algorithm converts the speech to text, which is projected on the glasses’ lens.
“Waveguide technology allows the projection to appear further away from you than it really is, so the focal point of the text would appear as if it’s about three feet in front of your eyes, rather than just directly on the lens itself,” says Fritz, who’s a software engineer. “It kind of creates this illusion with your focal point that allows the text to be viewed in a comfortable way.”
The glasses use augmented reality (AR), so the subtitles are overlaid onto the environment. Conventional AR glasses currently on the market are notoriously bulky because they’re performing many computational tasks, which require a large amount of hardware. HeARsight believes because its smart glasses have only one task—to create subtitles—the glasses will need much less hardware on board.
“Our goal is to shrink it down as much as possible and keep it pretty close to the same size as traditional glasses,” says Ellingsen. “We’re hoping to have a product that avoids some of the stigmas you see associated with other hearing assistive technologies. A big piece of that is having an inconspicuous or discreet design.”
HeARsight says it’s taking a “software first” approach—and glasses later—so the startup is currently focusing its efforts on building the software, or app, that will perform the transcription and projection of the text. The startup recently formed a partnership with a third-party hardware company, which is one of largest consumer electronics brands in the world.
“We’re in an early access developer program with [the company]; we feel pretty lucky that we get to participate in that, and we’re excited to see what we’re able to do in the next few months,” says Fritz. “This summer, we’ll be building the app for their glasses, then we’ll look to either scale up and launch that software product, and/or look into building our own [glasses] to integrate the software into.”
Fritz says it’s yet to be determined if heARsight’s smart glasses will need Wi-Fi to operate.
Although developing the app is the first task, HeARsight built a low-fidelity prototype that Fritz acknowledges “looks pretty goofy,” but “helps illustrate to some skeptics the speed of transcription and how well you can see subtitles in front of you in your field of view.” The startup says its glasses will cost significantly less than hearing aids, which are approximately $2,000 per device, and most people need one for each ear.
Ellingsen and Fritz say the 800 people who have joined HeARsight’s waitlist, each sharing their personal story, motivate them to keep working to deliver a higher quality of life for people who are deaf or HOH.
“Deafness and HOH isn’t the problem, it’s the communication access that’s the problem. This gives them an opportunity to overcome that and participate in in-person conversations in ways they’ve never been able to before,” says Ellingsen. “For age-acquired hearing loss, like aging grandparents, they have to sit out of conversations because they can’t hear. Allowing them to stay connected with their loved ones—that’s why we’re doing this.”
Ellingsen says it’s been powerful to see people enjoy using the smart glasses in preliminary testing.