Meta released an open source language learning app for Quest 3 that combines mixed reality passthrough and AI-powered object recognition.
The News
Called Spatial Lingo: Language Practice, the app not only aims to help users practice their English or Spanish using objects in their own room, but also give developers a framework to make their own apps.
Meta calls Spatial Lingo a “cutting-edge showcase app” meant to transform your environment into an interactive classroom thanks to its ability to overlay translated words onto real-world objects, which uses AI and Quest’s Passthrough Camera API.
The app’s AI not only focuses on object detection for vocab building, but also features a 3D companion to guide you, which Meta says will offer encouragement and feedback as you practice speaking.
It also listens to your voice, evaluates your responses, and helps you master pronunciation in real time, Meta says.
Spatial Lingo is also open source, letting developers use its code, contribute, or build their own new Unity-based experiences.
Developers looking to use Spatial Lingo can find it over on GitHub. You can also find it for free on the Horizon Store for Quest 3 and 3S.
My Take
I’m usually pretty wary of most language learning apps since many of them look to maximize engagement, or otherwise appeal to what a language learner thinks they need rather than what the answer actually is: hours and hours of dedicated multimedia consumption combined with speaking practice with native speakers. I say this as a native English speaker who speaks both German and Italian.
Spatial Lingo isn’t promising fluency though; it’s primarily a demo that comes with some pretty cool building blocks—secondary to its use as a nifty ingress point into the basics of common nouns in English or Spanish. That, and it actually might also be pretty useful for beginners who not only want to see the word they’re trying to memorize, but get some feedback on how to pronounce it. It’s a neat little thing that maybe someone could take and make into an even neater big thing.
That said, the highest quality language learning you could hope for should invariably involve a native speaker (or AI indistinguishable from a native speaker) who not only points to a cat and can say “el gato “, but uses the word in full sentences so you can also get a feel for syntax and flow of a language—things that are difficult to teach directly, and are typically absorbed naturally during the language acquisition process.
Now that’s the sort of XR language learning app that would make me raise an eyebrow.
The post Meta’s Language Learning App for Quest Combines Mixed Reality and AI appeared first on Road to VR.