Being There Via VR Helps English Learners Get Language Use Faster

By Marty Graham, Contributor

Miguel Ramos, a middle-aged student from Colorado, wanted to learn English to better understand what his teenage daughters were talking about. His work hours were long, however, and he could barely stay awake during night classes at the community college — especially when they involved hours of verb conjugations and memorizing grammar.

“The book says to talk this way, and my kids say it completely differently,” Ramos said. “I was no better at talking to my girls after months of classes.”

Fortunately, his teacher, master instructor Mary Pelanek, who retired from teaching ESL math and science at a Boulder, Colorado High School, made an interesting suggestion: Watch English language soap operas with closed captions and, at the same time, use Google Translate on the smart phone to provide real-time translations. When hovering the smartphone camera over the closed-caption text, the app uses an algorithm to convert the written word to Spanish as the conversation unfolds.

Translations aren’t always dead on and sometimes they’re dead funny, Pelanek warned, but they helped Ramos a great deal and definitely held his interest.

“It only took a few weeks of watching that I learned enough to understand my daughters’ conversations and to be able to talk with my children in their language,” he said. “The shows are about families, so they talk about things we talk about and understanding became easier.”

It’s through this example we see the way Ramos and his teacher were able to turn to a raw form of augmented reality (AR) into interacting in English – which was where he wanted to be.

Graduating to VR

Like with AR, virtual reality (VR) has been popping up everywhere in education, particularly in medical and manufacturing arenas, but it is only beginning to enter the world of adult-language learners.

That’s due in part to the uncertainty of whether or not it will work well enough to be worth the training and money necessary to roll it out ( and do it well), according to ESL teacher Michelle Cowans.

An Australian teacher who works with immigrant adults trying to learn English, Cowans reported that her students learn more and faster when teaching is augmented with VR. Her findings have landed on a new but growing pile of research that shows VR technology used for learning is particularly effective for ESL adults who benefit from immersive, everyday experiences.

The combination of more affordable VR technology tools, the availability of mass and proprietary programs like Google Expeditions, and the pervasiveness of adults with smartphones is also changing the game for immigrant adults, she said.

Last year, Cowans used Google Expeditions—an education smartphone app which offers both VR and AR “tours”—to take her adult migrant students on seven different virtual adventures. Students went from touring Aurora Borealis to climbing El Capitan in Yosemite National Park and, as they became more sophisticated, designing their own expedition.

“Students understand the subject matter much better than if they were seeing two-dimensional images,” she explained. “Its experiential nature is a multi-sensory experience, enriching learning by fostering curiosity.”

What Cowans sees in her adult learners mirrors what several other language education departments across the country—from the University of Maryland to Washington State University—are researching and documenting.

Washington University researcher Don M. McMahon, who specializes in studying these emerging mixed reality tools, said such immersion tools—whether clumsy or refined—are novel and effective.

“The virtual and augmented reality experience is a lot more real, so it’s very engaging,” McMahon said. “You’re actually seeing in a 360-degree video and you get to be right there. I get to hear the sounds. It has a higher degree of meaning to the learner.”

McMahon’s goal with his mixed-reality research is to create a knowledge base that demonstrates that the tools work, how and why they work, where they fit into the spectrum of teaching tools, and where there may be problems. He works closely with teachers for research purposes, and to train them on new tools.

“VR has the advantage of offering multiple means of representation, multiple means of action and expression, and multiple means of engagement,” McMahon said, outlining basic principles of universal design for learning.

“The more I can provide a variety in ways to learn, the better people’s learning experience will be,” he added. “Sometimes, the initial hook into something makes all the difference, and virtual reality is a really great hook.”

A Hook to the Brain

According Charan Ranganath, who leads the Dynamic Memory Lab in the University of California Davis, virtual reality is a hook that goes straight to the brain.

“In general, a very immersive experience means people are more engaged, and it’s a natural way of interacting,” explained Ranganath. “Having a rich, well-developed context makes memories easier for your brain to access and utilize.”

While remembering is part of learning, retaining a memory does not mean a person can act on that knowledge. For example, acing your French test doesn’t necessarily mean you’ll be able to successfully converse with locals in Paris.

Ranganath says that in order to truly learn a language, the student must use the memory in another context, for instance, starting a conversation in a new language rather than passively watching others speak it or engaging with a computer program.

Actively initiating an action stimulates the brain area called the hippocampus, a horse shoe shaped area of the brain that’s home to long and short-term memory, and emotional responses. And, the richer the context, the greater engagement with the hippocampus.

With VR, the 360-degree experience sends the learner’s brain much bigger signals, which, he said, the brain likes.“In VR you have very rich context. That memory is going to be very different than if I read it or hear a lecture.”

There’s fairly good evidence that shows that this engaged learning stimulates neuromodulators in the brain that help people remember things better and in greater detail, Ranganath explained. “Things that are surprising or novel or attractive or scary all drive neurotransmitter systems and you’ll remember more and better.”

This is precisely why VR is an effective tool for taking all of these different brain circuits and putting them into the service of remembering things, Ranganath concluded. “Our brain is going to pull out the most exciting memory first, and that response makes the VR experience very effective.”

Another part of what makes VR an effective learning tool comes from activating a person’s senses, which VR can stimulate. It’s possible, Ranganath went on, to design a VR experience to intentionally engage our brain directly and thoroughly. “You can set it up to engage almost every sense so you have a richer body of memories,” he said.

Yet, how much the experience touches the brain comes down to the design, programming, and creation of the experience. Poor design will be the biggest shortcoming, he said. Yet, the impact is there.

In June, researchers at the University of Maryland found that people retain memories from their VR experiences better than they retain information from flatscreen exposure.

Participants from the study said the immersive ‘presence’ of VR gave them greater focus. This was reflected in the research results, where 40 percent of the participants scored at least ten percent higher than their counterparts who used desktop computer displays.

Suspension of Disbelief

One reason VR fosters individual learning, according to McMahon, is that students are able to suspend their disbelief.

“Since it’s from your point of view, you’re able to approach learning with suspension of disbelief,” he explained. “You can attend to what attracts you. For example, in a city scene, I might be drawn to look at the park and you may be watching the people in the scene. That’s powerful and personalized.”

People get to see themselves functioning in the virtual scene — without doing it in front of others. According to McMahon, VR technology gives users the chance to bridge the experience between learning and context by immersing them into the experience of communicating without facing real social repercussions.

McMahon is also documenting the effectiveness of AR with the real-time translation. “These two immersive technologies have different strengths,” he explained. “VR is remaining stationary and being transported to the experience, whereas AR has the advantage of helping us learn in the context of our own experiences.”

McMahon says that Ramos using his phone’s camera for translation is a great example of learning through your own, lived experience.

Sure, it’s not perfect and there are errors in the translation, but getting a lot of it right makes it a great resource. It’s a simple way for a student — particularly a working, busy one — to have an immersive and contextually-relevant experience.

McMahon also likes to encourage students to play Pokemon Go in the language they hope to acquire.”If you can get your class excited about playing something familiar in another language, you’ve opened the door to learning,” he said.

A few teachers, including Cowans, are already trying mixed reality methods in their classrooms. McMahon said that many more want to. “I can tell you that teachers everywhere are excited about using these platforms because they can see this as another way to engage people in learning that allows more tailored, focused learning,” he said. Still, it’s early for these platforms.

“Part of what we’re trying to do is build a research foundation to help support educators’ decisions to use them,” McMahon added.

As for Cowans, she plans to keep expanding her use of VR where it’s appropriate. She points to a burgeoning batch of 360-degree videos available on YouTube. “The students are engaged and motivated,” she mused. “They report never having learned in this way before, believe that it aided their retention and interest, and came away from the experience enthused and eager to learn more.”

As for when her students felt AR or VR aided their learning experience? “In every instance,” she said.