In the 24th century, USS Voyager Capt. Kathryn Janeway frequently visits her starship’s holodeck to chat with Leonardo da Vinci and seek the maestro’s advice. Fans of Star Trek and other science fiction are accustomed to these kinds of encounters being as ubiquitous in the future as intergalactic travel. But holograms are no longer the stuff of fiction.
Closer to Earth, in 2021, famous artists’ holograms may not be roaming around in three-dimensional space—yet—but they’re making their way into real-life experiences.
At India’s Museum of Art & Photography (MAP), visitors will soon be immersed in one such experience. When the museum’s new building opens in Bangalore at the end of 2021, India’s renowned artist, the late Maqbool Fida “M. F.” Husain, will “come to life” on MAP’s new “holostage.”
Husain, who died in 2011, is considered one of India’s greatest modernist artists. Named one of 500 most influential Muslims in the world by the Jordanian Royal Islamic Strategic Studies Centre, Husain has been credited with putting Indian art on the international map.
Through MAP’s artificial intelligence (AI) experience, art enthusiasts can interact directly with Husain’s digital twin.
“The experience makes it possible to actually engage in a realistic conversation with one of the premier modern artists of India,” says Abhishek Poddar, MAP founder and trustee. “This project allows a personal glimpse into the artist’s life and practice, and is a much more engaging and lively way for viewers to learn about him rather than reading through essays or papers.”
The conversational digital persona is a collaboration of MAP, Accenture Labs (through its Tech4Good program) and OpEzee, the primary team at MAP Labs. At the intersection of art and technology, the project brought together the best of creative arts and advanced technologies, according to Shubhashis Sengupta, managing director and fellow at Accenture Labs Bangalore in India.
“The digital persona created for MAP is a great example of how technology can enable art education, as well as help create immersive experiences for digital-native audiences and offer virtual experiences to the public to increase audience engagement and revenue,” Sengupta says.
How the Husain digital twin was born
To help create the conversational hologram, first the MAP team combed through numerous interviews, videos, press articles, essays and books by and about Husain. This research had to be turned into a comprehensive list of questions and answers.
“The challenge was to simplify the information into questions that would be interesting for all audiences,” Poddar says. For example, the conversation needed to be engaging both for people who may have never heard of Husain or haven’t had a lot of exposure to arts, and those who are deeply familiar with him.
On the technology side, Accenture worked on making the AI persona life-like through techniques such as superimposition of Husain’s face onto a live actor; speech synthesis; and natural language understanding, natural language processing and emotion-detection technologies. Several aspects of the process, including the speech synthesis and the synthetic video, relied on deep learning, a machine learning subset that uses a neural network to simulate the human brain.
“[Additionally], text sentiment, voice sentiment and facial analysis provide the extra edge for the digital persona to be as responsive, expressive, immersive, proactive and adaptive as a real person would be,” Sengupta says.
The final step that still remained as of this summer—and perhaps the biggest challenge yet—was the setup of the viewer experience in the physical space. “It has to be carefully designed and executed in a way that allows people to feel they’re actually chatting with the artist rather than a simulated version of him,” Poddar says.
Human-centered design—which factors in “human feedback to enhance the user experience”—is at the heart of the project, according to Sengupta.
“While technology is the key pillar of human-centric design, the persona leveraged certain other non-tech aspects as well, which include a study of individual, social and organizational behaviors that govern relevant interactions; and user perceptions of interactions,” he says. “This helped design the conversational AI in a way that it increases user engagement.”
Thanks to advancements in technology, creating a simple synthetic video using face swapping and voice synthesis is relatively simple. “But developing a ‘living’ persona or an avatar of a deceased person that can interact seamlessly with users is much more complex,” Sengupta says. And that’s where the active collaboration with the art experts was instrumental, helping ensure the hologram can answer even complex art questions.
Sengupta adds that the project is a “robust example of how deep fake technology can be leveraged for good.” Potential applications of realistic avatars range from education, filmmaking and gaming to retail and even medicine.
Engaging the younger generations
The project came together in the middle of COVID-19 lockdowns. Originally, MAP planned to open a new, state-of-the-art, 42,000-square-foot museum in November 2020, but that had to be postponed. In addition to the Husain holostage, the five-story building will incorporate hologram technologies in other areas so visitors can view objects in an immersive way.
To give art patrons an opportunity to engage in the meantime—especially as physical spaces shut down during the pandemic—MAP launched the AI experience on web and mobile platforms.
While a conversation with the Husain AI would appeal to all ages, MAP and Accenture pursued the project specifically with young people in mind. “Currently, the younger generation of our country tends to view museums as spaces that are an extension of their history class—spaces that are boring and intimidating,” Poddar says. “We want to change that view and show them that museums are actually great spaces to learn about your past.”
MAP aims to build a museum-going culture that appeals to the younger generation and uses the persona to create an engaging user experience—one that helps visitors learn something new while still being fun.
—Shubhashis Sengupta, managing director and fellow at Accenture Labs
For this young generation, technology could, indeed, serve as the bridge between the past and the present. Millennials, for example, are more connected digitally than other generations, both in advanced economies and emerging ones like India, according to Pew Research. So it stands to reason that MAP hopes that technology-driven programs will draw younger people to the museum and broaden the appeal of arts.
“MAP aims to build a museum-going culture that appeals to the younger generation and uses the persona to create an engaging user experience—one that helps visitors learn something new while still being fun,” Sengupta says.
Blazing the trail for other arts organizations
The Husain AI is the first project of its kind in India. The massive collaborative effort and the advanced technology innovation by themselves are notable. But it’s the fusion of art, history and technology that perhaps makes it the most significant—and not just for India.
“The project really showcases that art, education and technology should go hand in hand to create experiences that can draw in millions of people across the globe,” Poddar says.
This draw is particularly important for India, where, Poddar notes, “museum visitation and art support are generally very low.” His hope is that these kinds of efforts will compel more organizations to support the arts, as well as encourage the public to learn about the arts by visiting museums.
“Even with the many technological advancements in India, the arts and culture sector has mostly remained underdeveloped and ignored,” he says. “MAP has taken the first step toward changing this and hopes that it will inspire other organizations and museums to follow in the footsteps and strive to create new, engaging experiences for its audiences.”
Lead photo of MAP, soon to open in Bangalore at the end of 2021. Rendering courtesy of MAP.