Episode 43: Telling Real Stories… with Virtual Sets

INTRO: Luminaries. Talking to the brightest minds in tech.

MICHAEL DELL: We are technologists, and we share an awesome responsibility. The next three decades will hold even more progress coming more quickly than ever before. A new age of miracles is literally just around the corner.

WOMAN: Your hosts are Mark Schaefer and Douglas Karr.

MARK SCHAEFER: Hey, everyone. Welcome to another episode of Luminaries, where we talk to the brightest minds in tech. This is Mark Schaefer with my co-host Doug Karr, AKA Dougie Baby. How are you doing Doug?

[LAUGHTER]

DOUGLAS KARR: I’m doing fantastic.

MARK SCHAEFER: I just haven’t called you that for a long time.

DOUGLAS KARR: I know. It’s been a while.

MAN: Yeah, it just– it was there. And we’ve got such a cool, cool show today, because we have talked to so many interesting people over the years. But this fellow, Glenn Gainor, that we’re going to be talking about today, is particularly interesting, I think, because if you think about how technology really changes our everyday lives, I would say number one, internet.

DOUGLAS KARR: Yes.

MARK SCHAEFER: Number two, health care. Number three, entertainment. If you think about both the creation and distribution of entertainment just in our lifetime, it’s just been transformed. So I’m so happy to introduce our guest today, Glenn Gainor. He oversees Innovation Studios, a state of the art facility housed in a soundstage at the Sony Pictures studios lot, which is just so cool, just that. The space features the latest in research and development from Sony Corporation and other areas, including volumetric video and customizable set scanning to help storytellers around the world create content in radically new ways.

He’s also the head of physical production for Screen Gems– we’ve all heard of that– a label under the Sony Pictures Motion Picture group. Glenn, welcome to our show.

GLENN GAINOR: Thank you, Mark. It’s great to be here.

MARK SCHAEFER: We’re going to have some fun. Now, this term I used in the introduction, volumetric video. That’s probably a term unfamiliar to a lot of people. What’s that about, volumetric video?

GLENN GAINOR: Volumetric video is a new way of telling stories. So imagine this. Imagine you’re in an amazing location. Let’s put yourself into your favorite monument or put you on your favorite set, and you want to go back to that environment, whatever it is. Could even be your living room. I imagine we all had great living rooms. I don’t know. Some of us do. Some of us don’t. But let’s put you back into a place you understand.

We can go in and through the technology of volumetric image acquisition, we can capture that real world, digitize it, and put you in that world virtually, so that you’re in a moving environment that is as deep as, say, 30k. Now what I mean by that is, you probably know 4K TVs and all this kind of stuff. But when you capture film or television or commercials, whatever the case might be, usually we capture in 4K cameras or 6K cameras. We’re going to be able to film in that virtual environment that’s going to be 30K in its resolution.

So it’s taking real world assets, digitizing it, and preserving it, and then virtually living in it.

MARK SCHAEFER: Now, is there some common example that people can think of that maybe they’ve seen or experienced that they can picture, oh, yeah, that’s what he’s talking about?

GLENN GAINOR: The great answer is that there isn’t anything that you can say, that’s a virtual reality set from us, because it’s indistinguishable from reality. Now we did take part in a television series called Shark Tank, great TV show. And they are now going to go down in history books because Shark Tank was able to film in the volume. It was the first time in the history of the world that a major TV show was using a virtual set to tell a story. And it was born out of necessity. It shoots in the Sony Pictures studio lot, and we just didn’t have the space.

So we went in, and we captured the exit interview room, and therefore, did a volumetric image acquisition of that set, and then recreated it digitally. And the contestants, about 100 of them, were coming in and out of that virtual set, staying in the moment. And no one was the wiser. In fact, funny story, the guy in the truck, because there’s always somebody in a truck in these TV shows, and they’re kind of removed and not in the stage. And they’re looking at a bunch of monitors.

I think he leaned and says, hey, guys, could you guys maybe move that plant just a little bit to the left, and then somebody cued it and said, I don’t think we can do it right now.

[LAUGHTER]

And he said, you’re at Sony Picture Studios. I’m sure you got somebody who can move a plant. They said, no, you don’t understand. It’s not really there. It’s virtually there. He goes, oh, that’s right. This is Innovation Studios. Sorry.

DOUGLAS KARR: That’s incredible.

MARK SCHAEFER: What an amazing story.

GLENN GAINOR: You can’t tell the difference.

MARK SCHAEFER: What an amazing story.

DOUGLAS KARR: So it’s virtual– virtual reality is a piece of this, right, but it’s taking it from a camera? You’re filming virtual reality?

GLENN GAINOR: We’re taking real and unreal, and we’re having them meet each other.

DOUGLAS KARR: Wow.

GLENN GAINOR: So it’s physically you and me on the set. The three of us can be physically there, but we’re virtually in a set that is made out of billions or trillions or quadrillions of points.

DOUGLAS KARR: Incredible.

GLENN GAINOR: It’s kind of staggering. I remember we went out, and we captured a set for Men in Black, which is in theaters this June 14th, Columbia Pictures. We were invited onto the stage at Leavesden Studios in the UK to capture their headquarters set. And we actually filmed a commercial for them that put the actors in a commercial and put them back on the set, but in Culver City at Sony Pictures, because there’s no difference.

So we’re able to do things we couldn’t do before. So real actors, smoke, everything else, real tables, all that stuff. But they were placed virtually back on that set.

MARK SCHAEFER: You know, one of the things I think about when I listen to a story like this or I watch some of these incredible movies– I love movies with lots of cool special effects, and if it’s an alien, I usually go to it.

DOUGLAS KARR: Yeah.

[LAUGHTER]

MARK SCHAEFER: And it makes me think about this mind blowing amount of data and technology that you’re talking about. To be a movie producer today, to be a movie creative, there’s this merging between being creative and also knowing enough about the technology to get the job done today. I just think that’s so interesting.

I mean, you almost have to maybe not be a computer scientist, but have enough appreciation of what the possibilities are to make a movie today.

GLENN GAINOR: Yeah, you really do have to understand the power of what technology can do to help you create the best story. It’s knowing the opportunities. Filmmakers have been around for over a century now, and it’s always been technology that’s been driving the stories. I mean we started out with mechanical devices.

And look, kids are so far removed from the mechanical world that we all grew up with or many of us grew up with. But a few weeks ago, I was showing my kids my grandfather’s eight-millimeter mechanical camera for the first time. I just pulled it out of storage. They don’t know how to wind it up.

We’re always going through change, but now we’re going to go through radical change should we need to do so in a very positive way, because we want to be able to collapse geography and be unchained from geography, so we can tell any story that makes sense.

So imagine if you wanted to film in Piccadilly Circus, Times Square, and perhaps just pop over to Paris, why not, and Tahrir Square in Egypt. Sounds like four disparate, totally different kind of locations. Imagine if we captured those locations in a volumetric image acquisition.

MARK SCHAEFER: With perfect weather, by the way.

GLENN GAINOR: Good weather. We want good weather.

MARK SCHAEFER: Yeah, perfect lighting and perfect weather.

GLENN GAINOR: And if the weather is not great, we have opportunity to change the lighting conditions, so you’ve got dynamic range, because if you capture it in a beautiful blue sky, and you say, well, you know what, this actually should be darker, because we have such an enormous amount of data, we’re able to make that a darker scene now.

MARK SCHAEFER: Yeah. I mean, it’s amazing that someone who was in the movie business, if they just stepped away for a decade, it would just be so unfamiliar today. That’s really going to be a core competency for this new generation of filmmakers, just to understand the potential of what’s there to tell the story.

GLENN GAINOR: Yeah, years ago, so many production executives would say, look, the two most expensive words are what if. And today, the two greatest words are what if.

MARK SCHAEFER: What a great line. What a great line. I love that.

GLENN GAINOR: Because now it’s what if we do this, what if we employ this bit of technology, what if we employ that kind of idea? Before it was what if we had 20 stunt guys falling down a bunch of stairs into the river? Well, you know, that’s kind of expensive, isn’t it? But what if you could do that using technology, and not actually have to close down the streets and be near that river and so forth and so on, and recreate environments? That’s the trick. You have to ask the right questions. And our partners and colleagues in the technology industry are here to give us the right answers.

MARK SCHAEFER: It’s fascinating because where my head is going on this is that sentiment really goes beyond even filmmaking is if you think about how technology is being applied in so many parts of the world– I come from the age of– I worked for a manufacturing company, and you’re right, that question, what if would just drive everybody crazy. And now with AI and modeling and stuff like that, you’re right, that’s the greatest question to ask today.

GLENN GAINOR: Yeah. Yeah.

MARK SCHAEFER: Across industries.

GLENN GAINOR: Yes, exactly. And there are so many important stories that should be told if we can allow some stories that couldn’t have been told otherwise, then that’s a pretty magnificent endeavor. And that’s our what if. What if we could empower a story that couldn’t have been told before? And we hope that with this new kind of technology, powered by Dell, that we’re able to have the compute power. We have the know. That’s for sure, and we have the dream. And we have the ability to turn those ideas, that inspiration into action. And action is always a good word in my business.

DOUGLAS KARR: We see with digital transformation the rapid adoption of things like this. And I’m just curious from your perspective. Mark and I both obviously do content at this micro scale for ourselves and for clients. How many years before the average person or average business gets access to this type of possibility?

GLENN GAINOR: Well, it’s a passion of mine as well, Doug, the sort of so-called democratization of storytelling, democratization of filmmaking. We’re seeing it. We’re seeing stories being told that could never have been told before. We’re hearing new, exciting voices from around the world. That’s the goal. The goal is to eventually get this kind of technology to inspire stories that couldn’t have been told before. There are parts of the world that are just really difficult to get to, whether it’s dealing with distance or whether it’s dealing with locations that are just challenging. And this will allow us to get into those locations.

MARK SCHAEFER: So your vision is is that eventually this sort of technology will trickle down to independent filmmakers and small filmmakers and even people doing videos on YouTube.

GLENN GAINOR: Yeah, I really do hope so. I hope it inspires stories that somebody might have said, you know, this is too expensive for this sort of independent idea. But let’s think about the power of that independent film. I mean, look at the Academy Awards. I mean, so many independent films are celebrated at the Oscars. So many people celebrate that voice that maybe doesn’t have the power of the studios. And of course, I work at the studio. I’m a very big fan of the studio. But even we want to make new, fresh stories, whether it’s the television department or the motion picture department, whatever it takes to enable. I mean, that’s, to me, the key.

MARK SCHAEFER: I’m going to switch it up a little bit, because I was reading in your bio that one of the things that you’re passionate about is bringing a new environmental perspective to making films. You’re passionate about our world, our planet, and how to create these films in a more environmentally friendly way. And it’s something that I’ve thought about. It’s almost like a movie set is like the world’s biggest single use package.

It’s created, and you’ve got all these intricate things, and it might be just used for a few minutes to create some image to tell the story. So what are some specific examples of how you’re beginning to make progress in this area, Glenn?

GLENN GAINOR: Well, we’re making progress by proving out that the technology works. Clearly in a television series, it works.

MARK SCHAEFER: Now, what kind of technology? So what’s sort of some of the key replacement items that you’re looking at?

GLENN GAINOR: We’re really looking at valuing the assets around us. It’s a big deal to make a movie set or television show set. Once you’ve created the set, you do film in it. You light it. You have that control. But then if you want to go back to it, you have to do what’s call the fold and hold, so fold and hold, meaning you have to take it apart, in the most careful manner, you have to store it in an environment that’s going to be great for that asset, for that set, and hold it indefinitely.

Sometimes you’ll hold it until the movie has a cut that you like. Sometimes you’ll hold it because a show gets picked up for a second season one hopes or another season. Whatever the reason is, it’s a lot of physical movement. But what if you built it, and you did a volumetric image acquisition of the set, now you can go back to that set again and again, and you don’t have to worry about its condition. You don’t have to worry about transporting all of that real large physical assets.

But also, a friend of mine does a television series that builds worlds. And he said the hardest thing for him is when he’s in season three, and somebody says, hey, you remember that set we built in season one?

DOUGLAS KARR: Oh no.

[LAUGHTER]

GLENN GAINOR: The one you spent a million dollars on, we to go back there. And he said well, guys, you only give me so much money a year, so I got to take 20% of my budget to do what I’ve already done before. I said, well, unfortunately writers have an idea. So imagine if he could have taken what he’s already built, go back to that set, and then build something more, because no matter what, shows, they have a finite budget typically. So why not get the most out of that budget? And why don’t we get the most out of our stories?

DOUGLAS KARR: That’s incredible.

MARK SCHAEFER: Well, it’s quite stirring, really, inspiring that this technology, the volumetric video, could have a huge impact on, of course, obviously, cost, but also the environmental footprint of some of the materials that are used, of just the space it takes up to store these things. Now, all of a sudden, you’ve got it in a server someplace, and boom, you can recreate any world you want.

GLENN GAINOR: Yeah, that’s exactly it. We have a few worlds that we’ve been building up. And it’s kind of fun, we have a filmmaker on the stage, and we’re in one location, a club in West Hollywood, and they just simply ask, would you to go to Tokyo now? Yeah, OK, well, hold on it. Takes us about five minutes.

MARK SCHAEFER: Wow.

GLENN GAINOR: And then you’re there. Now you tweak the lighting, because maybe the interior of a nightclub is going to be different from the exterior of a location, but besides that, you’re there. You’re able to move around the world.

DOUGLAS KARR: That’s incredible. And by move around the world, you mean, you have this 40K recording, so you can actually move your camera angles and everything else and still keep the perspectives.

GLENN GAINOR: That’s exactly it. You have what’s called parallax. So you’re seeing the world as we see the world. It’s not a flat image. It’s not a plate. But really it’s a world that you can walk around and take in and really be there, and the camera sees what you may not be able to see at that moment, but it’s taking in the real, and it’s taking the the unreal. And it’s saying, OK, you’re all existing in the same world for me. And so you’re living and breathing in that virtual world, and you can put whatever props you want to put in there, the tables, whatever makes you comfortable to survive in a world, to enjoy that world. You can go ahead and place it in there.

DOUGLAS KARR: That’s incredible.

MARK SCHAEFER: The technology almost sounds like you would make a good– I mean, there’s a germ of a movie there.

GLENN GAINOR: Exactly. I know. Yeah. It’s all those things that we’ve seen in science fiction sort of coming to life. What I love is that we’ve had camera assistants come to the set, and they’re the people who do critical focus using the camera and the lenses. And they always get surprised for the first time to be able to do critical focus on what’s physically not there and rack focus to what’s physically there. And that’s a lot of fun.

They’re just like, oh my god, that table’s not there, but I’m focused on it, and then that person just walked behind the table. The table occludes the person, and I’m back focusing to the person. That’s fun.

MARK SCHAEFER: Yeah.

DOUGLAS KARR: Wow. I thought you looked at me and said, wait a second.

MARK SCHAEFER: Number six.

DOUGLAS KARR: I know. I read in your bio– or let me do that over. As we’re doing research, Glenn, I read that one of your passions is for the environment as well. And it was interesting to see that there’s an intersection here. At what point did you kind of take one passion and then merge it with your filmmaking passion?

GLENN GAINOR: I’m led by passion in many ways. I think a lot of these ideas that we come up with are ideas that we’re excited about, that I’m excited about. And I’ve always been excited about telling stories in the most environmentally friendly way you can, whether it’s using fewer large HMI movie lights and using more LED lights or natural light. Definitely the switch from film to digital, but good digital, you know, 4K, 6K, high dynamic range, everything that helps a story but doesn’t do the damage to the environment.

DOUGLAS KARR: Right.

GLENN GAINOR: And that’s what’s great about using virtual sets is the reuse. I mean, we’re recycling real world. That’s the greatest kind of example of recycling the real world.

DOUGLAS KARR: That’s a great–

GLENN GAINOR: And that’s what we want to do. I mean, we want to preserve, and we want to be able to go back and back and back to great locations, reduce our carbon footprint while making our stories. It’s really a passion, as you said, and it’s important. And not only is it environmentally friendly, it’s great for the storytellers, because there are so many times when you have ambition, and the time doesn’t match the ambition.

Now, you have a choice. I remember an old filmmaker friend of mine said to a director, well, you’ve got two choices. We could drive or we could shoot, which would it be?

[LAUGHTER]

You know, now you can say, well, we don’t have to make that choice. You can be in those five locations. We just don’t have to drive to those five locations.

MARK SCHAEFER: So I was reading about this new effort where Dell, Deloitte, and Intel are collaborating in helping to expand your applications at Innovation Studios beyond entertainment to verticals, like medicine, retail, and education. So connect those dots for me. What’s this about?

GLENN GAINOR: It’s really about being transmedia. Let’s look at it from one perspective. If you’re in a hospital room, and we capture the hospital room with all the equipment that’s necessary and so forth and so on, what we can shoot now a movie, a television show, a commercial with and without the equipment. That’s no problem. But now what about training?

Let’s put somebody who’s practicing medicine who’s learning medicine let’s put him in that, her in that environment. Let’s get people to experience as close a reality as possible. So it’s enterprise solutions. It’s storytelling solutions. It’s as versatile as the real world can be. And that’s why we call it transmedia, because it can live in a VR experience, so a headset experience. You can film in it. You can use it for training purposes, marketing, just like the real world.

It’s just not physically stored. It’s digitally stored, and that’s why the partnership with Dell and Intel, and that’s why the partnership with Deloitte. They understand what transmedia means, and they understand enterprise solutions.

DOUGLAS KARR: That’s incredible. I’m curious what does this look like a decade from now, when you’re going to actually produce a film, is there one set, or a dynamics set that you’re moving around, and then you do the volumetric video, and you’re coming back to that over and over again, and it’s 1/10 the time to record and 1/10 of the budget and everything else?

And then are we going to have volumetric actors? Are we actually going to be able to do this with people in the future?

GLENN GAINOR: The answer to everything is everything’s possible. I mean, we are we really out of time where if you can dream it there’s a way to make it a reality. You want to do what makes sense for the story you’re telling, whatever that story is. Again, if the story is training, you may want to have machine learning give you all the data possible so that somebody can be trained in so many scenarios if it’s scripted productions you want to be able to act in these environments. You may want to have consistent sets across multiple language television shows. Whatever the logic is, that’s the beauty of it.

We’re authoring a big part of the story. But once the technology becomes wider, others will author what it can do. We’re sort of giving them the opportunity.

MARK SCHAEFER: What a fascinating discussion. Glenn, our time is just–

DOUGLAS KARR: My mind is spinning.

MARK SCHAEFER: Mine, too. I’m thinking, I like want to get into this business and tell stories like you, Glenn.

[LAUGHTER]

This is really, really awesome and very inspiring. And you said a lot of good nuggets here that I’m probably going to steal and use as my own, because that’s just how I roll. So Glenn Gainor president Innovation Studios Sony Entertainment Technology. Thank you so much for joining us today. I’ve learned so much. And I’ve been inspired, and I hope you have, too.

We appreciate all of you. We never take you for granted. Thanks for all your kind messages and reviews. This is Mark Schaefer and on behalf of my co-host Doug Karr, we will see you next time on Luminaries.

OUTRO: Luminaries. Talking to the brightest minds in tech, a podcast series from Dell Technologies.