Virtual reality for mobile: How to create realistic environments with millions of polygons

LUIS RIVERO: So welcome everyone. Thank you for coming today. I think there were a lot of parties yesterday. How many people was until 1:00 AM? Can you raise your hand? Until 1:00 AM? Nobody? 2:00 AM? 3:00? 6:00 AM? No? Was he?

OK so what I want you to show you today is [INAUDIBLE] experience about to have millions of polygons in a device, mobile. Having them completely in the cloud. You can go anywhere, anytime with a 4G or a Wi-Fi. You can download a [INAUDIBLE] store, a whole building, anything in just two minutes with millions of polygons.

The lecture is split in two parts. The first part is getting the experience for the company in terms of doing the largest database of retail in the world. We have a big client. That is this guy has-- [INAUDIBLE] this company has around 7,200 stores around the world. And they have to have control about it.

And the second part is we will try to build the methods for lighting and modeling that allows the system to work.

So let's go for getting the experience from the company. For us, reaching the beholder, the [INAUDIBLE] beholder, is key in communication. Let's go with the backing of the company.

Our company is focused on architecture and communication. We did a lot of movies around the world-- Mexico, Australia, whatever. The first thing we do is to study the project needs, what the project needs. It's completely different [INAUDIBLE] the client told you that they need.

When you ask to some client to-- a clientele, for instance-- we have a client with a big plot in Kennedy Island. He told us, OK, I need a movie. Really soon [INAUDIBLE] the hotel and keeping the attraction of how beautiful the design is going to be.

I say, OK, but what's your goal? What do you really need? No, no, I need to solve-- no, no, no. It's not the movie, the final goal. What do you need? OK. I need fundraising. Ah OK, that's the goal.

So which kind of target audience you going to have? And this guy told us, OK, this is going to be a Russian tour. These guys are with the Dolce and Gabbana big logos on their shirt. We have to reach him. We have to keep the attention of these guys. So we need to avoid the typical boring fly through that these guys are not going to pay any attention. And we need to see some [INAUDIBLE] Russian guys. Do you really want to have these guys focus on the movie, focus on the project?

Yeah, yeah, yeah. Really need. OK. So we start a script that you are going to see here that is based on storytelling. Please keep in mind this is only for losing guys. It's not for everybody in the world. So--

[VIDEO PLAYBACK]

[MUSIC PLAYING]

(SINGING) Nobody knows it, but you've got a secret smile and you use it only for me. Nobody knows it but you've got a secret smile, and you use it only for me. So use it and prove it, remove this [INAUDIBLE] sadness. I'm losing, I'm [INAUDIBLE] but you can save me from madness.

[MUSIC PLAYING]

Nobody knows it, but you've got a secret smile and you use it only for me. Nobody knows it, but you've got a secret smile, and you use it only for me. So save me, I'm waiting. I [INAUDIBLE] hear me bleeding. And soothe me, improve me. I'm grieving. I'm barely believing now. Now.

LUIS RIVERO: [INAUDIBLE] this up. There is other guy in the room. The Russian guys [INAUDIBLE] mind.

[MUSIC PLAYING]

(SINGING) I know there's something sacred and free reserved and received by me only. Nobody knows it, but you've got a secret smile, and you use it only for me. Nobody knows it, but you've got a secret smile.

[END PLAYBACK]

LUIS RIVERO: So for us, it's really, really important having a good storytelling is the key to following. All the story is a guide to follow everything around. Because it's maybe an excuse. All the people is watching the girl. But at the end of the movie, the people say, OK, you know what? Fucking awesome. Sorry. But you know what? The hotel looks very well. Where is that? And it start the questions. It start the people asking, OK, and how much do you need for this kind of hotel? Are the girls included?

OK. There were some guys, obviously. So but for us, having very good storytelling is the key for success on the movies. It's not kind of having a fly through movie, because [INAUDIBLE] movies are only focusing the people that wants to see the movie. They want to be instructed about how the product is. But we need to reach people. That is, OK, I don't want you to buy a house. I don't want to buy a hotel. I don't want to develop a visual. But we need to get the attraction of these guys to attend to this.

So for us, this storytelling is not having good looking guys. It's having the typical [INAUDIBLE] the frame we have in the school that was really good guy. So this is the demo reel of our work. So we [INAUDIBLE] different screens from different movies we did in the last three or four years.

[VIDEO PLAYBACK]

LUIS RIVERO: And for us, it's really key being focused in the basic emotions of the people.

[MUSIC PLAYING]

LUIS RIVERO: So the guys who are in parties yesterday, I'm going to turn the volume up.

[MUSIC PLAYING]

[END PLAYBACK]

LUIS RIVERO: So what we learned in the last year is we need to save time and save data. It's like the hard disk-- this is the story from the beginning. I was working in my house when I was 18 years old with a computer that was [INAUDIBLE] 386, and all my colleagues have the 486. They have to save a lot of time and data to have the same quality of rendering and the same quality. I have to improve my techniques constantly to have the same power that all the guys with computers better than mine are going to deliver.

So from that point, we learn a lot of techniques that we have to keep in mind in everything-- saving stories' data, saving lighting techniques, saving rendering time is key to success.

So why are clients need VR? We deliver [INAUDIBLE] movies. And some clients call us and, OK, yes. You know what? But we are having this shopping mall movie that is awesome, but we need to [INAUDIBLE] an exact locale in the middle of the shopping mall then we cannot go there. We [INAUDIBLE] pose the movie. And, OK, you can see this is the [INAUDIBLE] going to lives in that corner and play again. It was a long sentence.

Then they ask, OK, can you do a game or something like that? Then we delivered for-- we were for one year or something testing a lot of things. But what we learned in this year, we start with just one programmer. It wasn't a programmer. It was an [INAUDIBLE]. And the guy told, OK. In the empty time, we have the two projects. We can deliver something.

But suddenly, you have to think about, OK, we need to have a very good system. We got it. But suddenly, one [INAUDIBLE] person [INAUDIBLE] and say, OK, you know what? This is the real world. I have 7,200 stores. We learn a rule.

What is this rule?

[VIDEO PLAYBACK]

[COMMOTION]

- Wait! Wait. Colonel--

LUIS RIVERO: I'm obsessed with movies.

- I'll take action without authorization.

- And what do you think I am, some dung ho stupid son of a bitch?

- No. I don't think you're stupid. I think you're a talented soldier with sloppy impulse control, and I don't want you provoking an international incident because of a personal agenda.

- And my personal agenda is my business. Now you are not in Washington anymore. You are the real world. In the real world, there are nuclear arms heading for Iran. So whether you agree with me or not, get me authorized.

[END PLAYBACK]

LUIS RIVERO: OK. It wasn't nuclear bombs, but it was a no sense client asking us 700-- 7,200 stores in the next years. And we wanted in the cloud. We wanted in everywhere. It was the worst scenario.

Then rule number one, try the worst scenario and don't wait for support. Nobody believes in us, nobody trusts us. OK, are you thinking to have a store with [INAUDIBLE] inside and everything? Impossible.

Then what is the worst? 7,200 stores. [INAUDIBLE] every store were around 96 hours. It's like, OK, meanwhile our office is delivering the projects. We need to check the projects in 96 hours, because it's a lot of time [INAUDIBLE] to do that. And on the other side, you have low cost and you have to have a lot of high quality.

Then we started to test some render engines. The first render engine we tried was, do you know Unity and Real? How many of you guys are working in real time or have any experience? Can you raise your hand? OK. Do you know guys, Unity and Real?

So with Unity and Real, we test both of them with the same model to have the final feedback of what is the best render engine for us. I'm not trying to say this is the truth. It's our experience, our stores, our sort of geometry and scenarios.

So [INAUDIBLE]. OK. You can see. You can watch me in the reflection of the iPad. It's a proof of life or something like that. This is Unity. You can expect to have a very high frame rate. And the only thing [INAUDIBLE] good enough. Because this is an iPad. It wasn't an iPad. [INAUDIBLE] this was two years ago, something like that. You can move smoothly around. And [INAUDIBLE] it was good.

Don't look at the lighting or modeling, because it's coming from the other part. And this is the-- this is me, setting up the display, waiting for the reel to load. You can experience-- had you noticed that? Frame rate is slower. [INAUDIBLE] really badly. And there is lacks of movement. It was completely different.

I was with no smile on my face when I tried this. So we have all of the people telling us, VR headsets, this is the best, is the future. You can listen Mark Zuckerberg telling you, OK, Oculus is going to be in every home next month or something.

So we will learn the rule number 2 that I don't remember. OK, nice. [INAUDIBLE].

[VIDEO PLAYBACK]

- In the '30s, Jews refused to believe they could be sent to concentration camps. In '72, we refused to fathom we'd be massacred in the Olympics. In the month before October 1973, we saw Arab troop movements, and we unanimously agreed they didn't pose a threat.

Well, a month later, the Arab attack almost drove us into the sea. So we decided to make a change.

- Change?

- [INAUDIBLE] If 9 of us with the same information arrived at the exact same conclusion, it's the duty of the 10th man to disagree. No matter how improbable it may seem, the 10th man has to start thinking the assumption of the other 9 are wrong.

[END PLAYBACK]

LUIS RIVERO: So when I listen everyone say, OK, don't you have Oculus or don't you have-- how many people has here a cell phone? Please can you raise your-- iOS phones? iOS? And Android? And the Oculus guys? Oculus guys? Come on. And the [INAUDIBLE], it's-- you have here the [INAUDIBLE]. OK. Do you have the guy? OK. This is the guy.

OK. So rule number 2. When everybody say, this is the way, maybe not. You have to be the 10th man and explore everything around and understand what is happening. VR headsets were here 25 years ago or something like that. I remember the Silicon Graphics big workstation, 200,000 euros 25 years ago or something like that. And it was the same. For me. This is a personal feeling I have.

Some people tell you, you are right, but with a lowered voice. I don't know why. Because Mark Zuckerberg is listening everyone through our cellphones with Facebook or something like that.

But for me, VR headsets has several problems. Our brains release the delay, when you move really, really fast in your head, there is a delay of the display. And our brain says, OK, yes, I'm trying to understand that this built a reality but it starts to a little pain in the back of your head. You cannot support a long time with this, because you have experience-- I remember a lecture one year ago, some guy told us, OK, I tried with my mom. My mom was 80 years old, put a headset and said, OK, it's nice. And she you to use it again? No.

Do you know what? Because you are really tired after the experience. The other parties, there [INAUDIBLE] presence of the-- do you have right there? Oh, OK. You are [INAUDIBLE] guy then. Sorry, sorry.

So nobody has a VR headset here. It's unsellable. Yeah? You have there? OK. OK. You know? Industry has to work for 1% of the people. Or not. Sorry. It's my personal opinion. You know?

So which other things are in the market right now? The augmented reality is working. [INAUDIBLE] but I would wait one year or something to have a call to my clients and telling, do you know-- do you want to try this? When everybody has augmented reality, everything [INAUDIBLE], everything is working. I would try it.

And the other part is [INAUDIBLE], have you seen this? The RED Android cell phone? Had you-- how many people know this, the hologram from RED? RED is the manufacturer of all the cameras we are using to film. And out of the blue, these guys put a [INAUDIBLE] in the website and say, OK, we are going to do a hologram cell phone. Nobody has [INAUDIBLE] this. Nobody knows how it's working. But it's like the Princess Leia stuff, you know?

It's supposed to reach the market next quarter of 2018. It's really, really weird. So we are going to go to experience virtual reality mobile devices with high number of polys. And the other part is, [INAUDIBLE] packets around. There are Real, there are Unity, there are [INAUDIBLE], a lot of stuff. So we learned that rule number three.

[VIDEO PLAYBACK]

- [INAUDIBLE]

[END PLAYBACK]

LUIS RIVERO: OK. This is what I feel when somebody tell me, oh, why don't you move to [INAUDIBLE] or some kind of file with a lot of players, a lot of things like that? I say, but I feel like hitting backpack with all my stuff, and I have to share with somebody that this [INAUDIBLE] or since any other-- how can I say? Because this is really big. Can I use [INAUDIBLE]? I cannot use cloud. It's not flexible.

We need to go to focus to what the audience, what our clients need is right now. That's the reason. Because all we have cell phones and we don't have any kind of [INAUDIBLE].

So for us, because you need something flexible, run away from only one [INAUDIBLE]. So let me show you.

[VIDEO PLAYBACK]

LUIS RIVERO: This is [INAUDIBLE] some [INAUDIBLE].

[MUSIC PLAYING]

After the [INAUDIBLE] anyone that wants to take it is somebody [INAUDIBLE].

This is done automatically. We did only the [INAUDIBLE] and scanning since every [INAUDIBLE] is staggered. In every store around the world, when we update some furniture in some part of the pilot stores, it gets automatically updated in any [INAUDIBLE] world.

We can search by reference. We can do anything. Because we don't need a lot of [INAUDIBLE]. We need the [INAUDIBLE] you need in that moment. We don't need that very cute system [INAUDIBLE] huge capabilities to do anything. [INAUDIBLE] the capabilities you need any time.

[END PLAYBACK]

LUIS RIVERO: So the good part of having not all in one packages is you can do modeler. Since everything is in the cloud, the systems are downloading anything you need for any client. It's smaller. And you can plug it and you can change all the navigation system. For instance, we experienced with some of our clients that suddenly you start to point in the middle of the display. And they say, oh, maybe we have to do something to when the client touch in the middle of the screen, do something. Go forward, move 6 meters, something like that. We updated without any interruption or [INAUDIBLE] in the system. You know?

When you [INAUDIBLE] the typical update your Dropbox or Drive. Nobody does, until you have 25 days in your [INAUDIBLE] can you please update something. Then [INAUDIBLE]. Since it's mobile, since any part of the system is in the cloud it updates automatically without any update.

We need to get track of sales in every store. So we have to connect the data from the sales locally in the country and globally of any provinces. We have target, we can do it. We have three graphics-- one for local, two for the country sales, and third for the global sales.

So when you go to any pilot store, you see, oh, this is selling very well. But in the middle, it looks like the country's not selling very well. And locally and globally it's not selling out. Then you can [INAUDIBLE] any models from big data, from business intelligence, to make suggestions to any manager in any store in the world to, OK, maybe it's time to move this [INAUDIBLE] to any part that is locally selling very well or to other store.

So this is our [INAUDIBLE]. These are the parts of the system. We need cloud. We need the store, and the store has several parts. The first one is the geometry. Geometry is in a library. Since you have to-- you don't want to have the people charging again and again the same furniture in every store they are opening.

So we have a big library of all the furniture and all the products you have, we have in the pilot store, is sharing the geometry depending on the configuration.

So we have to have a good lighting, bake it in the geometry. We have to have the textures, [INAUDIBLE], the library with [INAUDIBLE] placed in the configuration of every store.

And we have some areas to define the cold and the hot areas of the store selling. So we have focusing in [INAUDIBLE]. It's not, OK, it's going to start. I love it. Most of you love [INAUDIBLE] because school. No explanation more than that.

But the graphics performance is skewed. They have really good disability. They have great development support. You can call anytime. They have great support [INAUDIBLE] the school, you know?

And since this is a system made for professionals, for people that is in a Zara store, or in a Prada store, or a Burberry store, they want to have an Apple because school. Because all the clients have.

[VIDEO PLAYBACK]

LUIS RIVERO: This is the system for several clients. This is for Zara, I cannot name it, because it's an NDA. I can show you, but you cannot but Zara on any website, please. And you can have aerial views. And you can take any part of any store around the world.

This was the first system. We have all the parts with historic information of the initial deliveries. This is an office, when you can sense [INAUDIBLE]. You can sense configuration, anything. And the good part is around-- I think that the office is around 100 megas. It's quite easy to [INAUDIBLE] right now in [INAUDIBLE] time. [INAUDIBLE] first recording.

This is the [INAUDIBLE] can take here in the [INAUDIBLE] you have done. And this is a sample of how the updates are working. That the managers of the stores can have very good feedback of, OK, next week is coming this, it's coming these problems. This is the place where I have to put in them.

We have [INAUDIBLE]. You can have anything. And this is modeler. You don't have to go to an all in one package that is going to charge and load a lot of information well prepared for everything. No. You have to have a specific system for any client.

[MUSIC PLAYING]

LUIS RIVERO: We tried this [INAUDIBLE] three notes.

[END PLAYBACK]

LUIS RIVERO: But nobody was-- nobody use it. Because it was a bit confused again, touching any part of the screen and displays. A big note, you can have a conversation with the people that nobody use it. There are several things you learn. And you can unplug it from the system and get a lot of light.

So in this third part, we are going to understand how low level lighting and modeling techniques will help you to achieve high realistic environments. Be aware how all the people here is working 3D, most of them. Most of you.

So do you know what happened where you are working with Revit files or AutoCAD files coming from other guys? Do you know what is [INAUDIBLE] our faces? This is rule number three.

[VIDEO PLAYBACK]

- You're right on time. You pick up special suits.

- Huh?

- Mr. Lee, tailor, makes special suits for you. Come on!

LUIS RIVERO: Really, really good movie. It's nonsense.

- Oh, dude! You got a tattoo!

- So did you, dude!

- No!

- Oh.

- Dude, what does my tattoo say?

- Sweet!

- What about mine?

- Dude, what does mine say?

- Sweet!

- What about mine?

- Dude, what does mine say?

- Sweet!

LUIS RIVERO: This is lighting. But you have [INAUDIBLE]. This is what is happening. Lighting cannot escape from there.

- Sweet! What about mine?

- Dude.

LUIS RIVERO: And this is [INAUDIBLE] waiting for the [INAUDIBLE] to wait. What is wrong with this?

- What does mine say?

- Sweet!

LUIS RIVERO: And that's the client.

- Idiot!

[SPEAKING CHINESE]

Your tattoo says dude. Your tattoo says sweet. Got it?

- Oh, sorry. Hey, [INAUDIBLE]. Sorry.

[END PLAYBACK]

LUIS RIVERO: It's a funny movie. So lighting, by aware of modeling because the coplaner polygons. Coplaner polygons became infinite light bounces and infinite calculations.

So for the system what we needed is, it needs to be lightweight, not using extensive lightmaps. Because devices are handling very, very well geometry. But it's not the same with the lightmaps, because you have millions of polygons and you have to build a lightmap for every million polygons there. You have a lot of troubles in wrapping, baking anything there.

So we store lighting in the most soft light metal. If you remember the inspiration of Silicon graphics and landscape, this render you can see here is real time. It was made 1993. Really old. But what it means is, this technique is in the market in 25 years ago, something like that.

So what's the difference? Is Bertha [INAUDIBLE] baking the light information, instead of having a lot of light mass for everything? Can you imagine having this lamp with a light map there? It's going to be nonsense for mapping there.

And you will have complex unwraps. It's really difficult to do it. So this is real time technique that is called [INAUDIBLE] inside 3D Max. It's in the renter-- default rendering scanner. Nobody use it. I don't know why, but nobody use it. And it's really useful. Because instead of having the lightmap with one pixel every inch, you have only information when you need it.

You can see how the [INAUDIBLE] are restoring the light in only the powers that have a very huge threshold of the lighting information. Since it was recorded in real time, it did it again, because I have to improve the resolution of the minimum size. It's quite easy, because you have maximum size. You have the minimum size you want to have the resolution of the lighting and the [INAUDIBLE] threshold handles how accurate you want to be.

You have the initial [INAUDIBLE]. That is how-- the minimum you want to [INAUDIBLE] the volume. And you have to wait at least two or three minutes, and you have a very good and soft solution of the [INAUDIBLE]. You can refine right there in the light, during lighting.

After this, you had to bake it. And instead of bake it again in a light map, because somebody is doing that, instead of doing that, we are baking in a very good method that is [INAUDIBLE] correlating. And the size of this is really small. That's the point of having a lot of good lighting with millions of polygons in a real time application. This is the method. You [INAUDIBLE]. There is a common script that you can find in 3D Max.

You have to activate participate-- you know, [INAUDIBLE], what it is. And you have-- in real time, you don't have to relight again. You have to render anything else. You have any [INAUDIBLE] with [INAUDIBLE] lighting there.

So obviously, you cannot move it. We are doing some fake stuff, fake silos to move the furniture in real time. But you have the typical ideas-- photometric [INAUDIBLE]. You can use any unit you want. You have aerial lights to have this soft surface around the furniture legs. And the soles, [INAUDIBLE].

So now we are going to get notes. This is the most boring part, because most of you know how to model. But this is how we [INAUDIBLE] between having photogrammetry, or having model.

The first thing we are doing here is, everybody tried to find the good stuff. OK? You know? Something like, I use only photogrammetry, I only use modeling, I only use [INAUDIBLE] or something like that. Rule number five.

[VIDEO PLAYBACK]

- Ow! Oh, boy.

- Oh, sir. You're brilliant! We'll build a motorcycle out of the alleyway.

- Ow! Oh!

LUIS RIVERO: You've got to get the bed method, every specific moment, you know.

- So didn't catch your name or anything about what you're up to or what we're doing here.

- It's brilliant, sir, that you pretended to be a useless nobody. But you can drop the act with me at school.

- Oh, the act. Whoa!

- Jump on. Let's go!

[END PLAYBACK]

LUIS RIVERO: This is the reason, because the directors get fired from Star Wars. This is not the kind of movie that the guys are looking for, a Star Wars movie.

So rule number five, use as many techniques as you need. There isn't the one. It's impossible to find. So we use photogrammetry and mixed photogrammetry. Photogrammetry is-- as you know, you can use ReCap. You can use reality capsules. You can use the [INAUDIBLE] photoscan that is maybe the oldest one.

You have to-- how many people here have done-- it's not the same. Have done, that I know how to [INAUDIBLE] photogrammetry. How many people have done any time photogrammetry? Obviously, [INAUDIBLE]. OK.

Photogrammetry is you have to use a lot of pictures around your object. This is for e-commerce. It's not the same like in architecture, but it's a good sample to address where the trouble is. You have a lot of cloud points. This isn't going to be nonsense for [INAUDIBLE]. This is going to be a big size. And that's the reason, because we have impressive tools in 3D Max to do this.

We can optimize. You can use Seagrass to catch some parts. You can use Macbooks. You can use any 3D modeling software to do this. And when you have our [INAUDIBLE], you can cut the object again to have some parts separated to do a good [INAUDIBLE].

When you are going to come back 2, 3 miles, you can compare the difference between the cloud and photogrammetry direct object and the one you have completely optimized. And you can see here these parts are modeling. I know from the photos you're-- photogrammetry.

When you find some clean volume, like a lamp or some books or anything else, you have to remodel this. This is stupid. You have a lot of [INAUDIBLE] when you have a flat surface.

So you use a rub. You put the pictures, like a texture, in simple polygons. And for that, you have a lot of good [INAUDIBLE] mapping to use, to change anything. It's really, really good. This was for e-commerce.

In the other part, you can model from real world. We are using two techniques. From [INAUDIBLE] lighting, we are using the Radio City [INAUDIBLE] you were watching before. But from modeling the real world, this is a tricky stuff. It's not a very fancy technique or something like that. We use street view. We capture some-- we do a screen capture, for instance, of this light.

We decide which one is the best one. We make a mix of everything, to save some parts of the [INAUDIBLE]. And we remodel from scratch. It's completely different. We are using different lighting from Radio City. And the ambient lighting is coming from the real world. You have in the real world, you don't have to imitate anything.

This is me, the typical day typing in the computer really fast. [INAUDIBLE] two times faster. So there are a lot of techniques really easy to use, low profile techniques. This is an offense. This is not using complex techniques or something like that, and it's only a wrap. A wrap in the model, and you have a perfect light. You can use it for [INAUDIBLE] or whatever.

So that's all. Any questions? Anybody wants to play with the system here? You are going to have a nap. These two guys are going to have an app. So anybody wants to try it? Don't be shy. [INAUDIBLE]. [INAUDIBLE] this. Any questions? Too much party yesterday? Yeah.

[APPLAUSE]

OK, thank you.