Viewing Data form Different Sources in the HoloLens

DAVID MENARD: Well, welcome, first of all, to our HoloLens class, How To HoloLens. I know the title of the class was pretty long. I kind of cut it short. Before we start, I just wanted to ask a question. Who in this room has access to a HoloLens and can actually develop?

That's about half the people. So the other half here is interested in this because, and I'm taking a guess-- I know the HoloLens shows a lot of promise. So you want to know that when the HoloLens 2 comes out or when another device comes out, you can basically develop or create experiences for that next device. Is that something that resonates a bit?

I hope so. Great. Excellent. So my name's David Menard. I'm a product owner for Autodesk Live. I'm based in Montreal. And my co-speaker is Louis Marcoux. He's a subject matter expert, and he's here mostly to help me with the entire optimization workflow.

We also have a guest speaker that I'm going to introduce in a few slides. So the way the class is going to be setup, we're going to do three things. First, we're going to make sure everyone's on par, at the same level with the devices, with the terms that we're going to use. So just general overview about the HoloLens and AR and VR in general.

Then we're going to get right into the grains. We're going to do the hands-on demo, the hands-on part of how to get stuff running on the HoloLens. And so really the how to HoloLens part of this class. Then the second big section is going to be all about optimization workflows. Because, as you know, the HoloLens a bit less powerful of a device, there's a lot to do to get things working smoothly on the device.

All right. So first of all, why use HoloLens? We know that AR Kit now just came out. We know AR is the thing, we know VR is a thing. Why would you ever want to use the HoloLens in the first place? Well, I stole this slide from Nick Landry. He's a Technical Evangelist on Microsoft.

They coined the term "mixed reality." So today, on this slide, we have VR as it is today. Everyone kind of knows that part. You put the headset on, you disappear into your own world. You have no idea what's happening on the outside world. You're really in the experience. Nothing else is happening.

And on this slide, we have AR as it is today. Most of AR today is just overlays on the real world. You have some panels here and there, but then there's the entire spectrum of mixed reality that comes in-between. And this is where the HoloLens is actually showing the most promise. The HoloLens is something that can do AR today, but is really designed to do mixed reality in the future where you can not only overlay virtual things to the real world, but you can have your virtual objects interact with the real world in different contexts.

So I know some people hate the mixed reality term. And just to expand on that a bit, I want to invite Dace Campbell with me. He's our guest speaker, so please giving a hand. This was very unpredicted. They saw you introduce yourself. And just give a--

DACE CAMPBELL: OK.

DAVID MENARD: --one minute talk.

DACE CAMPBELL: Sure, sure. So I'm Dace Campbell. I'm a customer success manager with Autodesk. I have a pretty deep background, and AR and VR. I've been in it for more than 25 years. So as a CSM, I work with our major accounts and I get deeply embedded helping them use the tools that they bought.

My night job, however, is working with Louis and David and a whole crew of people at Autodesk, helping define the AR and VR strategy, specifically the AEC. I just want to speak just a quick minute-- and I'll go back to the Microsoft slide here-- I'm a big fan of the HoloLens. Big fan of what Microsoft is offering. I'm not such a big fan of their attempt to redefine the terminology to fit their current offerings.

And I'm going to get into this here just a little bit. They're defining AR on the physical reality side when typically, AR tends to fall somewhere in the middle. And they're actually putting in a transparency opacity along the same axis, but they're using terms like immersive along the same access. To me, it's a bit more orthogonal.

And not to geek out too much, but this hearkens back to the original MR diagram from 1994, then was put together to say there's a mixed reality continuum. On one side, you've got reality. On the other side, you've got true VR. Completely immersive.

Immersive-- think of immersive as surrounding. If immersive equals surrounding, then there's different points in between. So typically, AR falls somewhere in between true VR and true reality that we're existing in today. We don't really talk-- I'm sorry, there's AR.

We don't really talk a whole lot about augmented virtuality. That's sort of like VR with windows into reality. And those could be visual, tactile, a little bit of feedback that's maybe 80%, 90% VR, and 10%, 20% real world. There's not much there today.

Anyway, it's an entire spectrum, but I would argue that along that entire spectrum, things could be immersive or surrounding or non-immersive. So it's not a matter of, oh, some of this is immersive, some of this isn't. You can have immersive offerings and not immersive offerings along the whole continuum. And for example-- and I know David's going to talk probably a little bit about this more-- we've got Revit Live.

Show of hands, who's heard of Revit Live? Show of hands, who hasn't heard of Revit Live? Where have you guys been?

DAVID MENARD: Awesome. That's like--

DACE CAMPBELL: [INAUDIBLE] people. You've been in AU for days now. Anyway, so Revit Live supports a VR experience. And it could be an immersive, head-mounted, display-driven experience, or a non-immersive, desktop, VR experience. That's just one example of how to differentiate immersive versus a non-immersive on any points on that mixed reality continuum.

DAVID MENARD: Excellent. Thank you. This is great. And the reason I wanted Dace to give his opinion on this is because the space is brand new, as you know. But it's so contested right now. Everyone kind of wants their own thing and everyone thinks it's the future.

And we're really looking at a future where you want that pair of lenses that just does pretty much everything on this spectrum, right? So just to keep going-- by the way, there's a lot of new people who just entered the room. I'm just going to recap the agenda real fast.

We're doing a quick overview on the HoloLens, AR, VR, everything there. Then we're going to get hands-on with how to actually use the HoloLens with Max Interactive. And then Louis' going to do a whole, big spiel about optimizing your content for this kind of device.

So to get back to this, AR core and AR kit just came out. Obviously, two great technologies. One for iOS, one for Android. They do some things really, really well, mostly plane mapping. So detecting floors, detecting walls, top notch, right?

They do GPS localization because your phone has a GPS. So that's fine. It's accessible. They have all the tech that you already have in your phone accessible in these devices. To be fair, though, the plane mapping coupled with the GPS? Not so strong today.

I'm sure some of you have tried these apps that try to put those-- you know those drawings in real space and then someone else can come and see them? They're never at the right space.

So on the other hand, HoloLens. The first advantage is that it's pretty freaking awesome. Not a real advantage, but I wanted to put it there. There's a lot of stuff that HoloLens has going for it, but the big one is the environmental mapping.

So first time I got my HoloLens, I booked a meeting room at the office in Montreal. And they started placing windows everywhere in the meeting room. I placed holograms and it was just a mess. And then I left the meeting room, went back to my desk, and decided to test this for myself-- the environment mapping.

So I went around the office placing holograms everywhere. And the second round I did, all my holograms were still at the right place, which is great. But the second round, I actually went by that first meeting room where I set all my windows, and the HoloLens, without me even going there, and between two different sessions, completely recognized that that was the place I was before and placed everything at the right place. And this is really the power of the HoloLens.

It knows where you are, it can recognize your environment, and it can create meshes in real-time for you to interact with so that your chair that you put behind your table-- your virtual chair that you put behind your real table. Now I have to specify these kind of things-- it can be included in your virtual experience by the HoloLens. So that kind of power is not yet available in the AR Kit or the AR Core.

Of course, then, it's hands-free. It's great for collaboration because it's hands-free, because everyone can have one. All of those advantages are probably going to come, eventually, to the rest of the platform.

So the overall workflow that we're going to look at today is taking your source data-- and that can be pretty much anything. Revit, Fusion, CAD, FVX, whatever. Then we need to optimize it because this source data, especially the CAD data, is incredibly heavy. And we'll see that the device is pretty limited.

Then we want to take that data that we optimized, and we want to push it to a real-time engine-- that can be, again, anything. Unity, Unreal, Max Interactive-- to be able to push that or deploy it to our device. Simple workflow in theory. And we're going to look at all these steps.

So before we get started, why is optimization needed? You probably just want to take your CAD data, shove it in a HoloLens, and view it, right? That's the simplest thing. Optimization is needed because the HoloLens right now, as it sits today, is the equivalent of an iPhone 6.

The iPhone X just came out. It's already three generations up. The iPhone 6-- imagine it has to do all this environment mapping at the same time. So your budget for computing, or to draw things, is very limited on the HoloLens. And you might be thinking right now, well, OK, great. But I'll wait for the HoloLens 2 and I'll have all of that power on my face.

But that's just simply not true anymore. And I want to quote John Carmack here. And I'm a big fan of his. I'll show about 20 seconds of his video before going on.

JOHN CARMACK: And I-- some people would like to think that this type of disciplined coding and design within very tight constraints that, maybe if-- like in the old days, you could just wait a few years and I-- and your lazy design, whatever, just works on the new computers. That was the way of PC development for a very long time.

But we're in a situation now, with mobile being important, and with the end of Moore's law kind of drawing nigh, that those types of skills are absolutely going to remain important for the foreseeable future. This future device that we all imagine where we have AR sunglasses that billions of people are wearing, that's probably going to have at least as tight of a power budget and design constraints as you have on gear VR.

DAVID MENARD: All right. Who knows who this guy is? Who doesn't know who this guy is? A lot of people. Wow. This guy's got-- this is good.

This guy's John Carmack, the inventor of the Oculus, essentially. This was at Oculus Connect just a couple months ago, so it's not outdated at all. And the skill set that he was just talking about is the optimization part of your pipeline. It is that important.

So we all imagine a future where you're going to have a GTX 1080 on your face or something as powerful as that. That's just not going to happen in the foreseeable future. So the optimization part of this class is incredibly important. And I can't stress that enough.

So let's talk about that. Let's go back to the HoloLens. We talked conceptually about what was going to happen. The HoloLens today can handle this: 300 batches, 300,000 polygons. Well, you're probably wondering now how big that is.

Well, this is Oakwood Hospital. It's a data set that I use a lot during my work. It's a big hospital campus. When we bring this model into a real-time engine, it comes up with 14 million polygons.

Remember, we're trying to hit 300,000. It comes in with 7,000 batches, and as soon as you add one light to this scene, it doubles everything. 15,000 batches, that's 31 million polys.

Now I work in Live. This is the only time I'm going to mention Live. Live, we optimize a lot of stuff on the cloud, and yet we can only get this down to 5,000,000 polys and 1,300 batches. Louis is going to talk a bit more about what batches and primitives are really concretely.

I just wanted to give you a sense of scale. Remember, 300, 300. We're a long way off.

All right. So let's get down to-- oh. I did something. There we go. Let's get hands-down on how to get this kind of data into the HoloLens before even optimizing it. Let's start with simple stuff-- cubes, lamps, that kind of thing.

Now I want to give you a big experimental feature warning here because we're going to use Max Interactive 2.0, or Stingray 2.0. By the way, same thing. Just a different name change. All of this is experimental.

Not only is the platform experimental, the way of getting your data onto the HoloLens is also experimental. In the past, to get your data onto this device, you absolutely needed to use Visual Studio. It's a programming tool.

If you're not a programmer, you never want to see that, by the way. It's sometimes terrible. It's sometimes great, as well. But we're trying to avoid all of that. We're trying to make this a bit easier.

And before I start this part, because it's going to be a part with lots of tools and lots of things, everything's available here. All the class handouts that I uploaded to the AU website are just one-pagers or one-liners, and they just contain this link.

This link contains-- it's a box folder with all our videos, all our zip files, all our material that is really relevant. The AU web site wouldn't let me upload 600 megabytes, so everything's there, including-- I think this is going to be recorded, so I'm going to upload the recording there as soon as I can as well.

So on the box folder, the first thing I did is I uploaded a zip file there. I called it the HoloLens tool kit. This is just how I called it, by the way, so don't quote me on that. It's a zip that contains a few tools for you to be able to get going. The first thing you want to do is unzip it to a local folder.

And you'll see it contains two folders, one called Builds, and one called The HoloLens Project. The HoloLens Project is pretty straightforward. It's something you open in Max Interactive. It's a template project.

The second, Builds, contains two other folders, and we'll see what that's all about very soon. So the way game engines works, or interactive engines works. My PC needs to be able to compile the data so the meshes, the scripts, everything for each platform I'm going to deploy to. Usually, this is not a problem because you're deploying to PC.

But let's say I want to deploy to Android or to UWP, in this case, Universal Windows Platform, which is the HoloLens platform. I need my engine on Windows to be able to compile that data for that platform. Now because it's experimental, we did not ship this capability with the engine. You usually need to compile the engine yourself.

What I did is I compiled it for you so that you can use it yourself back at home. So first thing you need to do is copy that little part of the engine where you installed Max Interactive. Usually, Max interactive is installed in C, Program Files, Autodesk, 3ds Max Interactive.

And in that folder-- this one here-- there's an engine folder. Just open that and then go back to the Builds folder that is in the HoloLens Toolkit. There's a UWP 32 folder. Just copy it over. That's the first step.

Now your engine, or your editor, whatever, Max Interactive, on your PC is going to be able to compile your data for the HoloLens. So now that that's done, we're going to look at a couple tools that Microsoft hands you out just to simplify the process.

The first one is from the Microsoft store. Simply type in "HoloLens" in the search bar. There's an app called Microsoft HoloLens. It's very, very useful to transfer data. For us, we're mostly going to use it so that you could see what I see through the HoloLens. So this is very useful to give classes or just to record videos if you're ever looking for them.

So to use this app, you need two things. First of all, your log in on your HoloLens. The second one is the IP address of your HoloLens. Now you need to make sure that your HoloLens is on the same Wi-Fi as your PC. And we'll get to how you get that IP address right away, but I want to start the stream first.

And it's very important to be on the same Wi-Fi because I've had people be on different Wi-Fi's and tell me, why is this not working? Obviously, well, if you're not on the same Wi-Fi and you enter an IP address, it just won't work. So make sure that's working.

As soon as you click Live Stream here, you'll be able to get a view of what I'm seeing through the HoloLens. Now this is super useful because now I can show you exactly what to do inside the HoloLens. So first thing's first, we're going to enable Developer Mode. Oh, I can turn off the sound here.

Yeah. OK, I'll leave the sound. Hopefully this doesn't get annoying.

You go in Settings, Update in Settings. By the way, who's used a HoloLens before? OK, most of you. So you know that the Bloom gesture is kind of your main menu, and then your air click is how you select things. That's what you're going to see me do, so don't worry.

You go in Update in Settings. There's a panel on the left called For Developers. First thing you want to do is open Activate Developer Mode right there. And at the bottom-- this is important, as well-- Device Portal. This will allow you to connect to your HoloLens via web browser, and we're going to see that right away.

Then you want to go in Network and Settings. Make sure you connect to the Wi-Fi. In Advanced Settings right there, there's one field called IPv4. IPv4 address. Right there in the center, that's your IP address that I used to connect.

And save this. Mark it down. You're going to use it a lot.

OK. Now that that's done, we're going to get back to it. Now that that's done, we want to be able to install the engine, or Max Interactive, on our HoloLens. To do this, we're simply going to connect to it via our web browser.

So you open a browser page and, once again, enter that IP. Second time we were using it, right? My IP address.

Another tool that Microsoft gives you and lets you play around with-- a lot of different, really useful things here. The only thing we're going to use at this point is the apps section on the left. From there, you can install the HoloLens app.

The HoloLens app, you're going to find it in the folder that I uploaded earlier, the zip file. The second folder next to UWP is called APPX. Simply in the install app section right here, click on Choose File. We're going to go browse to wherever you unzipped the Builds folder, the APPX folder.

APPX is just the extension for the Universal Windows Platform files. So you install this onto the HoloLens. Click on Go, and it's going to get deployed. Now at this point, you have everything connected. You can stream your view to the HoloLens. And when you Bloom, suddenly, you're going to have a 3ds Max Interactive app.

You launch this, and it's going to say you pin it somewhere because you always have to pin your apps. You're going to see a Waiting for Connections tab at this point. So this is where we're actually going to start going into Max Interactive and create our experience.

So now our HoloLens lens is basically all set up so that we can start connecting to it. So in 3ds Max Interactive, first thing we do is, always, you open a project. I mentioned earlier the zip file also contains a template project, so the HoloLens projects. Simply browse to wherever you extracted this and opened the project.

The project is a template to help you get going. It does not contain everything you can do in the HoloLens, but it's a really good starting point. So in the content levels folder, there's an empty level. And notice the first difference with this level compared to a normal Stingray, Max Interactive, level is-- that never gets old-- is that there's no environment.

If you have an environment, you're not going to have a see-through thing. You really want to keep this as empty as you can. It contains three cubes. Pretty simple.

At this point, we're just going to go activate experimental settings. For those of you who have already done that, file, settings, editor settings, and you make sure that that second checkbox right there is checked, and then you're good to go. This is the last configuration thing that we're going to do.

All right. So we're good to go. We have our template. The template contains a few Flow nodes that I wrote to help you get going. Notice the first one is a HoloLens hand-pressed. So it just triggers whenever you do this. The second one is HoloLens Throw Ball.

So I hooked them together, which means that whenever I click on something, whenever air click, a ball is going to be thrown right in front of me. Pretty simple application, but it's a good starting point. I encourage everyone to start there before trying to get more data into the HoloLens.

Once again, I'm going to open the Connections tab. I'm going to create a new platform. You just click on the plus. You change the Windows part to UWP. If you don't see UWP, it's because you didn't enable Experimental Features.

At this point-- last time we're going to use the IP, I promise-- you enter your IP address once more here. Now you're all set up.

So when that's done, you make sure you go back to the HoloLens, you run that application, the Stingray application. When it says "Waiting for connection," this is when you're good to go. You click on Run Game inside Max Interactive right there, and it's going to start streaming data towards your HoloLens. And this is a huge advantage, actually. It really gets the iteration time down compared to using Visual Studio where you always have to copy your files over.

This is the real big advantage here of this new experimental workflow. So once you're there, everything loads. My three cubes are in my scene. I can look around. And you'll see me pinch, and all the balls will start being thrown. It didn't recognize my first pinches.

All right. So here, my balls are obviously interacting with the virtual world. They're not bouncing on the physical world. The reason that is is that I did not set up the mesh for the HoloLens.

In the template, there's a couple lines in Lua that you can just uncomment to actually create the mesh for the physical world to interact with it. In this case, we're just going to ignore that for now. Just time constraints.

So next thing we want to do is to get some kind of data in here. It's fun playing with balls, but we want something else. So first thing I'm going to show you is actually a CAD model of our office in Montreal. One of the small parts of it is an electrical panel. And we never know what's behind that electrical panel.

So what I want to do is have this electrical panel show up in the real world on top of the real electrical panel so I can see what's behind it. So you saw me right there go into Max, export it to FBX. And once the FBX is exported, simply drag and drop it into 3ds Max Interactive like anything else. And at this point, you just drag it in the scene.

Now it comes in really, really big at this point because I did not set the units properly when I exported the FBX file. I just chose the wrong parameters. And this is really important because you want everything at life scale. You want everything at the exact, precise scale, or else your entire experience is ruined.

So here, I'm just going to scale it down arbitrarily to what I think the panel is. I'm also going to change the material. Dim materials in the HoloLens don't work very well. You're always going to see through them. So try and use very bright materials as much as you can.

In this case, I'm just changing it to a bright green. The material doesn't really matter to me. I just want to make sure I can see this panel very well because wherever this panel is in our office in Montreal is very dark. So I want to make sure I can get that contrast.

At this point, I select the unit. I replace it in my unit Flow. And I'm just going to change the hookup for the hand-pressed from the throw ball into the place unit. And I'm going to set the distance to 3.

This just means that every time I pinch in the HoloLens, it's going to place my panel right in front of me at 3 meters. I also put a little marker at the same place so that I can see where I'm going to place the panel before I place it. This is super useful in many, many cases, especially when you want to overlay like this.

So at this point I just restart the app. Waiting for the connection. I click on Play Game, and I'm good to go.

Once the data is done streaming, you're going to see through this all the cubes will still appear at this right place. My marker is there, notice that's new. And whenever I click, my panel appears at my marker. And then you're going to be able to just walk around that panel and view it in context.

So in this case, I could just walk up on my panel, place it, and give my HoloLens to anyone. Or if you've attended other classes on Flow or on Max Interactive, you know the possibilities behind this. But we're not going to get too much into Flow today.

So next use case is going to be from Fusion. Now you might know that Fusion doesn't allow you to actually export to FBX directly. You have to go through the web portal. But again, the standard workflow is you're going to export to FBX and you're going to import into Max Interactive. Now, the advantage with the FBX export here from Fusion is that in Fusion, everything is going to be exactly to scale.

So when I import it into Max Interactive and I place it at my level, I can expect that this lamp will be exactly in the right dimensions and the right measurements for the world I'm going to place it in. So once I'm in Max Interactive, again, you place it in the level. I'm just going to switch the Flow notes once again to my new unit.

Change the distance to one meter. Go back to my HoloLens. Just to make sure, I click. And I make sure that the "Waiting for connection" tab is open, then click on Play Game, wait for the data stream.

Now, pro tip. Whenever you place your window, your Stingray window, in the HoloLens, if you click on it right away, you're just going to start at the previous point where you were.

So if I just open it right away, I'm going to be wherever my electric panel was. You don't want that. So just make sure you close the window and relaunch it. It just makes sure that you start fresh and that nothing weird happens.

So here's the connection. At this point, once the data stream, I'll be able to place my lamp anywhere in my world. And if I actually activate my mesh in the HoloLens, you can actually place it and interact with the physical world.

In this case, I'm going to place it on my desk, and right away, you see that all my dimensions were completely wrong in Fusion. That's because I don't know how to Fusion. So I have to go back to my original content creation tool and probably adjust some dimensions because I really don't want it this big.

So that's the basic idea of the workflow. While I made this class, I was writing a lot of Louis' scripts, I was writing a lot of Flow nodes, I was experimenting a lot because this was fresh off the press. You can ask one of our developers right there. I kind of see him. I was bugging him a lot.

So I came to a lot of pro tips during this entire thing. I'm not going to go through them right now just because we're short on time, but I'm going to upload this presentation. All the pro tips are in the comments section. So there's five or six things just to keep in mind to keep your workflow smooth. But that's it.

So now you're actually good to go to be able to get your data onto the HoloLens without using Visual Studio. That's kind of the big, first hurdle. Now for the second part of the workflow, the optimization part, I just want to pass it to Louis, who is the subject matter expert on this.

LOUIS MARCOUX: I am not the subject matter expert. I am the necessary evil part of the presentation because one of the thing about the HoloLens, about VR, AR is you are trying to render something in real-time and you are doing it for a limited device. And the first time that David approached me to do this presentation, he asked me to do a little project for the HoloLens.

And I'm used to the HTC Vive, I'm used to the Oculus, and I'm running it on big computers. And when he told me 300,000 polygons and 300 batches, I said, what can we do with this? There's nothing we can do with this. So it's a very limited device.

But if you think about it from the beginning, and you plan carefully how you can render to this device, you will be able to do a lot more. So the optimization part-- when I have to do projects and Chris and I work on a project together recently, and I've worked with some of you as well-- the first thing we do is we try to optimize as much as we can so that we can run something that looks beautiful and it runs in real-time.

So optimization is part of the process. We need to sit down. We need to do it. It's necessary. I know it's evil, it's boring, but you have to do it.

And every time I do a project, I have to ask myself questions. And the questions that I ask myself are the ones that I'm going to share here today. And the way that I do my presentation is a bit like David. I'm going to show you a bit of theory, then show you practical examples, and you're going to see the thinking process.

So 300,000 polygons, 300 batches. I start by saying, welcome to the world of polygons. Games, when we build anything for real-time experience, even if it's not VR, if it's just an application for the-- everything happens in polygons. So you're going hear me talk about polygons.

We are working for a specific target. So when we build for the audience, we have to know the limitation of the HoloLens. Same thing if you build something in VR. You need to know the computer that you're going to run it on, you need to know the device that you're going to run it on, you need to know the specs and what can you achieve with this specific target.

So the target is very important. You don't develop the same way for the HoloLens as for something very powerful like if you have a GTX or a P6000 in SLI. And if you have all the gadgets, you don't develop the same way. So understanding your target is very important.

And the necessary-- I have to practice this word-- necessary evil is the optimization part, which I'm going to talk in a second. So when we're building something for a device, what we're doing in our workflow here is we are actually building an application. And the application is machine language, it's compiled, it's binary, and it's specific to that device.

So if you're compiling for Windows, you're going to build an EXE. It's going to have a bit of DLL's. And that's what you're building for doing the Vive or the Oculus.

If you build for the iPhone or the Android, you're going to build an IPA or a APK, and those will be specific applications that will run on these devices. For the HoloLens, it's the same thing.

As you showed a bit before, we are building an APPX file, or application for the HoloLens. And we send it on the HoloLens, and that's what gets run on the HoloLens.

How do we generate these things? As David mentioned, you can use Visual Studio, you write C++ code, and you compile it, and it becomes an application that can run on those devices. I'm a 3D person. Well actually, I'm lying.

I graduated in computer engineering. I did C++. I loved it at the time, but when I discovered the world of 3D, I'm excited about it. And it's much more fun to work in 3D than in code. But if you want to build with 3D objects or with CAD data, you need to use engines that speaks this language.

So Unreal and Unity are very popular. You're all familiar with those engines out there. That's what they do, as well. So Unreal and Unity, you open up your project, you bring 3D data, you trigger, you create interactions and all of that. And once it's ready, you package it, you compile it, and it becomes an app that is completely independent of the engine.

Today, we're using Max Interactive, or formerly known as Stingray. But this is the same type of platform. This is the same type of engine. The engine is there to bridge between compiling visual C++ code or C++ code and a 3D artist that has to build 3D content and speaks that language.

So the engine sits in the middle. We bring 3D data, we compile it, we build some interactions, and then compile something for the targeted device. To feed the engines with data, we use a format called FBX. FBX is the standard kind of thing where Unreal will load FBX file, Unity will load FBX file.

So the things that I'm doing today for optimizing, if you're using Unreal or Unity, same type of thinking process. So it's always the same thing. The reason why we use FBX is that it contains everything we need for the real-time part.

So it contains the polling-- well, it connects to all of them. So it contains the polygon part-- the models, the geometry, the primitives, as you call them. So the 3D model are in there. We also have the mapping, which defines how bitmaps or materials will be applied to those objects.

It also contains the material definition. And if you're doing any type of animation, it also carries with FBX. So that's the format that we use to carry data to the game engine.

The best tools, or the most powerful tool to generate data or to generate the FBX, are the tools that allow you to work with polygons, to work with mapping, to work with all these things that allow you to prepare the 3D data for the FBX format or for the game engine. So that's why I'm putting Max and Maya here. But if you like Cinema 4d or all of these 3D packages out there, that's the same kind of idea.

At Autodesk, we have 3ds Max and Maya, and I know you want to know which one is the best between the two. And I'm not going to answer that. But I think they're both very capable. And if you're a Max user, you're going to feel comfortable using Max to do polygons. And if you're a Maya user, the same type of thinking process applies.

But if you bring CAD data-- I wanted to mention that this is-- in India, when you are in Max or Maya when you are doing the DCC part, this is where you do most of the work. So this is where you're going to make a lot of decisions. And all of the concepts I'm talking to you about today, this is where you want to do this.

And those things that we're going to talk about is organize your files so that it better manage, do some movies, do some mapping, LOD's, level of detail. So I'm going to talk about all of that, but this is where the core of the work-- and that's why you asked me to come here and talk about this. But this is where we're going to do most of the work.

But if you bring CAD data that are presented here, 3ds Max has a lot of importers to import Revit file, import Inventor files, and we can import a lot of data inside of 3ds Max and convert it to polygons and work in the world of polygons inside of 3ds Max. Either you within the Autodesk family of things, or if you're outside of the family, all of this can be loaded inside of 3ds Max.

So we can load this data, and once it's in 3ds Max, this is where we do the work of converting it to polygons. I must say-- and this is very true-- if you're using Alias in any sort of way and you wanted to convert to FBX, I think that Maya is the best tool for that because of history of the product. You have a really tight connection with Alias, so I think that if you're going the route of bringing Alias data into AR or VR, I would strongly recommend Maya because it's the same kind of core at the core. So whatever.

I'm French, French Canadian, and I translate everything as I go. And sometimes the translator is not following with what I'm trying to say, so that's why I-- and Dave, you have the--

AUDIENCE: [INAUDIBLE]

LOUIS MARCOUX: Yeah. So we've defined a target. This is where we want to end in the end, but the whole process of the big part of the work is really in the optimization, like I said. So the main three concept, or the main three things that I think you have to worry about when you are optimizing data, is the culling, the draw calls, or as David call it as a programmer, batches, and computation.

So those are the three main concept that we have to think about. And I'm going to define them and talk about them in a second.

Culling. What is culling? So culling is the process of defining what is visible and what is not visible on-screen. When something is drawn on the screen, it takes time. Big surprise. So what we want to draw on screen is what's going to be the most rewarding when we look at our content.

So the culling process is to define what we're going to see and keep it to the minimum. Yes, there are some processes in the game engine that will automate with conditions that will define, is it supposed to be visible? Is this not supposed to be visible?

So we can write a few lines of code or we can define a few parameters that will allow the computer to decide if it's visible or not, even if-- it still takes a little bit of time. So we need to care to think about that when we prepare data for real-time. But the best way of making sure that we save time is by discarding everything that we don't need in the final rendering, and also a way to organize our data so that we can turn on and off different parts of the data at different moment in time in the experience.

So that's the idea of culling. So if we look at the workflow that we had at the beginning, there is a big part of this workflow where you have to decide what's going to be called out. So that's very important. This is where the work happens. But at the end of the process, when you're generating the application itself, the game engine, or the AI, or the artificial intelligence, will also do its part to make sure that not everything gets drawn on-screen.

But like I said, if you look at the part here, there's a whole part of the workflow where you can make decisions about how to optimize all of this. So if you think about your original CAD application, and like I said, I try to keep it as generic as possible. So whatever the CAD application that you use, the same principle applies.

So the first thing is narrow down the scope. So if you want to render a hospital, yes, you can render a hospital, but think about it very strategically so that you can organize it so that it can be optimized during the pipeline.

Export the minimum. A good example of that is, let's say that we have a Revit model. And inside of the Revit model, behind the walls, we have wires, we have electricity, we have plumbing, we have a lot of things that we are not going to see in the experience that we want to create. So don't bring this into the real-time pipeline.