Using Stingray and Forge to Create Powerful AR and VR Applications

BEN SLAPCOFF: All right, so, hello everyone, and welcome to our talk. Today, we'll be talking about Max Interactive, and the Forge AR/VR Toolkit, or how to get your Forge data in real-time renderers, and game engines like Max Interactive. So today, I'm presenting along with Paul Kind, who'll give you an overview of what Max Interactive is, and Cyrille Fauvel, who'll show you a demo of the Forge AR/VR Toolkit on the HoloLens.

My name is Ben Slapcoff, and I'll be giving you an overview of Max Interactive's stereo rendering pipeline that we use for both VR and AR, as well as a deep dive into how we're working with the Forge AR/VR Toolkit. So to start off, I'll hand it off to Paul, who'll give you the overview for Max Interactive.

PAUL KIND: OK, so I'm going to go through what 3ds Max Interactive is and what it might mean to you. So as a brief history, we'll go through that first. So you might have heard a lot of different terms around the idea of Max Interactive. You might have heard Stingray, you might have heard Bit Squid, you might have heard Max Interactive.

They're all effectively the same thing. What happened was we had acquired Bit Squid in 2014, that was massively rewritten, all the tools were rewritten, and made into one very nice interface-- that interface was then known as Stingray. Stingray was then bundled with 3ds Max, and it became known as Max Interactive. But they're all effectively the same thing. They're all running the same codebase.

And in this presentation, we're going to be talking about Max Interactive. I may get confused at time to time and say Stingray. Just sub in the word, Max Interactive, and we'll all be on the same page. So first we're going to start with what is a game engine? Well, in the early days, game engines were effectively a renderer with some basic infrastructure. That is long gone. Game engines have become much more complicated.

We've gotten all sorts of tools added to that pipeline. But at their core, a game engine is a real time renderer, and they have a bunch of tools that feed that real-time rendering engine, and give you massive capability. Some of those capabilities are going to be rendering audio tools, animation tools, interactive tools, physics, AI, and the list goes on. And they're being added to all the time. So that's effectively what a game engine is, in a very broad overview.

And just to give some credibility to Max Interactive or Stingray, these are some wonderful games, some of which I've lost way too many hours to. On the top left, we have Helldivers, which is a four player co-op game, where multiple people go to battle, and it's a top-down game. We have another game on the top right, which is called Vermintide. It's of the Warhammer franchise, wonderful game, first-person hack and slash, which is really interesting. On the bottom left, we have almost like a old-world first-person shooter, and then on the bottom right, we have a puzzle game.

The important takeaway here is that all of them are dramatically different, and they all came from the same codebase. That's really, really valuable and very important, and I'll explain why in a little bit. So what does the engine look like, and what does it feel like when you're working with it? Well, here we can see the interface. And it immediately might feel comfortable and very familiar. That was by design, and I think it's actually, of all the engines I've ever used, it's the most comfortable at first startup.

So we have some basic tool kits on the left like you might find in Maya or Max. On the top right, we have the list of the items that are in the level currently. On the bottom, we have a browser that shows all the content that's grabbable by this project. In the center bottom, we have a preview, which if you select an object, you'll be able to really rapidly preview it. You can also preview animations, all sorts of things in there, materials, everything. And it's all immediate, so as soon as you click it, you'll get that review, which is really quite helpful.

Then on the bottom right, we have a contextually based property set, which you can edit, but it all happens immediately. So you select an object, you see the properties for that object, you select the light, you see the properties for that light, makes it very comfortable and easy to use. There's no massive numbers of windows and things you have to dig through to make something happen. It's all right in the singular interface, which makes it very comfortable.

So what are some use cases for game engines outside of games? I mean, we're not all gamers here. Maybe not everyone's interested in games but we're all realizing that game engines of this thing, we kind of have to pay attention to. So what can we do with them? Well, we can do real-time visualization, we can do cinema and movies.

Movies more and more adopting this technology. We have animation that we can do. We can do serious games, which is almost like a whole different thing by itself. We have simulation for training exercises and all sorts of things like that. And with AR and VR, that domain is just blowing up. I mean, everyday, these creative people that are in this room are adding to that concept, and making it more, and more, and more, and I cannot imagine where tomorrow is going to take it.

So just a few of the use cases. So what were some of the considerations? Because the engine markets were relatively full. There's a lot of engines out there. Why would you want to select Max Interactive? Why did we select Max Interactive? What was the thing that made it interesting and comfortable for Autodesk?

Well, first of all, it's extremely lightweight, and I mean super lightweight, which is a really big benefit. It's super high performance. In fact, I would recommend anyone here, if you're using a game engine currently, please load up a model in Max Interactive, and load that same model up in the engine of your choice. And give them a frames per second check, I'm pretty sure you're going to find the Max Interactive is going to win.

And what that means is that you can do more. And you can bring more to your engine before you start hitting performance limits. A big win, especially when you're dealing with devices like an Android, where you've got very low amounts of power to work with. So frame rate is hugely important. In fact, I would argue that frame rate is probably the king of all kings in the game engine world. If you don't have the frame rate, you don't have a real-time engine. It's called the real-time engine for a reason. If it's not real-time, it's not real-time. So it's very easy to learn.

We have two different things that we've done. It's like a multi-pronged attack, I guess you could say. We've got Flow, which-- if you are not a program-- I'm not a programmer, that's a fact. But you can make full interactive experiences without knowing how to program. You can do Lua scripting, so that's actually the course that I took. I started with Flow script. I was doing some really nice stuff, started to realize, well, I'd love to make this just a little more performance, or take it to the next level. And I wanted to make these nodes that would do things for me, that I wasn't able to do before. So by learning a little bit of Lua script, I was able to create nodes.

And I actually now have a full robust library that I have made available online. So people are downloading my tools, and are using them in their engine, and are immediately able to do more than what the engine came with. And it didn't take a massive programmer to do that. Like I said, I'm not a programmer. I'm an artist, that's where my background is. But I was able to make a whole bunch of really useful and flexible nodes.

So that path then might lead you to the C route, and we have a full C API, which if you're really a gung-ho programmer, and that's your calling, you can just take the engine to all different levels. OK, so what are some of the benefits here? Well, it's lightweight, it's easy to maintain. It's got blazing fast performance.

It is game agnostic. I didn't touch on that but it's really important. So many engines start you with a first-person shooter, or a certain game that they're trying to have you work from. And then you've got to spend time undoing that work. Wouldn't it be nice to just start with a blank piece of paper? Stingray has that approach.

So you are going to start with a blank piece of paper. We have some templates to do some of that writing for you but for the most part, you get a blank document to start from, which is actually very comfortable. And it's very robust. It's beautiful, looks great, works great, has tons of tools, and tons of capabilities.

So some platforms. What's a game engine if we can't go to a device? So modern game engines are expected to hit as many platforms as possible. Max Interactive is no different. So we have a full and robust list of platforms, and we're expanding on that daily. We have Windows, iOS, Android, Oculus, all of them. Lots of stuff coming and lots of stuff already done.

So before I was mentioning that we had all those different games. And all those games were dramatically different, but they all came from the same codebase-- this slide is basically just touching on that point again-- because it is important. None of this had to be recompiled. When we import our content and work with the stuff, and we change our Lua flow, we change our Flow nodes, changed the way we are doing stuff-- we're changing everything about it. But we don't have to go in and recompile the entire engine. There's never that.

So this is all from the same renderer, same everything. And as you can see, we have a platforming, little cartoony game. We have an immersive arch-viz scene, we have another immersive arch-viz scene on the bottom left. The only difference is there, one is completely interactive, one is more of a view and walk around, and there's also an AR-- I'm sorry VR. And on the bottom right, we have a full-blown bottom down shooter, so fast-paced action game. All of them came from the same stuff. The only thing that changed here was the Lua scripting, and the assets that were imported. That's all, nothing else changed.

So we've talked about Lua scripting. Let's just go through that really quick. So what is Lua scripting? It's a very easy to learn-- like I said, I'm a beginner. I don't know programming but I was able to pick it up and make pretty much anything I wanted to make. So it's a very lightweight, beginner friendly, high performance, and super awesome scripting language. So it's got a very low entry level. You can enter it even if you're not a super programmer, which is really important.

And what does the C API do for us? Well, that gives us everything. If we want to change anything in the engine, we jump in to C. We can literally change anything. But it's really important to know that with Lua, you can do pretty much anything too. So you get both worlds-- just depends on which way you like to work. Oh, and really importantly, neither one of them have code modifications. So if you're a Lua programmer already or if you're a C programmer, there's no difference. We did not alter it or molest the codebase at all, so, pretty cool.

So what can we do with the C API? Where would we really want to jump into the C? Well, when you really want to change the renderer or the editor, you can do that from C. Here we have a couple examples. We have a real-time cloud system that is fully editable within the editor, and can run in game time. So we have dynamic clouds that are streaming through the air, and light that comes down, and beautiful god rays, and the whole nine yards, so very cool.

On the bottom right, you can see we have a garbage can with a whole bunch of little numbers floating around it. That's telling you the distances to other objects, which is really useful to some people. So wouldn't it be nice to just be able to click a button and have that kind of utility? Well, you can.

The whole way that we've done it is through a plugin manager. So you literally just click on the plugins you want, and it's downloaded and imported into the engine. And it's now running in your game-- so very, very easy and manipulable. You don't want it anymore, you unclick it, it's done, it's done.

So what is Flow exactly? Well, I'm sure you guys have heard of blueprint. It's very blueprint-like. One big advantage I think that we have over blueprint is just the fact that-- first of all, I think we approached it slightly different. We don't have big, massive, robust nodes, instead it's more of a low level. You have simple math nodes, you have a bunch of basic nodes. And then the concept is that if you really want a robust node to handle a specific test, you're going to go ahead and jump into Lua, create a little bit of a Flow node, and you can adapt it.

But the basis gives us pretty much the ability to do anything. This right here in front of you is a big, robust, bunch of nodes. It probably looks a little intimidating but the important takeaway here is that it's just one step at a time. It didn't start that complicated. It started as, let me place a node, and then let me make this thing work, and then that became another thing, and that became another thing. And with the organizational tools that we have, we were able to make this very complicated-- and this is, a very-- I know because I wrote this. This became a very complicated, fully robust, interactive experience.

And as you can see, anybody could jump in and just start reading it if they wanted to. This is not hard for a human to read. Another really cool thing about this stuff is that you can make it so that-- if you have a non-technical person on your team, and you want to create these Flow nodes, and make these Flow graphs, they can jump in and start changing parameters of the project. And they don't have to be programmers. So if you're a programmer, that might be very relieving to you, to know that you can actually create these Flow graphs for them. And they can just jump in there and change some parameters, and change the speed of the missiles, or in our case, change the position of a chair, so, pretty cool stuff.

So what are some of the toolkits that come with Stingray-- well, I'm sorry, Max Interactive? Well, we have navigation AI. This is outrageously cool. It's really fun to play with. I recommend everyone try it. It's really neat. So here I have some tanks that are just figuring out from their spawn point how to get to the goal. This is just a very simple example but we could do this with people trying to escape a fire, that might be a nice example-- want to figure out, do we have the exits done safely? Is everyone going to be able to make it out in time? So we can do that with this kind of stuff, with navigation AI. And they just compute their way out of the problem.

We also have the integration of PhysX. In this project, I have a VR example, where we have every one of those little rollers on that thing. Yeah, all those who are using physics, and they're pushing those boxes around. And that actually goes on forever, and you can add new ones, and all sorts of stuff, pretty fun. And you can grab the boxes and throw them, and that's all done with PhysX. So very, very powerful, physical world that we have here.

Audiokinetic WWise-- basically, if you can imagine it in your head, and what it would sound like, you can make it happen. Full 3D positional sounds, everything. I mean, it's a full audio toolkit, very, very powerful and very cool. Animation, we can do just about anything with animation. What's really nice is it integrates directly with Maya or Max animations. So pretty much everything you would need to handle animations, and any context you would want to handle them.

One of the really big things is live linking. So when you're dealing with your iPad, and you don't know what it's going to perform like, wouldn't it be nice to just say, let me test this on my iPad right now. And you press a button, and it's playing on your iPad. And then you say, great, that is not nearly the frame rate I need. So super nice because then you can just iterate really quickly, and know exactly where it's at on the machine you're deploying to. You don't have to deploy every time, you don't have to compile, you don't have to send, you don't have to do any of that. You press a button, it's running live in your iPad. So very, very powerful and that works with iPads, it works with Android, works with pretty much all the tools.

So what's the conclusion here? Well, Max is awesome, Max Interactive is awesome. I love it personally. It's easy to use and easy to learn. It's powerful, it's flexible, it's loaded with great modern tools. It looks awesome. I would argue, it looks really awesome, especially if you put the effort in. It's comfortable to use because of the way that we've designed the interface. It's got a very familiar UI. It's high performant, really, really high performant. It's scriptable, its codable, it's visual scripting for those of us that don't know how to program. It deploys anywhere, pretty much. Fits beautifully into your ecosystem, it's VR ready and AR ready, and it's part of your 3ds Max subscription, so why not try it? Give it a whirl. All right, so that's Max in a nutshell. I'm going to hand it off to Benjamin.

BEN SLAPCOFF: Thank you Paul. So yeah, that's kind of an overview of what Max Interactive is. We'll do a little deep dive into how our renderer works in Max Interactive, and how we're using that for AR and VR. So as Paul mentioned in the previous section, Max Interactive is capable of rendering for a wide variety of different devices, and in different styles. Here are just some examples. We have scenes that were done for architectural visualizations, to different types of video games, to stereo rendering for VR and AR.

These types of drastic visual changes are simplified by the fact that Max Interactive's Engine is completely data-driven. So as we've seen with just a few examples, Max Interactive is able to cater to very different needs with respect to rendering. Rearranged from frame rates of going to 30 Hertz to 120 Hertz, different screen resolutions, depending on the target platform, different artistic styles, post effects, and a lot more. To be able to handle all of these different requirements, you need a really flexible renderer.

For us, this means exposing the majority of our pipeline through human readable data files. Shaders, resource creations, resource manipulation, and the flow of our rendering pipeline is entirely expressed in data. The format that Max Interactive uses is a simplified-JSON, or SJSON. This allows us to make the renderer hot reloadable for quick iteration times, and fast experimentations. And that means that you don't have to rebuild your engine to get the renderer changes you want. You just change some data files.

So the main entry points to a Stingray renderer, or a Max Interactive render is found in the files that have an extension of type, render config. These render configuration files drive the entire renderer by binding all rendering subsystems, and dictating the composition, and the flow of a frame. They also define quality settings, device capabilities, and default shader libraries to load.

The three key ingredients that build up these configuration files are resource sets for GPU memory allocations and the allocations, resource generators for resource updating or manipulation, and layer configurations for general frame scheduling. More details are in the handout if you guys are interested about the render configs.

The takeaway message here is that you can tailor the render to your needs by modifying the data files without the need to compile a single lighting of C++ code, or having source access. So having a flexible render was key to getting initial VR and AR support, up and running quickly in Max Interactive. However, it's important not to underestimate the many challenges, in terms of rendering that VR brings to the table.

On mainstream, head-mounted displays, such as the HTC Vive, or the Oculus Rift, the requirements are a 90 Hertz refresh rate to have a smooth experience, with the frame buffer resolution of 2160p by 1200. So that's the resolution you have for the lenses in your devices. On top of this, to reduce it, the aliasing artifacts, the offscreen buffer is, in reality, super sampled by a scale of 1.5 in each dimension, for a final resolution of around 3240 by 1800.

So in other words, we only have 11 milliseconds to shade 5.8 million pixels for one stereo frame, since we have the 90 Hertz requirements for VR headsets. To really see the impact, it helps to look at it in terms of numbers of pixels, to shade per second. When you put it in that perspective, we observe that VR is more demanding than running an application or a game on a 4K screen at native resolution. So you can see here, the table-- the 2160p is 4K and at a 60 Hertz, you're shading around 490 million pixels per second. For a native VR at 90 Hertz, you're doing almost 530 millions per second, so it's more demanding than a 4K application or game.

So let's talk a bit about the optimizations that went into a renderer, to make sure that the stereo renderer was as efficient as possible, given how costly stereo rendering can be. So in Max Interactive, the camera frustum is used during different stages of a pipeline. Firstly, we use it to optimize a rendering by performing frustum culling. This step gathers and prepares all objects for rendering that are either partially, or totally inside the camera view volume, while the others are discarded.

We also use the frustum to compute cascaded shadow maps, in order to reduce perspective aliasing. In terms of work, this means we've sliced the frustration into four different parts, and compute new shadow map for a directional light for each different section of the view volume. So basically, shadows that are further away can be lower resolutions than shadows that are closer to the viewpoint. And finally, we use the frustum during our clustered deferred shading pass. This step voxelizes the view volume and stores different local lay contributions inside the affected voxel buckets. This data is passed on to the shaders to compute the final leg contributions. So this is an optimization for lighting that we use the frustum for.

VR devices have two cameras though, one for each eye. And doing these frustum optimizations twice is costly and unnecessary. So considering that the general view direction of each eye is pretty much equivalent, and the distance between both eyes is quite small, it's worth computing at single enclosing frustum to minimize the computations on the renderer and GPU threads. So you can see here that you have the left and right eye frustum for each eye. And we just calculate a frustum that encapsulates both of those and we do the optimizations on that single global frustum.

Another optimization we use is the hidden area mask to early out on pixels that are invisible in the final images as seen through an HMD. In other words, it's the first thing to be drawn to our depth stencil buffer when in VR mode. This ensures that we cull out geometry that you can't see and applies processing only on visible pixels. So this is the mask, and in your VR headsets, the black areas are pixeled that you can't even-- you can't see in the headset due to the way the lens is. So we just mask out those pixels and we don't shade them, and that saves us some computation time.

Finally, another main VR optimization was to implement instant stereo rendering. This means using hardware-accelerated geometry instancing to produce both the left and right eye version of the scene in a single rendering pass instead of two. Distinct eye transforms are stored in the constant buffer and stereo rendering is handled by the vertex shaders. Even instances use the left eye matrices, and the clip position is shifted to the left, while odd instances use the right eye matrices, and is shifted to the right. We also adjust the clip plane dynamically to prevent spillover to opposite eyes. This removes the need to double draw culls and state changes. So basically, you can just do one draw cull for both of the eyes using this method, which is a huge optimization.

So those are some of the main optimizations to get a good stereo rendering pipeline. However, for some of the lower end devices, this is not enough. So far, we've only talked about tethered devices, such as the Oculus SURV1V3, but there are a wide variety of untethered devices that a stereo rendering pipeline supports. Obviously these devices are way more limited since they don't have access to a full desktop GPU, but the flexibility of a data-driven renderer makes it easy to modify, so that the engine can run smoothly on them.

We have a stripped down version of our full render config that we call, the mini renderer. This mini renderer strips out a lot of the deferred shading features of the full renderer, and also removes a lot of the post effects, so that we can achieve a smooth experience on phone-based VR devices, such as Google VR, Gear VR, or Google Cardboard, and also AR devices, such as the HoloLens that also use the stereo rendering pipeline. So with all of that, you should have a pretty good idea of what Max Interactive is, and how it can be used to power great VR and AR experiences. Sir?

AUDIENCE: Do you have a slide that shows us the workflow from Revit to that?

BEN SLAPCOFF: Yes, yes, so that's--

AUDIENCE: I would like to see it. What does it require? And what do we need to do? How much it's going to cost me to go from point A to point Z, and to point whatever.

BEN SLAPCOFF: So that's exactly where we're going to-- that's the Forge AR/VR Toolkit. Yeah, so what is the Forge AR/VR Toolkit and how can it impact you? Well, I'm sure many of you here are aware of how useful Forge can be for managing and streamlining the use of your 3D data. And the AR/VR Toolkit aims to push that even further by making your Forge data instantly accessible on any VR or AR platform, such as the HoloLens. And since your data will exist inside a game engine like Max Interactive, you can craft engaging experiences around your data too.

The toolkit is still in beta, and we'll talk later about how you can get access to this beta, and be a part of it. But that means that a lot of features that would really benefit the customers still need to be worked on. The first step though is actually getting your data in the engine, and that's what we've done. So let's dive into how we've gotten to that point. So as I've said, the Forge AR/VR Toolkit is our first pass at integrating Forge with Max Interactive. With the Forge AR/VR Toolkit, you can easily visualize all of your Forge data in AR and VR, and any other platform supported by Max Interactive.

Currently, it's set up with a client-server architecture. So the clients in these cases are AR and VR apps creative of the 3ds Max Interactive, and they communicate to a server app that uses Forge to prepare any design data for Max Interactive clients to consume and render. So as you can see in the diagram here, the server app is the intermediary between the devices, which are the clients, and the actual Forge APIs. So first, let's talk about what happens on the Forge AR/VR Toolkit Server.

AUDIENCE: What does [INAUDIBLE]

BEN SLAPCOFF: Sorry?

AUDIENCE: What is different from that one?

BEN SLAPCOFF: So--

AUDIENCE: But where's the model?

BEN SLAPCOFF: The model, well, you'll see how we get the data that we want to get uploaded to Forge and processed into a format that will be supported by Max Interactive.

AUDIENCE: All right.

BEN SLAPCOFF: So we'll get that. That's the server application handles that. So the server is a standalone application that interfaces with Forge and sends data to incoming clients. The first thing that the server does as it boots up is it reads in a config file for some of the initial setup called, the server config JSON file. Its contents look something like this. There's not much to explain here. You have a client ID and a secret key that you are acquired when registering an app on the Forge development portal. You need these in order to make different Forge API calls. And we also define password, data, cache, and temp folders, which are folders using different part of the data preparation process, which we'll get into.

So once the config file has been read, we first go through every item in the data folder. And that was specified in the server config. And in the data folder you can put your Revit files, your Navisworks files, and any design data that you want to be uploaded to Forge and to get into Max Interactive. The server will go through all of the items in this data file and register them as resources that you want to be available on the server that Max will be able to read. They can be any of the 60 or so file types supported by the model derivative service that you see here.

So once the app has registered all these data files, we actually need to upload them to Forge, the Forge data management service. So first, we use the client ID in secret to get an access token from the Forge OAuth API that we can use for future API calls. We can then use the Object Storage Service to create buckets, which are used to store objects we want to upload to Forge. Each bucket is created with a unique identifier for each resources. Creating this bucket allows us to perform another request to actually upload our resources. So we upload the resources, and we keep track of the URN it returns for future use.

So at this point, we're finally ready to start interfacing with the model derivative service, using the Forge Model Derivative API. The Model Derivative API lets users represent and share their designs in different formats, as well as quickly extract valuable metadata. So this API can translate any resources into different formats, such as STL or OBJ, but in our case we want to translate the resources into the SVF format.

So using the URNs we received from the Object Storage Service in the previous step, we post the translation job to the Model Derivative Service to convert each of the files that we put in our data folder into SVF files. The SVF format is normally used for extracting data and for rendering files in the Forge Viewer. But we perform another step and convert these SVF files into a viewable format, known as bubbles. The Forge API we used to do this conversion is known as the Extract API. These bubbles are downloaded in the temp folder that we specified in the server config as zip the files containing all of the relevant data for the Forge Viewer, in our case for Max Interactive, including manifest data, fragment lists, light lists, camera lists, hierarchy information, geometry data, and so on.

So now that we have all of the SVF bubbles on disks, we can read in the relevant information that we want for Max Interactive, and convert it into a format that will be easily consumed by the different AR and VR clients. So we use two libraries to get the required data from the SVF files. The LMVTK, or Large Model Viewer Toolkit, and the PropertyDB Reader library. We unzip the zip files. We find the SVF path and look for any PropertyDB files, which contain a lot of the metadata information about the object. And then we start the conversion process. The goal here is to convert the SVF model into our own Scene Model representation that's shared between the client and server.

So the Scene Model is basically just a collection of scene elements, which each maps a node in the SVF file where parsing's instance tree. So using the assortment of different readers made available in the LMVTK, we can extract all the necessary data to each scene model node. The Fragment Reader is used to get information about each geometry node, including its transform, material, and geometry reference. The InstanceTreeReaderNode is used for building up the scene hierarchy. We would traverse the instance tree one node at a time, and keep track of the nodes ID and parent ID to be able to rebuild the hierarchy in Max Interactive.

For each node, we also want to fetch some data from the property database. We can get things like some attributes depending on where the data is coming from to names of the different parts. So it can really be useful in your application. We use the MaterialReaderStream to get all relevant material information, to be able to build an actual Max Interactive material, once it gets on the client side. So from the SVF file, we get a simple material JSON file that we can parse different material parameters we want, such as the color or the opacity of the material. Yes?

AUDIENCE: So why are you going through this translating process? Can't previous Max Interactive not understand [INAUDIBLE]?

BEN SLAPCOFF: Well--

AUDIENCE: Is [INAUDIBLE]

BEN SLAPCOFF: Yeah, it does but the goal is here that we're creating from any format, any design format like Rivet or something. We're just serializing it into our own format that will be consumed by the clients.

AUDIENCE: Like you, [INAUDIBLE] that is not possible?

BEN SLAPCOFF: Directly into Max Interactive?

AUDIENCE: Yes.

BEN SLAPCOFF: No, you can't, well, there's some services, but not all of these formats are supported by Max Interactive. But using this Forge service, all of these different designed data formats.

AUDIENCE: All the best product called [INAUDIBLE] supporter, so RVT, data today, APIs--

BEN SLAPCOFF: Those are all supported by the Model Derivative, API, yeah, so all of those types of formats can get into Max Interactive with this toolkit.

CYRILLE FAUVEL: Yeah, so-- just a bit more information, going through Forge although you to read, almost like 60 different file format on streams that information directly on to get an engine running on any device. Going through the epics transformation, like exporting to epics, you're going to lose information, whether you lose the archetypes object or you lose metadata, not all epics exporter are a core in terms of information that translate. When you go through Forge, you get everything that Forge raids out on your find without losing information. That's a main interest of using the Forge AR/VR Toolkits. You have a direct connection to the find.

AUDIENCE: Well, whatever translation Forge is doing, can Max Interactive do the same thing?

BEN SLAPCOFF: Well, we're getting Forge to do it for us. So that's--

AUDIENCE: Well, I know that but for us we have to go one place to another, to another, to another, to get that, but the less step we do, faster for us and the better.

BEN SLAPCOFF: But yeah, it's pretty-- it's like straight from your like from Revit or anything, you can just put these directly--

AUDIENCE: That's not what I'm trying to say. Maybe, I can [INAUDIBLE]. When you go to Navis, it's one click from Revit you go to Navis. That's not a legal step or that's not middleman. Why in this process, there's so many middleman, every time I stop, somebody is asking me for more money, for more money. Why can't we go from A to B, and done? Why do I have to have 8.5, 8.3, and stuff like this?

BEN SLAPCOFF: I think the goal is to streamline this as much as possible, and there's only really one step and that's putting whatever data file you want, and then that gets uploaded. But your feedback is extremely valuable, and it's in a beta right now.

AUDIENCE: Think about, we are the designer. First of all, time for us is very important and the design of the problem is changing on the fly.

AUDIENCE: Use Revit Live, maybe you've done. It's fun.

BEN SLAPCOFF: Yeah, there's a Revit Live that does that.

AUDIENCE: [INAUDIBLE]

BEN SLAPCOFF: We'll get into-- yeah, we'll get into questions after because we'll get through this. But we can definitely talk after.

AUDIENCE: Revit Live is wonderful. You should not have any [INAUDIBLE].

BEN SLAPCOFF: Yeah, you should definitely look into Revit Live because that's using Max Interactive as well. Yeah. So anyways, once all of this is done, and we have a complete C model representation in memory, and we're able to serialize and deserialized on demand, so that's where the cache folder comes into play. Each resource, after being uploaded to the model derivative service and transformed to the viewable SVF format, to then be downloaded and converted to our own scene model, is serialized in a binary format, known as a .extract file, in our case, ready to be consumed by any client on the Max Interactive side.

So at this point, we're ready to start the server loop and wait for any incoming Max Interactive clients. So let's talk about the clients. So the server has all of our Forge data prepared and is ready to serve it to any client that connects to it. So all we need to do to get Forge data into Max Interactive now is to make Max Interactive a client.

So to do this we have created the extract I/O engine plugin as the client side of our Forge integration. The engine plugins are C modules and interface with the Interactive Engine through our plugin API. So if you're comfortable in C and want to interface with the plugin on that level, that's totally doable with our plugin API. However, we can also expose some Lua bindings to expose some of the extract plugins, functionalities, to projects developed in the editor.

So the first step towards receiving data from the SVF server is actually connecting to the server. So we exposed the Lua call in our plugins ExtractIO Lua namespace. Users can call this, server_connect to Lua method and specify which IP address the server is running. When this is called, the TCP socket is created in the plugin, and a connection to the server's requested. When the server receives this request, it responds with the number of resources it has available and also the list of all resource names that it has.

So we've also exposed the method to get a list of all of the available resources. This can be useful, to have a menu in your project with the list of all the available resources, so you can select which ones you want to load. The Model Derivative service even provides thumbnail generating services. So you can request thumbnails and display a thumbnail for each resource in the app. To request the actual data from the server, you just need to call, load_server_resources.

This method will send a request to the server with the index of the resource you want in the resource list, and the server will respond with that resources binary data. The client will receive this binary and deserialize it into the same scene model format that was on the server side. We have lots of other Lua methods that we've exposed, such as utility functions to get bounding boxes for different mesh fragments, or query metadata about the objects, but these few calls will get you up and running with your Forge data.

So once we have deserialized the Forge data scene representation from the server, we can use the plugin API to pass all the information that the engine needs to handle it appropriately. There are many different plugin APIs to get the Forge model up and running in Max Interactive. We use the SceneGraphAPI to create some of the scene graph nodes with the correct pose information for each scene element in the SVF representation. This way, we can have the design data's complete hierarchy represented in our engine.

For nodes that have a geometry, we have the mesh API to create mesh objects in the engine, and we also use the RenderBufferAPI along with it to fill in the meshes of vertex buffer information. We also want the material properties parse from the simple material in the SVF file. So in the project that will load the SVF file, we create a basic material graph with some parameters exposed, and then using the material API, we can fill in those parameters with the values saved.

So here's an example of a shader graph that has two material parameters exposed for base color and transparency. It's an extremely simple example, but with our flexible shader graph system, it's incredibly easy to add new parameters or texture samplers, if you have that data. The last thing we want to create for each mesh of physics representation using the Physics Runtime Cooking API. This creates a physics mesh based on the mesh vertices, so that we can do things like raycast or collide against individual fragments.

So here's an example scene, a Revit file, showcasing all of the individual components that we've just gone through. The exploding functionality is possible due to the hierarchy being preserved through our Scene Graph API. And then all those simple, in this example, the material properties were correctly applied to different parts of the model. And finally, the green outlines are showing that these objects have physics enabled for them.

So of course, now that we have this data and Max Interactive, we can get it running on any of its supported platforms. Here it is running on iOS with ARKit augmented reality. You can view the models on your phone in the real space around you. We've even started working on custom invisible materials that can still receive shadows, specifically for this platform. Using materials like this in conjunction with ARKit's plane finding abilities, we can have ARKit objects that really look grounded in reality by having them cast shadows on the world around it.

We also have it running on HoloLens, Microsoft's AR headset. With HoloLens, we can have hand gestures to manipulate the objects, as you see here, swiping to the side to rotate, or up and down the scale. You can have voice commands like explode, or commands to load the next model. The HoloLens also has spatial mapping capabilities that we've taken fully advantage of in Max Interactive. So they can be used to place objects on the floor, or on the wall, or on a table, or you can have real world objects occlude to your virtual ones. And we also support Microsoft's mixed reality headsets, running on the holographic frameworks, as well as any other standard VR devices such as the Vive or Oculus.