Drones, Photogrammetry, 3ds Max—and You!

BEN BISARES: Hi, and welcome to my class on drones, photogrammetry, 3ds Max, and you. My name is Ben Bisares. I'm a senior technical support specialist here at Autodesk. And we have a lot of interesting stuff to cover, so let's get started.

And a good place to start is the agenda. So you may have noticed, or you may be wondering after reading the title, What exactly is this class about? And the best way to do it is to basically go over the problem that we're trying to solve. We'll follow that up with a couple of solutions. Spoiler alert, one of them is using photogrammetry to create low-polygon models.

From there on, since we're discussing photogrammetry, it's not really the goal of this course, but I'll give a brief overview and some best practices. After that, we'll take a look at how we can actually create 3D models from the photos that we're taking using ReCap Photo. And then the star of the show is basically creating low-polygon textured models based on the high-polygon models that we created in 3ds Max.

So hopefully that gave a better idea of what we're going to take a look at. So let's get right into it, which is the problem. So I find that every good class starts with a clear and an intriguing issue to solve. And in our case, imagine that you're tasked to create a detailed, visually accurate model of an existing structure. For what reason? It could be for various reasons. There's some good reasons out there.

For example, you want to preserve a historical building digitally, not physically. Or, for example, accident reconstruction. For example, you must have heard of the fire of the Notre Dame Cathedral, and they used drones to capture the damage that's done. There's many examples out there where you may want to create a very detailed and accurate model of a structure.

Now why must it be low polygon? So for example, yes, you probably would need a high-resolution or high-detailed version of that. But for example, they may want to also create a separate application where they can walk through the building in question, and then be able to share it with other people, other teams, for example, either on the web or on a mobile.

So that would require, again, a low polygon version. And other reasons also, so digital maps, your Google Maps or your Bing Maps, for example. VR, another good example, will need low-polygon, yet good-looking versions of models, and video games.

Now the most important part is that they have to look as close as possible to the real-life counterpart. We should minimize, for example, any instances where someone has to guess as to what the data is. So what are the solutions that we can have for this?

So the first solution is what I would call your traditional method. You would go on site, take reference photos, so generally, that means you and a camera or a team and a camera walking around taking photos of the subject in question. Then you would go back to the studio and do things like, for example, create a low-polygon version of what you took. And this is based off the reference photos in any 3D package out there, be it 3ds Max, Maya, Blender, et cetera.

Then because it's going to be in low polygon, we obviously will-- we'll lose textures. The polygon budget to create some details that are there. So for example, for trying to create a church, the church may have things like stones, bricks, things of that nature, that we may not be able to model individually. We maybe have to use, for example, a flat wall. So how can we simulate or give the illusion of those details basically with textures? So we apply a texture onto it, so like a decal that would put on a surface, to give the illusion that there's more detail.

Now these textures could be created from the photos themselves. So for example, we'll take the reference photos, we'll bring it into an image editor, for example, Photoshop, and then do some kind of manipulation to create a texture based on those photos. It's not necessarily difficult to do, but it can be very time consuming and it does involve a good set of skills, knowledge on how to manipulate the images, to create a good-looking texture that doesn't have, for example, seams. Like you don't want to be-- you don't want it to be obvious that there's a border between one area and another based off the photos.

So it comes with its risks. So like I mentioned, it is time consuming to do. And there is a risk of having missing or inaccurate details. So for example, if you were tasked to create this elevated walkway that you see here on the left, now it's fairly simple to go around and walk taking photos, but what are you going to do when it comes to the structure that is above? So what you see on the right photo.

Now in my case, I had access to a building that was right beside it. So I can simply go inside and take a few photos. But again, I mean, it does limit you to essentially the angles that you can reach.

So for example, the elevated portion of the walkway that you see here, What does it look like on the other side? I mean, you could take an educated guess and say, well, it probably looks like exactly what it looks like over here on the side that we can see. But again, we don't have benefit of guessing in this case. Like I said, it has to be very accurate.

For all we know there may be [? ticks ?] or damage that needs to be captured. So we can't just say, well, it probably looks like this, and create it as such. So let me do this. Sorry, there we go.

So what is another potential solution? Well, we have the-- what we call photogrammetry-based method. And this is essentially the topic in question for this class. So the way it differs, well, it actually starts off fairly simple-- fairly similar, excuse me, where you start off with taking photos.

Now in this case, it could be either terrestrial or aerial. Now terrestrial is a fancy way of saying you standing on the ground holding a camera, taking photos. But you can also take them from above using some kind of unmanned-- and unmanned device, so for example, like a drone, fixed wing, whatever you want. Generally, though, we use drones these days to do that.

So it can be a combination of both, even if the-- it's required. And sometimes it is highly suggested. And I'll show an example where it could have been of use.

Now from these photos, what we'll use is we use some kind of software. In this case, ReCap Photo, to create a high-polygon and textured model from those photos like what you see on the screenshot on the right. We're going to take a look at-- or we will be using the example that you see here on the right.

Once you have that high-resolution textured version, then, once again, what we're going to do is we'll create a low-polygon model. This is similar to the other method. And this is done manually. There's no really-- there's no way to get around it. You could, for example, try to create a low or some-- sorry, let me back up a little bit.

You do have to create a low-polygon version of the model, obviously, as per the project's requirements. Now you may be asking, well, Since I have the high-polygon version, why can't I just go and use that? And for example, use some kind of software magic to make it low polygon?

And the reason is generally the tools that we have for reducing the amount of polygons-- I mean, in my opinion, aren't very good. I mean, you maybe have this tool that's magic and it's great and it gives perfect results every time, but the ones that I'm familiar with generally give results that are so poor that I really don't explore that area. I actually kind of dismiss any kind of optimization tools off the bat.

I am aware of one that seemed to work quite well. But again, my preference is, is that just create it manually because the low-polygon requirements generally mean that you have to be very deliberate in terms of the polygons that you do have. We have such a low polygon budget, they have to be used as best as possible.

So you can see the example here. Now originally, this is the high-polygon version. And you can see it is very dense. From far away, it just looks like white because we have a huge mass of polygons. And as you can see-- let me just move this to the side-- it does consist of 2 million polygons, which is extremely high, even optimized to a point that-- I mean, I wouldn't even use this. This is still 10,000 polygons.

And you'll see that the low-polygon version that we'll be working on-- now it is a section, granted, so maybe we can say this would be 2 million polygons if we were working on the same section. The version that I have is 300, and it looks much better. So that is the problem generally with optimizers. They generally don't consider the flow of the objects. And it may create polygons in weird sections. Like this window has just become this polygonal, distorted shape. Excuse me.

OK. And then what we're going to do is we will use the high-polygon model to create textures for the low one. So this is the magic, we'll say, of the technique that we're going to create today. You don't need to have, for example, someone who has skills in image editing to create the textures. It just does it automatically for you.

So this could be quite useful in a lot of scenarios. For example, you may not even have someone on the team that has these image editing skills. If you're, for example, an architectural or construction-based company, you may not have anyone with the skills, for example, let's say, versus a video game or movie company where they have plenty of people that could probably do very good textures just based off random photos.

So in this case, this takes the manual guesswork out of creating the textures. It does it automatically for you. If the high-polygon version has good textures, you'll get basically the same results. And because it's aerial, you will have access to areas that you may not be able to capture from the ground.

So let's take a look at how this is done. Now because I mentioned photogrammetry, I will do a brief overview of photogrammetry in general. I don't want to go too deep into it because most likely, most of you already know all this. And there are classes that are-- let's say, would be better suited for learning this kind of thing.

Photogrammetry is somewhat detailed. There's a lot of information, so we run into the risk of having-- the term I'd like to use is, a little knowledge is dangerous. So either don't do it at all or learn it in as much detail as possible. But having a little bit of knowledge can generally get you into trouble. So I won't spend too much time on this.

But in general, what's photogrammetry? It's like what I mentioned before. It's a digital recreation of a physical object based on multiple photos. It can be based on photos taken from the air or from the ground or we can have a combination of the two. And like I mentioned in the solution slide, it is a hands-off method. So it is an algorithm that will create the model based on the photos that you take.

So in this case, you have no influence, really, in terms of the end result other than the quality of the photos that you take. Now when I say quality, I don't necessarily mean higher megapixel is better, but more of having photos that capture as much detail as the-- of the subject as possible. So generally, what this will mean is you will have to take it from as many angles as possible.

Now I did mention that we have two different ways of taking photos, from the air and from the ground. So aerial, and I will talk a little bit about aerial photogrammetry because if you're going to be creating large structures, you'll be likely using aerial photogrammetry, which has some important considerations that we must mention, or that I should talk about. And one of the more important things is well, Are you even allowed to fly a drone in the area that you want to take photos from?

So there are quite a few things that you need to consider. So no-fly zone, so areas around airports, shipping ports, certain buildings, for example. Borders, borders of countries, you can't just go and fly wherever you want. So at best, for example, the drone that you may buy may not even allow you to fly.

So for example, DJI, because it has a GPS, it knows where it is, it knows where the no-fly zones are. So you don't want to drive, for example, two hours to your location just to find out that you can't even fly. So plan ahead.

Also, drones over a certain weight may require registration in your country, and pilot certification. Generally, the weight is somewhere around 250 grams. You may see drones out there that advertise as being under 250 grams, and it's for this reason. And also keep in mind that if you are doing this for commercial purposes for your business and not as a hobby, it may require that you apply for additional permits to do this.

And there's also rules with visual line of sight. Now what that means is, is that generally, you're not allowed to go fly your drone to a distance or an area that you can no longer see it. So you can't take off and have your drone fly to the other side of the mountain or, for example, behind buildings. It must stay always within line of sight.

Now also, this applies-- this doesn't apply to the pilot itself in the sense that you cannot fly and also be the one maintaining visual line of sight. You may be required to also have a spotter. So basically, someone beside you whose job is to keep an eye on the drone. So, I mean, long story short, just be aware of all the rules that you may have in your country when it comes to flying drones.

But once you are able to go and fly, you have to basically plan flight paths to maximize the-- like I said, the quality of the photos that you'll take. Generally, you want more photos. You'll never come back from a session disappointed that you took too many photos, but you will if you took too little. So always, more is better.

Generally, depending on the subject, you want to start off by creating a kind of grid pattern path that's flowing over the subject. Now you may also do some other paths that may be used to optimize, for example, capturing the detail that we have. So let me show you an example.

So for this church, if I enable these shots, now you'll see a series of these little pyramids in the sky. Each pyramids represents a photo that was taken. And as you can see, in this case, to capture the church that you see here, we did fly a grid path over the subject. And it's always-- you really should do this just to what we call establish like a-- the territory, so to speak. So just to kind of create the map.

And then in this case, because the-- excuse me-- the church was a subject, what we did is we also did an orbit around the church to capture the detail from as many angles as we could. Now you do see these ones here. Now this was a structure that was just on the side, which we removed for this one because it wasn't the subject of this project. But because of that, we couldn't fly lower than that. So there was a building in the way.

Now because of this, so the photos, as you can see, if I try to line this up as close as possible, are somewhat from this angle. Now it looks OK from here, but if we go kind of ground level, whoa, things look a little bit smeary. And the reason for that is that's simply because we weren't able to capture any photos from this angle. Now this is where doing terrestrial capture may be a good idea or may be the solution for this issue. So have someone go with a camera and take photos of this area of the building to fill in the areas or the spots that we can't capture from the air.

Let's see. So flight paths. Now you also want to capture a wider area than that's desired because photo overlap is the key to creating a good result. So if you just have one photo that's by itself, generally, the algorithm won't be able to create anything from that because it needs different angles of that. If it's a flat ground, you might say it's fine, but imagine it's a tree. Just taking one photo of a tree obviously doesn't capture all the detail that we have there.

And the algorithm does have a certain threshold in terms of what it believes is accurate. If it believes that it's-- the confidence in creating subject, it's not high, it just rejects. So note that a lot of times, the borders that you have won't be included in there. So for example, if you're doing some kind of land survey, you want to be able to-- or you want to fly outside of the borders that you want to capture because the edges generally won't be in there.

So let's skip ahead. Now after hearing all of this, and if you haven't done this before, you may be thinking to yourself, wow, that sounds kind of difficult to fly these paths. And to be honest, you are correct. I actually-- just as a hobby, I do fly FPV drones, so I'm quite-- I won't say I'm amazing at it, but I enjoy doing like acrobatics and doing some light racing, we'll see-- some light racing.

But the hardest thing I found was flying these lines. This is the hardest thing to do. I have tried it just for fun to see if I could do it manually. And from the ground, I'm looking at it, it looks like I'm flying nice, straight, parallel paths. And when I view the shots, it looks like I was trying to write my name in the sky. It's just completely different from what I thought.

So gladly, you don't have to do this manually. There is software out there, which is generally called a mission planner, which is an autopilot for your drone. So what it is, it's a software where you plan the flight paths that it's going to take. You have a map, you say fly over this area. You can do different patterns, as you can see. You can even do, for example, orbits around a point of interest.

So you plan all of that out. You then go to the site, and then you basically take off, and with great trepidation, you watch your drone fly off by itself to go do the mission with your fingers crossed that it comes back. But you don't have to do this-- you don't have to do this manually.

So once you've taken all of those photos, well, you just have photos. What do you do? Well, basically, need some kind of software to take those photos and create a 3D model from it. And in this case, I'll be discussing ReCap Photo. Now ReCap Photo is essentially a feature that we have in ReCap Pro. So if you have ReCap Pro, you will have ReCap Photo. But it is-- excuse me-- it is a separate software that's installed on your computer.

So just briefly, the projects that you create in ReCap Photo can either be an aerial or an object project. And there is an additional cost in this. So generally, one Flex token for every photo-- 50 photos that are used in there. So what are the difference between the two projects?

Basically we have object, and an object project is generally used for close proximity objects. So things like statues, cars, little trinkets that are on the tabletop. That is what we use object for. But it can also be used or-- I won't even say can, sometimes it's necessary to use that if the subject you want is essentially more vertically oriented. So if want to do a building, so building scale, now you may think, OK, this is probably an aerial, but to be honest, it may not even work properly with an aerial or the results will be quite disappointing.

Now the reason is, and I'll come back to the object project, is that for aerial projects, now these are generally done for things like land surveying or mapping, where the scale and accuracy of what's being created is critical. Now because of this, it has a much higher threshold in terms of what it considers to be confident in the results. Now because of that, generally it expects usually, something like flat photos, photos of flat land. I mean, there could be undulations and trees, that's fine.

But if the majority of your photos just consist of a building-- so, for example, you have the drone on the ground. You take off, you fly up taking photos, you fly down taking photos, so very little, for example, horizontal movement, but mostly all vertical, generally, aerial will fail at that because it's not expecting those photos. It doesn't really know what to do with these.

And it says, OK, I'm really not confident in the results. I'm just going to reject. So in this case, you need to use an object project which essentially will build what it can based on the photos that it has. So it takes its best guess and just goes ahead and creates an image.

So there's no GPS required on this. So data isn't needed for object projects. It's not even used if you had it. And you do have a maximum of 300 photos.

With an aerial project, it does require photos with GPS data. Now this is provided by the camera itself. So usually, the camera on the drone. It's not you manually putting the GPS data there. And you do have more photos.

So you would basically create your project, submit the photos, and then after a certain amount of time and processing, it gives you a high-resolution result such as what you see here. So in this case, once again, the church. So it does comprise of the actual model itself. So this would be the mesh. And then you have the textures that are applied onto it.

Now you could do some basic editing in here, which was already done. So if you wanted to delete, for example, some of this ground area, this could be all done. And once you have a final product, you can then export this out to whatever format the 3D package that you're using supports. So in this case, FBX generally the best results that we need, at least for 3ds Max.

OK. You have the high-resolution object, you have it textured. What do we do now? Well, this is where we bring 3ds Max into play. And like I said, it's the star of the show. So with 3ds Max, what we can do is we can basically use the high-polygon mesh that we created to enhance a low-polygon version that we create in 3ds Max. Now-- oh-- skipped way too far ahead. There we go.

So the way that it works is we create the low-polygon version, and then we import the high-polygon model. And then what we'll use is something called a projection tool to take the details that are on the high polygon and apply it to the low polygon. Now we can do things like create textures for it, so things like diffuse maps, and even things like bump and normal maps. So we can create something quite interesting with the detail that we have here.

Now when it comes to the low-polygon version, when you create-- and like I said, this is something that you should be creating manually. Like I said, there are tools out there to reduce polygon count on objects, but I've never liked them. So I've created it manually, which you can see here on the right.

So it has to be as close as possible when it comes to the size. Now obviously, it's not going to be a 1 to 1 match because you have a lower polygon budget. So you may not be able, with the polygons that you have, match the exact detail-- surface detail that you have there, but as close as you can. It also needs to be unwrapped, which I'll discuss just in a bit when we get into 3ds Max. And although it doesn't have to be initially, when you're ready to project the high details onto the low-polygon object, they need to occupy the same space.

So let's take a quick look at the process for this in 3ds Max. So we'll go and open up the file. So what we have here are the two different objects appropriately named Church Low Poly and High Poly. Now in this case, I am doing just one section, one wall, just for the sake of, let's say, simplicity in the example.

And you can see the shape that I have here is very similar, but it's not exactly the same. But it also has much lower polygons. You can see, like I mentioned, this one has 360 polygons, and the polygons are better used in the sense that they flow better with the shape of the object. So we have these stone structures-- it's a bit hard to see-- that stick out or, for example, along the window, these are replicated here.

Now of course because it's such low poly, we can't create all of the details. So for example, the tiles on the roof, those aren't there. It's a much simplified version. And the same thing goes for the wall. It's generally flat, I mean, with just a few exceptions, but we can't generate all of the little bumps and imperfections that we have here.

So the way this works, and I will go, let's say, somewhat quickly over this. Don't fret on any of the small details. It will be in the handout. So if you're wondering, What's the option that he clicked on there? And, oh, what's the value he used?

Don't worry about it. In this case, just I want you to understand the general workflow of how this works. All of the details are in the handout.

So what we want to do is basically take the low-polygon version, and like I said, it needs to be unwrapped. If you don't know what that is, well, this is let's say, out of scope for this. But the important aspect to understand when it comes to unwrapping in this scenario is that you cannot have any overlapping UVs. OK So all of the UVs have to be in its own space.

And the reason is quite simple. We're doing a texture projection. You don't want to have two areas of texture applied on top of each other. So everything has to be by itself. How you do that, it's not the scope for this class.

So we have it unwrapped. And what we're going to do is we'll simply do our Render to Texture. Again, the details are not that important, but we will pick our High Polygon. So we have the Low Polygon selected. We will use the High Polygon version as our source. We'll use the existing unwrap that we have. And then we can define the different maps that we want to create.

So for example, Diffuse Maps, a map based on the shadows or the lights in there. Now in this case, I'm just going to create a diffuse map and a normal map, and that's it. I mean, we can choose the size, but we're ready go-- we're ready to go. So I'll go ahead and render this off. Let's just skip these things.

And there's the texture. Now you may notice that there are some areas that are in red. Now this red is basically a-- what we call a ray miss. Basically what that means is that it tried to project a texture onto this area and it could not find one to project. Now it's a bit difficult to kind of show with this more complex model on how this works. So basically, I do have a simpler version for us to take a look at.

And the reason we're getting these ray misses is because of what we call the cage. Now when it does the projection, what it does is that it creates a shell that's based on the low-polygon version. And it casts rays from that low-polygon version-- the cage towards the low-polygon version.

Now the rays will see whatever is basically from there, so in this case, a source. and then will apply what it sees onto the texture. So if it doesn't see anything, then we get what's called a ray miss. Basically, it says, I'm trying to create a texture. I don't see anything there to create a texture from. And we get these, by default, red marks.

So for example, here, you can see there's some red here and red here that corresponds to this area and that area. And again, the reason is, for example, here's a cage-- I know it's a little bit hard to see. We'll see it in a, let's say, more interactive example in a second. But the cage is this blue line that you see here. We have our low-polygon version, which is here. And then we have this yellow high-polygon version.

And you can see we have a ray miss here because it is basically projecting towards the low-polygon version-- whoops-- and there is nothing there. The high-polygon doesn't intersect this line. The other issue that we have here, for example, is that the cage is actually behind the high-polygon version. So again, we get the same issue that it has-- it doesn't see anything. And so it just gives us red. And we can take a look at that in 3ds Max-- let's see-- because, yes, I'm fairly certain that it wasn't so clear.

But in this example, this simplified example, let's say the low-polygon version is what we have in green. And I'll say the higher polygon version is in yellow. And we want to do a projection. So again, we repeat the process, select the Low, go to Render to Texture, choose the High as our source-- Let's see, I think everything's OK-- and we go ahead and render.

And again, I see those ray misses. now, if I go to the projection, and let's make it a little bit easier by shading it. So you can see this kind of mouth-looking red area here. And again, that's because the cage is actually behind-- as you can see here, behind the high-polygon version. We have these misses because again-- and I don't know how easy it is to see, it's projecting towards the low-polygon version and there's nothing there to see.

So how do we fix this? We can adjust the cage. So the easiest way usually is just to do something like a Push, which pushes the cage-- excuse me-- towards or away from the low-polygon version. So in this case, I can push it-- excuse me-- so that, in this case, the cage is now in front of rather than behind the high-polygon version.

And we can render again. And yes, now you can see that we no longer have the miss in this area, but we still have these top sections. Now the way this can be fixed, because no amount of pushing and pulling will fix that, we can manipulate the cage manually. So I can select sections and then try to adjust so that a ray projecting from there will intersect the high-polygon version, which I believe something like that, I mean, might be easier to see from, let's say, this angle here.

So I'm imagining a line drawn from here to here, and think it's intersecting the high poly. And when I render, there it is. You can see the red is gone. So generally, any kind of misses that you may get is fixable by adjusting the cage. You don't really need to do anything else in terms of the actual objects themselves.

So if we go back to our other example and repeat one more time, so select the Low, Render, Render to Texture, choose the High as the source-- sorry the window always goes offscreen. And I believe we're OK. Now actually, one other thing I want to mention, so it will generate a cage that it thinks is appropriate. Now I don't know about you, If you've done this before, but, I mean, this cage doesn't look correct to me.

Now what we can do is basically reset the cage and then push it out a bit manually. So let's say something like this, which t me, seems to be much better than this kind of weird shape that we had before. And then go ahead and render. Now you can use various renders for this. I am using, in this case-- what am I using? Scanline, simply because it's just faster for this scenario.

Now do we worry about everything here? Not necessarily. For example, this red area is actually the underside, which we don't have in the high-polygon version. So we'll never see it. We can ignore it. But there are some obvious areas where we might see some red. And we'll look at that in the example later.

So you'll need to fix the cage to do this. Now at the same time, what it's done, it's also created a normal map. Now this normal map, and we'll take a look at it in another example, is an image that represents surface detail. So again, the texture is great, but technically, we're just applying a texture onto a flat surface.

Can we replicate the kind of surface imperfections? Yes, that's done with the bump map. And let's take a look at that now. So we do have-- let's see-- an example usage of this.

Now keep in mind that the end result doesn't have to be all 3ds Max. Once we've generated the textures, what we can now do, if I was creating, for example, a VR experience, well, then I would bring these-- the low-polygon version and the textures into whatever engine that we're using to create the VR. So it's not all done completely in 3ds Max, but am doing this just for the sake of simplicity.

So once again, we have the low-polygon version and open up the materials, I have a material already prepared. And I'll just apply it onto that. And if we render, I'm using Arnold in this case because it does make it nicer or easier to see the results. You can see once again what we have here.

So now from this low-polygon version, we have what looks convincingly like a high-polygon version. And we also have-- let me just disable-- a normal map applied to this. So let me just disable that. And you can see that it also gives us some-- the look of some of the imperfections. I'm trying to find an angle that kind of best shows it.

So the bricks kind of appear on there. And it is, again, a simulation of surface detail. So it will be affected by light. So if we change the light angle, the shadows that we'll see here will also change. So it's quite convincing.

So the object hasn't just magically increased in polygon count. It's a trick of light and shadows to give the illusion of something that's more detailed than what we have, actually. So a combination of both the texture and a bump map can give us a fairly convincing result of something that's high polygon. But in this case, it's much lower. It's more than order of magnitude in terms of the lower amount of polygons that we have.

And there you go. Hopefully this has given you some idea of what we can do with this technique. Like I said, some examples would be things like interactive walkthroughs for web applications or mobile. Virtual reality is another good usage scenario for this. But even things like 3D maps or games.

So there we go. I hope this has inspired you to use this technique to create basically some projects on your own. Thank you very much.