Better Site Planning Using Commercial Satellites for Reality Capture

JEREMIAH JOHNSON: Hello, everybody. Welcome to this session, Better Site Planning Using Commercial Satellites for Reality Capture. I am pleased to be speaking to you at this time in Autodesk University 2023.

I'm really, really excited to bring this topic to Autodesk University. We talk about reality capture a lot. There are a lot of sessions around reality capture. I am really excited to talk about a brand new way to think about reality capture. And I want to expand how we think about reality capture, especially for some of the projects that we have there are very unique projects where this type of workflow might be well suited for.

I'm Jeremiah Johnson. And I've been with Esri since 2019. Before Esri was a drone subject matter expert. I worked for a company that did reality capture using drones. I did that for a number of years before that product and team was acquired by Esri in 2019. Since joining Esri, I have expanded from just being drones to incorporating terrestrial, and aerial, and as you'll see soon some satellite based imagery acquisition. So it's really, really exciting times for this space.

This is my fifth Autodesk University presentation. If you've seen my talks in the past, it's usually been around drones and what capabilities drones can bring to those of us in this space. This is my first talk on satellites. So like I said, I'm really, really excited.

I am from Texas. I'm from Central Texas. I went to Texas A&M with a degree in remote sensing. But today, I'm here in Berkeley, California just across the Bay Bridge from San Francisco, where Autodesk is.

So first, I want to kick it off with a little bit of a thought experiment, maybe some reflection. Reality capture is a buzzword. You see this a lot and I just want to make sure that we as an audience understand what we believe is true for reality capture. What is reality capture? Is it a verb? Is it a subject?

Personally, I view reality capture as a verb. It is the idea that you can take sensors, whether they be active sensors like LiDAR or passive sensors like cameras, and we can take little snapshots of our world, bring them together, capture reality into a 3D sort of experience that generates an as-built of our environment. And we can use different sensors to capture this reality. I would sometimes even consider RTK GPS as a form of reality capture, albeit not the most exciting form of reality capture.

So for the purpose of this talk, let's define reality capture as using active and passive sensors to capture the world, to create 2D and 3D data sets for our workflow. That's the basis that we're going to use for this discussion.

If this were a live audience, I would maybe propose think about some trends in reality capture. Think about where reality capture has gone or has been in the past maybe six, seven, eight years since it's really become a buzzword. A lot of us associate reality capture with terrestrial LiDAR. You know, FARO sensors or like BLK 360 sensors.

Some of us, like myself, came into reality capture from this drone space where we use flight planners to fly drones in our projects. The drones take a lot of pictures and then we pull those pictures into a software to convert those pictures into 2D and 3D data sets. Maybe that's how you view reality capture.

I want to talk about the next trend in reality capture, which is using commercial satellites to do the capture for us. This is really exciting. It's not as sci-fi or out of this world as it might sound. In fact, it's really easy. And we're going to do it semi-live for you here. I'm going to show you the step-by-step workflow on how I did it.

But in order to talk about reality capture from space, there's really something I want to bring to the table. And that's how important one company has been to this entire industry. And that is SpaceX. Many of you are familiar with SpaceX. SpaceX is a phenomenal company. But they have really changed the way this commercial satellite arena has been operating.

In 2010, SpaceX launched kind of a little funny pun there the Falcon 9 rocket. This was a rocket that was designed and capable of delivering serious payloads into Earth orbit. So this wasn't mini SATs or microsats this was a rocket that was intended to bring cargo and people into space. And we saw that happen recently. The first crewed launch was a few years ago, maybe two or three years ago.

In 2010 was the first flight of this rocket. The main component of the Falcon 9 was this reusable booster. This means that in 2021 there were 31 Falcon 9 rockets launched. There were only two boosters made that year, or two new boosters were used that year. So the reusability of this rocket was really key for the success of this SpaceX program.

This cut costs for our customers tremendously. They would be buying space on rockets that would be disposable. We for decades had been launching rockets into space that its intent was to deliver the payload, and then burn up in the Earth's atmosphere never to be used again. Those of us now living in this post Falcon 9 world, that's a pretty crazy concept to think of that we would just completely burn up an entire rocket. But that was the basis for how we launched satellites up until Falcon 9 was commercially available.

So what did this do? The cost to launch originally was something about $10,000 per kilogram of payload, so pre-2010 this was the going rate. If you had a satellite that you wanted to launch it was roughly $10,000 per kilogram. This significantly reduced the number of companies that were able to do this. In fact, the majority of the satellites were owned by the US government, the Department of Defense. And any private companies that had satellites in space were funded by the government the likes of Maxar and such that had satellites that they would spend hundreds of millions, if not billions of dollars into, in order to launch into orbit.

After Falcon 9, this cut the cost down to $2,500 per kilogram after the Falcon 9. Look at that. I even scaled these bars appropriately, so that we can understand the sheer magnitude of that cost cutting after the Falcon 9. Today, the going rate for a Falcon 9 is $1,500 per kilogram.

In a matter of 12, 13 years, we've cut costs. We've cut costs to almost 10% of what they were. Has that done to the commercial satellite market? Well it's created a boom. The free-falling cost to launch satellites has created this economy of nanosatellites. We're now seeing startups being able to design and build satellites, then launch them on SpaceX rockets in order to test different types of constellations, different types of sensors, all being venture funded. It's just because of how cheap it is now to launch these satellites.

Categories making up the most constellations in orbit are communications and Earth observation satellites. In this conversation we're having today, we're going to be talking about Earth observation satellites, EO.

This has caused a democratization of satellite imagery. Satellite imagery is no longer owned and controlled by governments. Right? We now have free options like LandSat and Sentinel. These have been around for a long time. We have the legacy premium options like Airbus and Maxar who have been providing imagery to our governments for a long time, very high resolution imagery. But now we have more accessible options, options like Planet and Blacksky.

Premium satellite content, this is Maxar, Airbus. They provide very high-resolution imagery but it comes at a cost. These satellites have global coverage. Maxar and Airbus satellites are relatively similar in terms of capabilities. Maxar is an American company and Airbus is a European company. These satellites, just to give you a little bit of scale, these satellites are about the size of SUVs, maybe small buses in some instances. But these are relatively large satellites but they have a lot of capability. They're meant to stay in orbit for a long, long time, measuring the Earth with very high-resolution sensors and bringing those imagery down to Earth through ground stations located around the world.

Well, let's talk about these more accessible options. These options that were not available pre-SpaceX Falcon 9. The first that I wanted to mention is Planet. Planet Labs is located in San Francisco. Believe it or not, they actually manufacture the satellites in San Francisco as well. They consider their satellites doves, and they call their constellation or their launch sequence of these doves as flocks.

So they'll typically launch a lot of these in one go. They have these containers that launch alongside other satellites that SpaceX might be launching. And they'll shoot them out in a specified interval, so that these flocks of doves will orbit the Earth in a very organized manner.

Planet Labs, they no longer hold the world record. But they broke the world record of the largest private satellite constellation in orbit. This was not too long ago. I think this was in 2015, 2016. They had 150 satellites in orbit.

Planet Labs, because of the number of satellites that they have in orbit they're able to map the entire landmass of Earth every 24 hours. Think about that, the entire land mass of Earth photographed every 24 hours. They have daily updates of the world's surface. Again, just a sense of scale, we were talking about Maxar and Airbus satellites being the size of SUVs or small buses. The planet doves are about the size of a breadbox.

So they're about this big, and they're actually meant to stay in orbit for not too long, I want to say less than five years. After that time they burn up naturally in the Earth's atmosphere. And the Earth's atmosphere operates as this natural trash composter. So these things don't stay in orbit for too long. They don't stay in orbit and become space junk. They naturally burn up, and Planet just continues to manufacture and launch these.

The next organization that I wanted to talk about is Blacksky. They named this slide, it's kind of funny, Selfies from Space. But this is endemic to how these satellites capture the Earth. Blacksky specializes in low latency observation from space. So their satellites actually take single photos or a burst set of photos that are tasked from users on Earth. So Planet satellites are a more these sweeping type of capture. So they will capture the entire Earth in swaths. That's how they operate.

Blacksky, they have 30, 35, 38 satellites in orbit. And these satellites have gyroscopes on board. And they can orient themselves however they need to orient. And they're able to orient themselves and take a single picture, almost like a selfie from space. And then they send that single image down really, really quickly.

You can see on the left. I have their orbits in place. And they have a really unique orbit in that they're not orbiting the entire Earth all the time. They chose, I believe, it's plus 40 and minus 40 latitude, which is where the majority of human population is. And that's the orbit that these satellites live in. Because of these tight orbits, these Blacksky satellites are over some point of this 40 by 40 latitude ring every 30 minutes, very, very fast, high revisit rate.

For this workflow, I'm going to be showing how we can use Blacksky satellite tasking to build a reality capture model of Angel Island in the San Francisco Bay. Let's take a look at how we do that.

So let's talk about tasking a satellite acquisition. Through a web browser, Blacksky enables the precise tasking of photos from space. You can task one photo, two photos. You can task over a period of time. The more photos you take of an area just like with any reality capture strategy, the more photos you take of an area, the higher resolution the elevation and the details will be.

For this example, I'm going to show how it can still be done using only two photos. Remember, I'm coming at you from the drone space where you really can't do too much with two photos, three photos, four photos. In the satellite space it's a little bit different. You can do a lot with only two photos.

So let's look at how we task it. So with Blacksky they have an application called Spectra. And what I'm going to do is click on Angel Island, select the product. Instead of a single photo, I'm going to ask it to do a stereo pair. The stereo pair means that as it's orbiting over the desired location, it's going to take two photos at a designated angle of incidence. It happened really quick there.

But in the tasking section, you can define what that angle of incidence would be and it will follow that. So as the satellite orbits overhead, it's going to go click till it gets to the different angle as specified, and it's going to go click.

Like I said, the revisit rate is every 30 minutes. So it is almost ready to go immediately after you hit send. After the photos are taken, they are immediately beamed down to a ground station. I can get these photos in my inbox within 90 minutes of capture, pretty quickly.

All right. So let's talk about the keys to a successful acquisition. It seems obvious. But I did want to specify that clouds won't work for satellite acquisition. There's no way for a satellite overhead to see anything underneath that cloud layer. So it really only works if there are limited to no clouds. Now that being said, within that Spectra tasking app that Blacksky has, you can specify what percentage of cloud cover is acceptable. And it will not send you or it won't collect if the clouds are there.

So typically what you will do is you'll set, hey, I want a stereo pair or I want a burst collection over this area. And try for a week, or try for a day, try for three days, depending on the weather for that area that you plan ahead for. You can say, hey, keep trying. And then when it finds a time that works for your specified parameters, it will then take the photos and send those photos down to you.

Incidence angle, so incidence angle is just a fancy way of saying what is the relative angle between the two photos or multiple photos that it takes. Optimally, you want to have an incidence angle greater than 5 degrees. If you're thinking like me, again, from the terrestrial space or the drone space, 5 degrees is not much. But from space, it is significant enough. Obviously, an incidence angle greater than 5, when you get to 8 to 10, 12 degrees you're getting more and more and more oblique from each other.

So it'll actually produce higher quality 3D. So you definitely want it greater than 5, but no less than 5. Because less than 5, it'll either fail or not produce the 3D that you want.

Something else to consider is that natural features are more successful than built structures for reconstruction. And this is just related to the resolution of the imagery that comes down. I know that some of us are used to, when we get on our phone and we try to navigate somewhere, we turn on the satellite view. In the satellite view we can see all the way down to our cars and the paint stripes on the road.

Don't be misled. Google and Bing, they use satellite when you're zoomed way out. And then they use airplane or aerial data when you're zoomed way in. And so the satellite resolution, even really good satellite resolution isn't never enough to really zoom all the way down into what you think is satellite view when you open up something on your phone. So for Blacksky satellites the resolution is somewhere around a meter, a little less than a meter, so a little less than three feet. And that's the size of a pixel.

So for something like this island, it's great. We're going to be able to map this island well. But individual buildings, cars, even definitions of roads and bridges, are not going to be seen very well in this imagery. I guess you can see it, but it won't have enough definition for us to be able to reconstruct it in 3D. So natural features are a lot more successful than built structures.

The same-day capture is ideal but not necessary. Again, when you capture with drones or even terrestrial stuff, you tend to want to capture everything not only the same day, but the same time of day. You want to do all your capture in the morning or all in the afternoon to prevent the difference in lighting, in structures, to produce a successful model. For satellites it's less necessary to capture things within the same time period.

The differences in how things look from space really don't change too much as the day goes on. Obviously, we have things like shadows. So if you have skyscrapers or things with the shadows that are moving across maybe that's something to consider. But that's really just a visual thing and not a reconstruction thing that you're looking at then. So same day capture ideal, but you can certainly capture images over the course of a week and stitch them all together into a successful model.

As with any reality capture project, you want to capture as many images as reasonable given time and budget. So for this, again, I just want to show you what's possible. So I only took two photos of Angel Island. If this was a real project, if I was going to do this for real, I would do a lot more than two. But not hundreds, not thousands, but maybe 10, maybe 15. For satellite captures like this, Blacksky they'll charge per photo.

And don't quote me on this because I don't work for Blacksky. But just to give you a range of what you should be expecting per photo, it'll be between $200 and $800 per photo, depending on what kind of plan you have. So that's the budget that you should be thinking of when you're capturing photos like these.

All right, once we've got our photos from Blacksky, we're actually going to open ArcGIS. So as I mentioned in the beginning, I work for Esri and ArcGIS is our platform. Esri and Autodesk have an alliance partnership. So a lot of our products work together. A lot of you have maybe seen ArcGIS connectors within Civil 3D, and InfraWorks, and the like. They play really nicely together.

In order to do this step, it does require ArcGIS. I'm going to be using ArcGIS Pro. It gives me the tools that I need to take this data and process it. And I'm going to use photogrammetry methodology that is very similar to aerial triangulation mathematics.

Because it's a satellite, its location and orientation is known very, very precisely. There's no vibration or things going up there. You're not banking in the wind. There's no turbulence. It's not a drone that's drifting around. There's really not a lot that's affecting its course. And so being able to precisely know exactly where the satellite is when it took the photo is pretty easy to determine. Because of that, we can use traditional photogrammetry methodology in order to create a 3D reconstruction of the imagery.

So for this, I'm going to take my two images into ArcGIS Pro. And I'm going to pause this here really quickly, because I wanted to walk you through the tools here. So within ArcGIS Pro, we have tools and a toolbox. And I'm going to describe to you what each of these tools are.

So if you look at my model builder, anything that's blue is an input file. Anything that's green is an output file. The yellow boxes are the actual tool that is running. So the first step is to create a mosaic data set. In ArcGIS a mosaic data set, it is a layer or a feature that consists of multiple images. So this mosaic data set will contain the images that were captured by the Blacksky satellite.

So I'm going to create a new mosaic data set. I'm going to give that mosaic data set a name. Then I'm going to take my two Blacksky images. So you see the Blacksky image 1, Blacksky image 2. Those will be the inputs into the next tool, which is Add Rasters to Mosaic Data Set. We're going to pull those images in. I'm going to give it a spatial reference to store those images in. And then it's going to create this mosaic called Blacksky in this case.

Now here's where it gets cool. The next tool says Build Stereo Model. And the only input for this is the mosaic data set, so this tool assumes that the images inside of this mosaic data set have been captured at an appropriate angle of incidence in order to build stereo models. So it's going to run. It's going to produce a stereo model. And then the next step is to generate a point cloud. So it's going to take the stereo model which is basically a mathematical model of how it tied everything together, and then it's going to generate a point cloud.

From that point cloud, we can use that point cloud as is or what I did was I interpolated from this point cloud a digital surface model. And that's what I ended up using for the rest of this workflow was this raster GeoTIFF that contained elevation values for the entire island. So that is the step-by-step process in ArcGIS Pro in order to convert these two images into a digital surface model using ArcGIS Pro.

But you see what I have in here, this window here. If you're not familiar, this is called ModelBuilder. And it allows me to string tools together. I can save this tool in my toolbox. And so the next time I do this, all I have to do is give it two images or however many Blacksky images, hit Run, and then it'll run through the entire model and spit out the output digital surface model. So it makes things really easy for me.

What's really nice is because we're using typical traditional photogrammetry methods, this process takes really, really fast. So again, drone world, terrestrial world, you're processing data sets. It might take 15, 30 minutes, an hour or three hours depending on the project size. This stereo model to output digital surface model took 3 minutes to run, really, really quickly.

So again, given good weather, I'm able to task a satellite collection, receive the images within 90 minutes, and then run this to get a digital surface model in 3 minutes. That's 93 minutes from tasking to having an output. That is really, really powerful. And frankly, it feels crazy because when I was a kid I would watch movies of people tasking satellites from space. And now I can do this from home. I can do this right here in front of all of you and generate data sets useful for my organization using the satellite tasking.

We talked about how to do this with two photos. I just want to show you a typical capture. So this is what I would do typically for an actual project. If I turn this on, you'll see how many images I've collected. So this is something like 15 images of Blacksky satellites over northern Italy. And this is the result of that. So I'm able to get more detail on some of these buildings located in Northern Italy, some of the detail on that the mountains there. And all of that was possible because I collected more than just two. I collected 12 to 15.

But don't let that scare you. I'm going to finish this workflow with the stereo pair that I started out with. And I believe you'll be impressed with some of the results that I was able to get.

All right. So now that we have our digital surface model created from our stereo pair from ArcGIS, the next step is to import the 3D data into Civil 3D. You can use the digital elevation model as is. It is a georeferenced GeoTIFF. So you can pull that raster into Civil 3D, or you can generate contours from that raster using ArcGIS Pro into whatever interval you want, and then use those contour lines in Civil 3D to create a surface. Either way works, just depends on your preference and workflow and how you like to create surfaces. In this example, I'm going to use the contours.

So here I'm going to open up Civil 3D. Those of you with eagle eyes are going to notice that I'm opening Civil 3D Metric. And that is because the satellite imagery comes in as metric. All of this can be changed. You can change all of the settings in ArcGIS Pro whenever you run this. But to be honest, I didn't do anything. I just took the raw images from Blacksky, ran it through ArcGIS Pro to get the digital surface model to create the contour. So everything here is going to be in metric. And so that's why I'm using the metric stuff here.

So once we have Civil 3D loaded, again, I'm going to define a coordinate system for this. Now here's the tricky part. I know I just talked to you about metric stuff. But my map was in State Plane California. So here I am going to select California State Plane Zone 3. And that's what my design is going to reside in. That's just what everything else is going to be in.

The next thing I'm going to do is I'm going to open my tool, create surface from GIS data, define how I want my surface to look like, and then import that shape file that was generated by ArcGIS.

I'm going to choose which attribute in the shapefile is the elevation, and it's going to run and produce my surface. All right. My surface is created. So if I zoom Extents here, you can see my surface loaded into Civil 3D of the island. Isn't that cool? So this is 1 meter contours. If I turn on my map, I can just verify that it's located in the correct place. And here is my surface of Angel Island collected in 93 minutes, located correctly here in Civil 3D. Super mind blowing, it's crazy that it happens that fast. And it's crazy that if you look at the contours, the contours turned out that well.

A couple tips here. Pay attention to the coordinate reference systems. I talked to you a little bit really quickly about how they come down as metric at first. If you want to change that, you can change it. But again, when you're stringing things along in a workflow like this, pay attention to the coordinate reference systems. Satellites aren't going to collect in State Plane. They just won't. They're a global constellation. They're collecting data all around the world. The data is going to come in something different.

Like I said before, the resolution of the imagery will mean that features like buildings are not going to be defined. Also consider that dense forests will behave like terrain. Again, it's very similar to clouds. There's no way for the satellite to see underneath the tree canopy. So if the tree canopies are tight enough to where the satellite can't see below it, it's going to treat that tree canopy as a surface itself. So don't think that you're going to get ground points almost like you would LiDAR. Individual trees, similar to the reasons why buildings wouldn't be shown, individual trees would not be shown just due to the resolution of the data.

All right. So what's next? Let's take our surface, and let's create an InfraWorks project, and start to build out a site plan of the island. So those of you who are familiar with InfraWorks know that this is a really easy workflow. All I'm going to do is create a brand new InfraWorks model, named Angel Island, and give it my coordinate system. I'm just going to give it the same coordinate system that I've been working with this whole time.

For model extents, all I'm going to do is give it the extent of the contour shape file that I generated from ArcGIS. I figure that's the easiest way to define extents for the project, so I'm just going to give it those contours, and move on. And I believe I'm going to leave everything else default.

All right. So we're going to create our InfraWorks project. Here it is. And the next step is to import our surface. So I've got something that I forgot to mention is that within Civil 3D, I took my surface and I exported it to LandXML, and I'm going to use LandXML to import that surface into InfraWorks. So that's me doing this here.

Look at that. Here's Angel Island in all its glory here sitting in InfraWorks now. So already, I have a site plan or a rough site plan for the entire island. Again, I collected this data in 93 minutes without ever having to set foot on the island. I was able to do all of this from the comfort of my own home or office-- collect, process, and analyze the island. This is really, really cool. But I want to show another workflow that we can use to make the site plan even more detailed.

For that, I'm going to turn to artificial intelligence. The satellite imagery, so the actual images that come down can also be used to analyze and detect constructed features like buildings, roads, or even natural features like trees. This analysis can also be run online in a browser using ArcGIS Online. So you can either do this in ArcGIS Pro or ArcGIS Online.

Before we were just using the satellite imagery to create 3D content. Next, we're going to use the satellite imagery to actually start to extract features that are already there. If I go to Esri's Living Atlas of the World and I search for DPLK, this stands for deep learning package. You can see all of the different deep learning packages that are accessible and free to users of ArcGIS.

One of these deep learning packages is the Building Footprint Extraction Model. So what this is going to do is it's going to analyze imagery from high-resolution satellites, or airplanes, or even drones. And it's going to create boxes or footprints for buildings.

So here I am in ArcGIS Online. I created a new map. And I'm going to pull in the image, the single image created from my Blacksky acquisition. Next, I'm going to go to my Analysis tab, and just to go down to Use Deep Learning, Detect Objects Using Deep Learning. For input layer, I'm just going to input that raster image.

For the model I'm going, to go to the Living Atlas, search for building footprints. And then select that model that we were looking at before. I'm going to keep everything else as default, give it an output name, and hit Run. This is going to run inside of ArcGIS Online. It's going to analyze this imagery from top to bottom. And it's going to extract building footprints from the imagery.

It's going to save those building footprints in my ArcGIS Online account, which I can then access through InfraWorks. All right. So let's go back to our InfraWorks project. Inside of InfraWorks, I can connect to my ArcGIS Online account and navigate to the building footprints that were saved after it ran its analysis. So here I'm going to deselect some of these other layers. I'm going to create a buildings layer using this objects detected layer. And now it's going to pull in all of my building footprints as structures on the island. Look at this. Isn't this incredible?

This is rough, I'll admit. But again, this is how powerful it is to have this type of automated analysis done completely remotely. This is an island. This would take me a bit to get to, a ferry even. But I can start to generate a site plan, including 3D data, buildings, roads completely automated using AI and machine learning.

So let's recap. Commercial satellites have become cheaper than ever to design, launch, and operate. Photogrammetry workflows that we've previously associated with aircraft, and drones, or even terrestrial reality capture can also be used with satellite imagery. A lot of the workflows are exactly the same. Just because the satellites are capturing data in very similar ways, they're just a lot higher up.

In this workflow we saw how we could use a combination of ArcGIS and Autodesk products to quickly and remotely create a site plan of an entire island using satellite imagery, again, in what 93, 95 minutes from start to finish.

I hope that this workflow not only gets better and more scalable over time. But I hope that this helps organizations like those here at Autodesk University to think about how we can perform reality capture or pursue projects in areas of the world that we might not have budget to get to or maybe it's difficult to get to. Think like data centers, or dams, or construction, road construction projects. In places of the world that are relatively difficult to get to, we can now use commercial satellites and off-the-shelf shelf workflows, no custom scripting or anything like that-- off-the-shelf the shelf workflows to generate accurate site plans.

Now, I will never say that this is here to replace survey and surveyors. But I hope you understand that in the pursuit phase of projects or even just in the preliminary stages of projects, having this capability is really, really powerful. And with that, I would like to thank you for joining and you can hit me up on LinkedIn, Jeremiah Johnson here at Esri, and hope you guys have a great rest of Autodesk University. Thanks.