Using Entwine LiDAR Data Sets Within Civil 3D and ReCap Pro

JUSTIN EHART: Thanks for joining me. Today, we're going to be covering leveraging Entwine LiDAR datasets within Civil 3D and ReCap Pro. I know that's a mouthful, so if you want to go with the Too Long; Didn't Read, we're going to exploit online data for use in Civil 3D.

Who am I? Why am I teaching you this session? Well, my name is Justin Ehart, and I work with Ware Malcomb. I'm their engineering BIM manager. And I've spent the last 28 years working in architecture, civil engineering, and land surveying.

I've previously spoken at AU & BiLT Europe in 2019 over Drone 2 Design. And most recently, I spoke at BiLT North America this year covering Dynamo for Civil 3D. And I absolutely love coding, whether it's Python, IFC.js, .NET, PowerShell, Dynamo, LiSP. You name it, I probably have my fingers in.

And most recently, I've created an engineering and Revit chat bot in Microsoft Teams that answers questions for users. I'm currently working on some Civil 3D TensorFlow, artificial intelligence integration. But primarily, I'm focused on geospatial and data science applications. And I'm an avid comic book collector and sneakerhead.

So a little bit about Ware Malcomb. In 2022, we had our 50th anniversary of the firm. We are a full service architecture, interior, civil engineering, branding, sustainability, building measurement services, land surveying, and a workplace strategy and change management firm. It's a mouthful, I know. We have 28 offices across North and South America and we pride ourselves on not being like every other firm. We have our own R&D lab called WM Future.

So what can you as a participant expect from this session? So our course overview. In this session, we'll explore Entwine Point Tiles. And currently, there's over 43 trillion with a t individual points. And we're going to figure out how we can consume them quickly inside of Civil 3D as a surface.

As more large data point cloud sets become available or ubiquitous inside of the AEC community, we've got to find different open source libraries and software that's dedicated to manipulating the data sets so that we can harness those. And so this session, we're going to learn how we can quickly find that LiDAR data for your area of work or even a business development case.

We're going to quickly clean and generate a consumable point cloud within Civil 3D. And we're going to learn how we can create better-informed proposals before a contract is created. And we're going to gain a more refined understanding of the grading constraints early in our project lifecycle.

So how are we going to accomplish this? Well, we're going to geolocate and create a new drawing in Civil 3D from the map and save it. That's step one. Step two, we're going to gather. So we're going to browse to the national map and define an area of interest and download those LiDAR files. Then we're going to clean it using an open source software called Cloud Compare.

We're going to use this to filter, subsample, and merge our tiles based on the ground point, classification 2. Then we're going to create a ReCap Pro project with an LAS file and convert it to ReCap. And then finally, we're going to round robin. We're going to go back into Civil 3D, import that ReCap file, and create a service from the point cloud.

You might be asking yourself, great, but what is Entwine? Well, it's a data organization library for these massive point clouds. They're designed to handle data sets of trillions of points. That data that's stored there is completely lossless, even in terabyte-scale datasets. So what that means, there's no point loss. There's no metadata loss. And there's no precision loss. And even better yet, Amazon has agreed to pay to convert the files to EPT and to store the data on their AWS S3 servers.

So step one, geolocate. So we're going to geolocate a new drawing. You can save it as a drawing template or a DWG. You can use the command EDITDRAWINGSETTINGS on your command line. Or you can browse to the tool space settings, right click your drawing name, and edit the drawing settings.

So what I like to do is set it to the correct coordinate system that I'm working in. I usually do a lot of work in state plane coordinates, so a NAD83 in the US. You might be using UTM elsewhere. And if you have a boundary, you can use that to import your partial shape files. So let's kind of see how that's done.

So right here, I've got just a blank brand new drawing. You can see that the coordinates down here are 0, 0. And so this is where I talked about you can go into tool space and right click, Edit Drawing, and browse to your coordinate system. And here, I'm going to set it for New Jersey. NAD83, New Jersey State Planes, US Foot.

I like to script everything, so I'm going to just go to my trusty tool palette. I'm going to click go to Elizabeth, New Jersey. It turns on my map, zooms me to that area. So I'm going to scroll down here. The site we're going to explore today is on the border of Delaware, New Jersey, and Pennsylvania, just to kind of see how much data is available to us.

And this site looks good. We're going to trim the map. And then we're going to turn off the background map, so that it doesn't try to recalibrate as we zoom in and out on the site. And now I'm going to go ahead and import-- oh, I can show you that the drawing settings were set correct here. Great.

Next, let's import our boundary. Again, I'm going to use a shapefile, because I'm going to trim my surface to this later. And we can see that the coordinate system of my drawing is New Jersey, but the input of the shapefile I'm bringing in is lat, long, 84. So I'm going to assign my drawing layer. I'm going to add my object data. Import polygons as closed polygons. And there's my boundary.

So if we go over to the Properties, we're going to see that it is set to lat and long. And I'm always a little clunky. I always have to find it. Every GIS information is a little bit different. So we can see that it's lat and long. And the acreage of this site is 462.16 acres.

Great. So now that we've geolocated our drawing, we can go ahead and gather our LiDAR data. And so what we're going to do is browse to the national map. So apps.nationalmap.gov. And then if we scroll down, we're going to see LiDAR Explorer. Once we go into the website, we're going to define an area of interest and download those LAZ files. Let's look at that now.

So here's my drawing of this Delaware River Project in New Jersey. And if we go to the usgs.entwine.io website, we can see this is where we have coverage of LiDAR data. And as of August 4, 2022, there are 43 trillion points of data on there.

Here is the apps.nationalmap.gov. It's the 3DEP, so 3D Elevation Point LiDAR Explorer. And if we click here where it says show where LiDAR data is available, it's going to mimic that JSON file from the other website. And so we can see this is where we have coverage.

So I like to work with a satellite image. So I'm changing the base map right now. I'm going to zoom in to the area. So again, this is where Delaware, Pennsylvania, and New Jersey border along the Delaware River. So that looks like our site from Civil 3D.

Let's go ahead and define our area of interest. You can do that by holding down the Control key and dragging your cursor or clicking this draw area of interest icon. I like to use the icon, because then it starts aggregating the data for me automatically.

And while it's thinking, you can see the progress bar across the top. So the downloadable products within our area of interest, let's zoom out a little bit. You can see the different LiDAR projects that are available. And some of the dates on these, 2006, 2015, 2013, '07. A little out of date.

So we're going to try to get the most recent LiDAR data that we can. And so by clicking the LiDAR with an area of interest, it will aggregate that information for us. We can use the filter at the top, if we wanted to make sure we get New Jersey. Or we can just start scrolling down and seeing what the most current is for this area.

Down at the bottom, we're going to get there now. We'll see New Jersey and we'll see five different tiles. We're going to go ahead and download all of those. It's a quick download. Some of these .laz files have 15 million points in them. You can see how quick they download.

Great. I think those are downloaded. And so you can zoom out and see the area that we're working in. So again, Pennsylvania, New Jersey, and Delaware. Awesome. So next, we're going to clean the data.

So we are able to clean the data in Civil 3D. But we kind of have to round robin. So what I like to do is there's this open source software called Cloud Compare. And we're going to browse to this. And I'll show you in a moment. You can just search on Google for it.

And once installed, we're going to open and browse to our downloaded files. And we're going to filter by a value. And then we're going to do a subsample and a merge and then save it as an uncompressed LAS file. So let's go ahead and click on this.

So here I am on just your favorite search engine. You can just type Cloud Compare. And we're going to go to the results. And we'll see this one here, Cloud Compare. It's an open source project. On the Download tab, you can go here and select the download for your operating system.

And it is freeware. So if you are able to donate, it might be in your best interest. You're going to use the software a lot. We've launched the software here. And so I'm going to go to File and Open. And I'm going to browse to my LAZ files. Those are the compressed files that we downloaded.

And we're going to toggle off all of these check boxes here on the standard fields, with the exception of classification. And we're going to go over classification in just a moment. And since we have multiple tiles, we're going to say apply all.

So this first tile we're importing, there's 24 million points in it. That's a large area of work that has a lot of detail. This is the longest portion of this. So it's going to take just a few moments for us to aggregate and bring those five tiles in.

And so this software gives us the ability to merge all of these tiles into one. It gives us the ability to filter them by the classification. And all LiDAR information does have a classification. So classification 2 is bare ground. It's the common ground shots.

If you were wanting just the buildings on the site, you could use seven. Those are classified for buildings. And here in a moment, we're going to go to a website that shows all the different classifications of the files. So we'll just let it work and do its thing here.

We're almost done, down to the last one. Great. So what we're going to do is we're going to go over and select. And using the Control button, select each of the point clouds individually. You can see them highlight on screen.

Then we're going to go to Edit. And then we're going to apply a scalar field filter by value. And again, 2 is the common ground shots. So that's what we're going to filter by. And we're going to see all of the tree canopies and everything go away here. Those are automatically highlighted.

So let's go ahead and do a subsample. We don't necessarily need 100 million points of this point cloud. So we're going to do a minimum space in there. And so we're going to take it down from about 100 million points to roughly 10 million points.

All right. This is the last one. Now, we could export these five individually. But while we're here, we might as well merge them together. So they're all highlighted again, because we just made an edit. And we'll just go down to merge.

And we can see the five point files in there. Honestly, that looks better. 5.3 million points in this area. We can handle that very well in Civil 3D. And if we click, we can see that that one is 10. We see just the different values that are there. So let's go ahead and save this.

And saving the file format as a .las. So it's an uncompressed file of what we downloaded from the web. We're just going to click OK, set the original resolution. And then ultimately, we're saving a point file that has about 10 million points.

And while you're in here, you're able to explore around and view some of the features on it. Just quickly orbit, just like you can in Civil 3D. And then here, we're going to show the lighter classification. So you'll see that 2 is ground, 3 is low vegetation.

And they've kind of changed some of the classification flags. They've updated it to allow for some more. But 2 is still going to be ground. That's why we filtered by classification 2. We'll see in here that there's bridge decks, road surfaces, water. But yeah, any LiDAR formatted point cloud should be classified to make it easier on you.

Awesome. So now we're going to convert our data. We're going to take that LAS file that we saved and we're going to convert that to a ReCap Pro file format so that we can bring it into Civil 3D. And while in ReCap, we might as well do a little cleanup. So we're going to clean up that area outside of our area of interest and save and optimize.

And then I also like to do a double check. I kind of like to export an additional ReCap file called Cleaned or Cropped so that if I need more data, I can go back to the original. There we go. So what we're going to do in ReCap Pro is we're going to launch a new project and import a point cloud.

And give it a project name, if that's how your company works. Or if you're just doing this as an exploratory or BD tool, you might not have a project name yet. So we're going to go to-- I saved it in my Downloads folder. And then we're going to import files.

ReCap is very simple to work with. And then this is the longest part, where it's indexing the scans. It could take 30 seconds. But some of the nice features of ReCap Pro is you're able to define areas that you want to include only or exclude. You can use a fence, a window. There's a lot of neat features in here. So if you're not familiar with it, you're going to get familiar with it fast using this workflow.

I feel like we're going to get close on this. There we go. It's starting to do something now. Great. Now we can click Launch Project. And we're going to see that point cloud that we had merged and cleaned in Cloud Compare appear on our screen.

I like to work top down. So we're going to go ahead and change this to a fence. This is essentially the lasso command. And we're just going to draw around this. What I'd like to do is I'd like to clip this inside area. So I'm left with this outer area that isn't relevant to my workflow on this particular site, so I'm go in here and delete it.

And the functionality of ReCap is a little different from Civil 3D. I always have to find my bearings on it. So we're going to unclip the inside. I'm going to save it and optimize the point cloud. And as I mentioned, I do like to export, just as a double check sometimes, or if I want just the reassurance that I have everything that I need.

So we'll just browse. I've got a ReCap folder that I like to save those items in. And again, this is where you could label it as your project tracking number, if your company uses one, or whatever that you need. And then we'll go ahead and unify your project scans. And this goes pretty quick. Even though it's under 10 million points, it still works well.

Come on, baby. All right. We can close out the software. And to the last part, creating our surface. So we're going to, like I said, round robin right back into Civil 3D. And then on the ribbon, we're going to go to the Insert tab and Attach Point Cloud.

Then once we have that attached, we'll be able to see the outline of it, and actually see it when we're in 3D view. Then on the Home tab, we're going to go to the ribbon and surfaces. And we're going to create a surface from the point cloud. And then after that, I'd like to generate a snapshot of the surface and delete the point cloud.

So why don't we go in there and see how it's done? So here's our drawing. And on the Insert ribbon to the Point Cloud, Attach. We'll bring that over. And we'll browse to that ReCap file. Great. We see it come right in.

And again, we are working in a coordinate system. This was all done in lat and long. It's really nice that the software has it just marry it perfectly. So we're going to go in here. We're going to name our surface. And I typically just use EG for Existing Ground. But I'm really particular on the description.

I want to make sure that, should somebody go in there, they realize that this wasn't generated by a surveyor. The LiDAR information that we're putting in here is good to 0.24 meters or 0.8 feet. So it's sub 1 foot. But you still don't want to design from it.

So I like to put in here when it was downloaded, where it was downloaded from, and that it should be used for preliminary design only, not for final design. So you cannot overcommunicate when you're working inside of design software. That's exactly my feelings on it. So I'll put in here preliminary design only.

I'm going to change the style of my contours. Now I'm going to click Next. I've already done a selection on it. And then I changed it, because I don't want it to interpolate. I want it to just take the ground shots that I had given it.

And so we can see down here that it's going to create the point cloud from the surface. And this process takes maybe 20 seconds. We'll start seeing it catch up, let's say. It'll start aggregating that information for us.

And each point cloud is a little bit different. The bigger they are, the longer it takes. So we can see that now, to adding points to the surface. Three, numbering the points in the triangles. We're getting close. Computing contours. And we should have our surface now. Displaying the contours. Great.

So what I like to do is go into the surface itself. And I want to set a boundary. And that's why we imported that shapefile in the first video-- in the first process of this. And so I'm just setting the boundaries from a shapefile. Super simple. I'm going to zoom in here. Or maybe I'll just cowboy it and select it from afar.

And this shapefile, again, is 462 acres. But along the river, it's very jagged. There's a lot of different vertices that it's following, when it was established, this parcel. And I promise my Civil 3D won't crash. It's just really struggling to think.

I could have picked a different site that was more inland with a little cleaner boundary. But that wouldn't be fair to you. And so we should be pretty close to it. There we go. And so if we were to delete the point cloud now, the surface would go away. So what we need to do is go back into the surface and create a snapshot.

Again, I'm working with a very temperamental parcel as a shapefile. Again, I should have probably picked something a little less-- with fewer vertices. But it just wouldn't be fair. We can't pick and choose our sites. That's our developer's role.

Now that we've created our snapshot, we can go in here to our references and we can detach or unload our point cloud. I'm going to switch it back to a 2D wireframe. And since I've already done that snapshot, I'm going to go ahead and just delete it anyway.

So there we go. In just a few moments, we were able to aggregate online information, generate a very accurate and clean surface from online data. So we've taken all these different sources of information, we've made this homogeneous surface that works well in a PD environment or Preliminary Design environment, while we're waiting on the surveyor to get us topography.

So we'll just look at the object view of the surface. I do tend to extract the minor and major contours and then do a quick select to remove any of the little bit of noise in there. Some of those points could be two feet. But we know that you could kick the mound of rock and it would be lower.

So like I said, I like to extract some of the major and minor contours. Do a quick select. And anything less than, say, 100 feet I'll delete. Then there you have it. It's just that simple. So very quickly, we were able to generate that surface. Thank you for taking time to watch Leveraging Entwine LiDAR Datasets Within Civil 3D and ReCap Pro.