Speed Up Large Data Sets in Civil 3D: Best Practices and New Techniques

LOUISA HOLLAND: Hi, and welcome to this session, Speeding Up Large Data Sets in Civil 3D. I've compiled some best practices and talking about some new techniques.

Don't forget the safe harbor statement. And in this presentation, I will be discussing future looking functionality. So make sure you're not making any purchasing decision based on those future looking statements.

The contributors to this information today are my colleague Sean Hulbert, who won't be here recording with us today, and myself. We have been working together for many years. And over that time, we've compiled a lot of best practices that we've learned from working on tech support and working with our customers.

So my agenda today is going to be the top five items to check on with regards to performance. I'll talk about your hardware. We'll talk about cleaning up drawings and making sure your project is organized in a logical and clean way. I'll spend a good amount of time talking about working with laser scans or LiDAR and how you can make that perform better when it's in the Civil 3D platform. Last, we'll wrap up with some tools on the horizon. There's where that safe harbor is going to kick in. So let's get started.

The number one thing you can do to speed up Civil 3D is to be in the newest version of Civil 3D. There's been a lot of work in the last few years making that faster, and I'll go into more detail on what exactly is faster and how it's faster. And we'll talk about keeping drawings clean. That is, no unnecessary blocks, no phantom objects, and so forth. I'll talk about data shortcuts and best practices when it comes to breaking your project up into multiple files.

We won't be focusing on it a lot today, but I do recommend that people use the Autodesk Construction Cloud as their data management tool. And that's particularly because CAD files through a VPN can be very slow. And the larger the file, the more complex the project, the more that problem compounds. So we'll talk about that, but just sort of only in an incidental way. I'll spend a lot of time talking about processing point clouds and ReCap, and we'll get into detail on that in a few slides.

So first, let's talk about hardware. And if we're talking about 2025, hardware is probably not the problem. So these are the system requirements dating back-- actually, they haven't changed much in the last few years. Really, the only thing that's been added or changed in this list is support for Windows 11. And there are some of the new graphics cards that are available on the market. So you can get away with a not terribly powerful computer. So even down to 8 gigs of memory, which seems really small to me. Even a tablet would have that today.

So your hardware is probably not the problem, if you do have a high performance graphics card, I'm talking about NVIDIA or something of that nature. If you've got the driver up to date. So those graphics cards have firmware updates that come from the manufacturer, and you'll want to keep those up to date to make sure your Windows and your graphics card are all basically in harmony. If one updates, usually the other will update. And if you've got more than 32 gigs of RAM, you're in great shape. And it seems like most people have that these days.

So I mentioned that Civil 3D is faster in 2025. So how do I know? Well, some data that our product team collected is that the processing of corridors is about 20% faster. And that's just the under the hood type processing. There's a lot of places in corridor modeling where things are just easier. There's fewer clicks to get from point A to point B. So not only is there faster just generation of that information, but it's also just less clicks to get there.

There's improvement in the event viewer performance. That's that panorama window that pops up. And in previous presentations, I've actually recommended disabling that tool. But since it's so much faster, I would say only turn that off now if you're really needing that extra boost.

The section view performance when you're stepping through the stations of your corridor design, it would sometimes lag a little bit when you were going from station to station. That is 60% faster now. And some of those improvements come from the base AutoCAD platform, and others come from some of that generation that's happening on the Civil 3D side.

So generally speaking, file handling is better. Opening, saving, regenerating labels. That has been worked on quite a bit by the product team in the last few years. And the way data references and external references are handled is a lot cleaner now. You won't have AutoCAD sitting and spinning looking for an orphaned reference as much as you would in previous versions.

I'll begin this by talking about project health and drawing health. So first, let's talk about file hygiene. And I like to say, let's keep these files free of gunk. So what is gunk in the CAD sense? That could be duplicate line work. That could be applications that have left x data behind in the drawing, things you can't even see. And so we're going to use some tools to clean those out.

Map clean is a tool that is unique to Map 3D and Civil 3D. If you're in AutoCAD, there is a very similar command called overkill. I like map clean because you have a lot more control over what that interface or you have a lot more control over what gets removed and the tolerances on those. So if you can use map clean, that's what I recommend.

And purge. That's something that can be scripted. So if you want to purge many files, I'll show a script in the next slide on what that would look like to process many files to remove gunk. Another command that is new to Civil 3D is a tool, key in command. This is a command line only tool called clear phantom parcels. So if you have a site and there seems to be some lingering parcel data, maybe you graphically deleted something and it didn't quite behave as expected, clear phantom parcels will flush those out for you.

Let's talk about external references when we talk about graphic external references or xrefs. The simpler, the better. So I like to keep things in, if the project organization allows, keep a relatively flat folder structure. Relative path is the way to go. That controls basically how AutoCAD is looking for the file. If I were to say full path, it would be looking for things like a C drive or a mapped drive letter, where relative path is looking from the drawing kind of going right to left in that path.

The other thing I highly recommend, especially if you are working in Autodesk Docs, is to make sure the reference type is overlay. What that is saying is am I getting references of references when I attach this to my drawing? And keeping it at overlay is recommended, because it's simpler.

Controlling your external references is also helpful in not just being able to navigate your drawing faster, but just for good CAD management practice. I like to have people put their external references on separate layers. That way you can very easily freeze those layers. And then processing and AutoCAD won't take those into account.

I'll talk about this in more detail in a few slides, but using external references instead of data references where you can. And I'll bring up some examples of those in a little bit.

So let's talk about this batch save utility. This comes installed with Civil 3D. If you can't find it, go to your Start button and start typing in batch like I'm doing here. What we're doing here is we are setting the CAD version. So what's going to happen is this will automate the processing of multiple files. I'm going into the LISP routine that will run.

And what you're seeing on screen is the different commands that will clean what I call gunk out of the drawing. We're doing a purge of registered apps several times, saving the drawing, doing an audit, and saving. This is what comes installed with Civil 3D. And you can use this right out of the box. I didn't make any changes to it for this presentation. So let's see what that looks like. Just wanted to show you under the hood what was happening as we process these files.

I'm selecting the folder. And in this case, I'm selecting a folder that is hosted on Autodesk Docs. And that is a completely doable move. It doesn't have to be on a local machine for this batch save to work. I'm telling the program to only go for files that were created in Civil 3D and skip those that weren't. And I've sped up the video a little bit. We'll see that after a little bit of processing time, the batch save utility will come back and we can see what it did.

And it looks like it did skip one of my drawings because it was read only, and that makes sense. And the three others in that folder, it succeeded. So you can do multiple folders at once. You can have a folder set aside just for those processing if you wish. But that is the batch save utility.

OK. I'm going to change gears from the data cleanup to data organization, and that involves data shortcuts. So when it comes to data shortcuts, many different object types have a data shortcut associated with them. Some object types do not. But these are really the big ones we are worried about as far as collaborating with other designers and things that take up a lot of space. So let's get into it.

When do we want to use data shortcuts? And I say on every project. But if you have a file that once all the data is in the file, if it's going to be larger than about 80 mg, you definitely want to start thinking about breaking that up into multiple files, and that will prevent any performance issues due to file size. If you're in older versions of Civil 3D, sometimes that threshold is lower, maybe closer to 70 mg.

Surface models are one of the heaviest object types in Civil 3D, meaning they take up a lot of memory. And so you want to keep those separate where you can. And sometimes you'll even have more than one existing surface. Interchanges would be in their own file. Typically, roundabouts would be in their own file, and production drawings are separated. We use data references and external references in combination to pull the project together and make all the pieces visible.

All right. OK. So when it comes to-- oops, X on that. Forgot I had an animation in there. Let's go to project data organization and what are we talking about when we talk about big data. Usually it's a surface that is large enough that it has generated an external memory file, that MMS file. A parcel project, if there's more than 200 lots, can start to get a little laggy. And corridor models, once you reach about 10 kilometers is where we start to see some performance problems.

But drawing organization makes all the difference, and usually a project starts with surface modeling. And something that I wish people were more comfortable with is breaking up surface models, even if it's one continuous site, but breaking up that surface model into multiple areas. That way, that object, we can pull it together using references, but no one DWG file is hanging on to all that data. So I like to see that.

When it comes to alignments and profiles, that's usually the second phase of the product design. Alignments and profiles from a memory use perspective are quite lightweight. They're not much more memory intensive than, say, a polyline. So you can actually have quite a few alignments and profiles together in a drawing. And sometimes that can be convenient, especially if you're trying to match crowns on a roadway, for example.

So once those designs are in and you're starting to model corridors, we use data references to pull that information in to do our corridor design. And in the plan production stage, this is something I mentioned earlier was using external references rather than data references where applicable. And this is where it's applicable when we're talking about corridors and surfaces. When you cut sections, I think of it as sample lines are powerful enough to pull that data across from drawing to drawing. You don't need a data reference to do it unlike other situations.

So now this is all new content, and I've never presented it before. So this is how to deal with large laser scanned surfaces. And these can be huge. And if you need to pull them into CAD, they can just be a beast. So I like to point out that in 2025 Civil 3D, if you have multiple surfaces in your drawing, each one of those can have its own separate level of detail mode turned on or off. So in previous versions, it was kind of an all or nothing thing. And in 2025, we can pick and choose. So if I need more detail because I'm working on a grading project, that's not an issue.

I will show in detail some of the surface definition options. I've got a clip on that coming up. And one of the things we'll do is prevent extraneous triangles from forming by restricting the length. So especially if I have a laser scan, I definitely don't want to triangulate against things that don't belong touching. So I'll show you that. And of course, we can simplify and crop surfaces.

MMS files, I mentioned before, they get created if a surface becomes large. So 2 million kind of traditional XYZ points or one million grid points. And I would say it's not a bad thing to have an MMS file. Just be aware of it, because if your drawing file moves, the associated MMS file must move. So just be aware that they're there when they do form.

And most importantly is use ReCap to process LiDAR data before using it in Civil 3D. ReCap has gotten really powerful over the last several years. We can classify points now and remove non-ground points. And then we'll take those ground points into Civil 3D. But when we create a surface in Civil 3D, we'll only use a portion of those points. So we don't have a big heavy surface. And we don't need the point cloud anymore. We can detach it from the drawing, so that'll speed things up when we open it later on.

So we'll go into detail about this. Point cloud to surface best practices. In ReCap Pro when you import your files, there is a decimation option as you import. So I just let people know that if you're decimating in ReCap, you may want to change your decimation settings in Civil 3D, and we'll see that firsthand in a couple slides here.

One thing that I've seen customers do is try to combine all the data from LAS files into one. But I say it's actually you'll get a lot better performance if you keep those separate. So if it's practical, keep them separate.

All right. Let's see this first hand. Let's show the ground classification. And for the record, I did speed this video up quite a bit, because ground classification can take some time, depending on the level of detail that you choose. So I chose more detail in this example, and I chopped out a good chunk of time as the classification was happening. But what it's doing is it's trying to separate out as best it can the ground points from things like the trees and light poles and wiring that we're seeing there.

And now I'm doing a Save As. Because what I'm going to do next is remove any points that are not ground. Here it's saying, you've already removed some points. I was fiddling with this file before the presentation, so I just said, go ahead and keep those points.

Now, if I go to the project navigator, I'll see that ReCap has classified those points. And I see it's still a pretty good idea. We got rid of that, the non-ground points. Give it a save, just a good measure. And now as I pan around, I'm seeing some data that I still want removed. It looks like some of those points did not get removed. So I can manually delete them just using the selection tools and the Delete icon. And let's see. Yeah, that's what we just saw.

So the next step is to pull that information into Civil 3D and make sure when we do, we are going to decimate on the way in, meaning I'm not going to use all those millions and millions of points from the file. Just browsing out to the location on Docs where have the stored. And I'm choosing the ground only file that I created in the last video. And see, it's much smaller than the full file.

Now, the next step is to create a Civil 3D surface model from this. And like most dialog boxes, we can give it a name. Here's where I tell it the distance between the points. And I'm telling it half a foot. And that takes me down to about 12% of the original number of points that were in that point file. And I sped this up quite a bit. But after the software processes, you'll see the tune appear in the drawing. And you'll see that we still have some unwanted triangulation and that's what we'll take care of in the next video.

So let's take a look at some of the surface properties that will help control some of those unexpected elevations and some of the extra long triangles that are forming. So as you watch this video, keep an eye on this kind of interior corner of this flown data from an interchange. I'm using the surface properties to constrict the length of triangles. So I won't have any unwanted triangulation.

First, I'm setting the elevation exclusion, triangle angle exclusion, and the triangle length. And that's really all there is to it. I also turned off show tooltips. That can speed things up a little bit too.

Now I'm going to show just that level of detail. What's the difference? So if you want all those contour lines every time it changes elevation, it's going to show that really dense level of detail. We'll turn that off so it's a little less taxing on the CAD. And actually I'm going to go back just a smidge and just pause there to show that CAD data or that surface data is much tighter in to the laser scan data.

OK. I won't spend too much time on corridors, but I do have some tips and tricks that we've learned over the years. And yeah, let's talk corridors. When it comes to naming your assemblies and baselines, I like to encourage people to, even though it's not convenient when you're in a hurry and you're really cooking on a design, it's not always convenient to stop and pause and give your objects descriptive names.

But I say that in the long term, it'll be worth it, because it'll save you so much time if you get confused. So many of these objects look alike, assemblies especially. Is this the 6 inch curb or the 8 inch curb? So naming those things nicely and clearly is going to really pay off in the long run.

When you are modeling corridors, especially when it's a big, long one, you don't necessarily need all the regions active as you're working. So you can go into the corridor properties and uncheck various options or various regions if you don't need to work with them.

Sort of along the same vein is the frequency distance. And when I say frequency distance, what I mean is basically the distance between those lines that run perpendicular to the direction of your road design. The closer together those are means that Civil 3D is computing information at those stations. So the tighter together they are, the more Civil 3D is doing math. So if you can make those a little, I'd say, chunkier on the long stretches, you can maybe turn that up right when you're about to maybe do quantities. But when you're creating the design and you need to change quickly, changing that frequency distance might really help you out.

Turn off rebuild automatic. That's something we've said since Civil 3D came out in 2005. But rebuild automatic can bog you down, because just any slight change will have the CAD program stop and you got to wait for it to generate.

Object viewer is in the program to help you examine items in 3D. I always recommend that people save before they jump into object viewer always. And actually, I'm going to bring this up again, at the very end of my presentation, but object viewer is a great way to select your corridor, look at it in 3D, and see if you have any design problems.

If you have what I call a waterfall or one of those where the road appears to go down to 0 elevation, those are really easy to diagnose and find in object viewer. You might not be able to see them in the corridor properties. So that might not necessarily speed you up, but it is a way to just-- it's a tip to help your corridor modeling.

And I mentioned before that it is possible to turn off the event viewer or you might see it come up as panorama when your data appears. But you can turn that off. And if you're in Civil 3D 2025, I would say that's not necessary. But I think we said that was 30% faster in the newer version of Civil 3D.

And as I mentioned before, if you're creating cross-section sheets, you want to use the xref rather than data reference. That way there's just less information coming across. It's actually pretty slick way to do it.

OK, so now I'm going to close this presentation with some really cool tools that are on the horizon. And I always hate saying cool in presentations, because it's cool to me, but it's work for you. So coming soon to Civil 3D 2026. And I just want to remind you that this is one of those forward looking statements that is covered by that safe harbor statement from the very beginning. But these are things that I have seen demos of from our product team, and it's really exciting.

So the number one thing I'm excited about, especially when it comes to performance of large data sets, is the ability to create a data reference, but only to part of a surface. So instead of breaking up that original surface into multiple files, as I recommended today, you could actually still have one big file with your surface. But once you've closed out of it, you don't need to pull in the whole surface to do work. And that's just the beginning. There may be other objects that have partial data references, but right now it's surfaces that the team is working on.

Another tool that I'm really excited about that replaces object viewer that I just talked about is a new tool going to be called the Civil Viewer. The name might change when it comes out, but right now is called the Civil Viewer. And this is a way to examine items in 3D without having to go into a mode where you can't click on other things. So that's one of the downsides of the viewer today, is once you're in it, you're kind of locked in it. It's not, as we call it, modeless. So you can't click on other stuff.

The Civil Viewer, what I've seen, it is going to be modeless. So you could even maybe have it up on another monitor. You'll be able to see many different object types, things like pipes and corridors and surface models. So that will be a really powerful tool once that comes online. I'm really excited to see how people use that and how frequently they are going into that box to spin their designs around in 3D and diagnose problems.

There is a new tool that's coming out, and this is I'm going to give a shout out to Sean and his team on this, and that is a tool coming out called the batch cleanup tool. So earlier in this presentation, I showed the batch save utility using scripts to clean up drawings. But Autodesk is working on a dedicated tool that will clean up drawings. No scripting knowledge or even dabbling in scripting involved.

And the next thing, last but not least, is something I'm excited about. And this will affect people who are working with Autodesk Docs. And that is some new functionality coming out in desktop Connector. Desktop Connector today only uploads one file in a queue, so it's not a dual upload. And the next version of desktop connector that comes out is expected to have dual upload. So we expect to see double the upload speed or pretty close to it.

So that concludes my presentation. Thank you very much. I appreciate you coming here if you made it all the way to the end of this. Have a great day.