Underground Utility GIS Features to Civil 3D Pipe Networks with Dynamo

JAE KWON: Hi, everyone. Welcome to our instructional demo, Underground Utility GIS Features to Civil 3D Pipe Networks with Dynamo. My name is Jae Kwon. I'm joined by my colleague Maxime Carrier. We are technical consultants at SolidCAD and we're based in Canada. Now let's take a look at the problem that we're trying to solve. So Civil 3D designers typically create utilities as pipe networks. They consist of pipes and structures, often with custom part catalogs. They may have additional data stored as object data or property sets. Now, on the other hand, GIS technicians manage asset data as map features. They consist of line and point geometry with attached data tables. Often the process to translate from one to the other is a manual labor intensive process. A lot of data entry is involved and it is prone to human error. Overall, the entire process can be very time consuming.

All right, so why Dynamo? Three types of solutions are available. So one is kind of pre-made. So that can include export to SDF tool, ArcGIS connector. Now, these solutions are easy to use but they're fairly rigid. You can't customize them to your custom data needs or your custom workflows. Now on the other hand, we have custom coding with LISP, VBA, .NET. This is the most customizable solution, but it does require quite a bit of a time commitment to get started and to maintain, especially if you're not already proficient in coding. Now, in the middle we have visual scripting solutions. So the one where we're using today is Dynamo. So it achieves like a balance of flexibility and ease of entry. It's a custom solution that's within immediate reach for most people.

So packages used. At the core we have the Civil 3D toolkit by Paolo and Safi. This toolkit basically made this project possible. It contains all the nodes that we're going to use for pipe networks, map features, and object data tables. We also used additional packages for support. So for UI we use data shapes package by Mostafa. Now this package provides UI that lets us get the user input without having to go through the Dynamo player. This, in turn, makes the graphs macro friendly and this allows us to separate the graphs into smaller chunks and combine them into different combinations as we need them. And for utility nodes, we also use a clockwork package my Andydandy, so nodes control flow of data and to measure runtime of the known groups. That helped us sort of optimize our graphs a little bit.

Now so presentation agenda. Part one, I'll be covering the GIS to Civil 3D data flow. I'll be creating the pipe network, structures and pipes from map features. I will also be transferring some asset data from the map features to the pipe network objects in the form of object data records. Now part two, Maxine will take it the other way. So take the data flow from Civil 3D to GIS. You'll be creating some AutoCAD objects with object data from the pipe network structures and pipes. Some data transfer from a property set to object data will be involved. And we'll be packaging it all up into a GIS ready format.

So there is some downloadable content for the course. All graphs will be made available. So you don't have to take notes on what nodes are used and how they're put together. You'll be able to see all of that. There will be additional details covered in the handout. So our presentation will sort of remain a high level overview of the implementation. And we would very much welcome any comments and tips for improvement on the graphs. So GIS to Civil 3D. Let's talk about the setup of requirements first. So we're going to need to first thing we need to do is connect to the GIS data source and add them in as feature layers through the map 3D interface. Now, the data source can be shape files, as SDF files or web feature services. Can be anything, really, whatever format you want to keep a story or GIS data in.

Now, we'll use that data in the map features to create the correct pipe network parts. Now, to do that, we'll have to have the available parts ready to go which means we'll need to create a parts list and load up all the part sizes and families that you will need. Also it will probably involve creating and setting a utility catalog for all the junctions and conduits that you'll be using. Because they'll probably be custom parts that's not standard to storm and sanitary. Now, we also need to talk a little bit about the schema, also called property names or object data field names. Basically what property names do you want to use to store and transfer data? In our case, it's a fairly new implementation of new GIS database. So only a handful of existing data needs to be incorporated. This means we're fairly free to use new property names in our setup.

So we kept things simple and just simply used a property names that Dynamo nodes use. Now, we still have to deal with the existing data files. So you can kind of tell that the existing data has property names that are not very descriptive. We'll be translating those property names into the property names that we're actually going to use. So once you process all the existing data, going forward you don't really need that translation step anymore. And we can just go forward with the cycle of data transformation between GIS and Civil 3D.

Now let's take a look at the graph details a little bit. And we'll start with the drawing that has the source data. OK, so I'm in Civil 3D. I have a couple of feature layers loaded in. One for structures and one for pipes. Now, each one of these objects have a bunch of data stored. That's the data that we're going to use to create the pipe network parts. Pipe network structures and pipes. Let's take a quick look at that data in table form. So most of that data is data that's been pulled from the pipes and structures themselves coming the other way, which Maxime will show later. Now, at the end, we have some data that are custom. This is sort of asset related data like construction, date, inspection, level of service, condition, that kind of thing.

So we're going to be bringing in that data as well and we'll need to sort of bring them in as object data. Now I'll run the first street script. So just give me one moment. OK, so this is a Dynamo graph. And the first part of that will be to get the inputs from the user. And the inputs that we're going to need include sort of like the map layers that we want to convert for structures. The map layer that we want to convert for pipes. And also the target pipe network where the converted parts are going into. So once we get the input, we're going to get all the features from those layers. So we're going to pull all the individual features from that.

Now it has to go through some kind of a text filter or location filter. In this case, we just use a text filter with no default filter value. That just brings in everything from that layer. OK, so once you pull the objects from those layers, we're going to get the geometry and the data from those individual objects. So that's the next part. So some of the properties that we're going to pull from that include rotation, rim elevation for structures, for pipes, start and end inverts, that kind of thing. So those are properties that are going to help us position those parts correctly in the drawing space. Now one of the most important properties to pull out of the features is the structure part size name. That is basically something that's exported by the graphs going from Civil 3D to GIS.

Now, we're going to use that to identify the correct part to use from the part list. Now, the first kind of run through you might have to-- if you don't have the part side's name already exported out-- you might have to sort of pair them up using sizes and materials, that kind of thing. But in our demo it's basically already done. So once you get the part size name, you'll have to pull the family name. It's not easy to get the family name out of the pipes and structures using Dynamo currently. So we're just going to use the part size name and just look it up in a search.

So basically you're going to pull all the family names and all the part size names for each family name. And then just see what family name that those part size names are part of. Once we get the family name and the part size name, we're going to use those names to get the correct part from the parts list. And then create the structures and the pipes. And then the process, we'll add in some key properties like rim elevations, rotation, invert elevations, that kind of thing. And finally one last step we have to do is take those structures and pipes and connect them together. So to do that we're just going to compare the coordinates of the structures and the pipe ends. And then just make connections where it's appropriate.

So I'll go ahead and run that. So the inputs again, are just the map layer for structures, map layer for pipes, target pipe network. And I'll just hit create. And then once that finishes running, we'll take a look at the pipes and the structures and see that they properly came in. So that completed and now we're in the drawing. So it looks like the structures came in correctly. The pipes came in correctly. They are properly connected together. And we just need to check the elevations at this point. So I'll turn it into isometric view. And looks like all the invert elevations and the rim elevations have been properly set.

So the next step in the process is to open up the Dynamo script to basically transfer the data over into object data. So I'll just go ahead and close that and open up the next script. So this is custom data that we're transferring over. A few inputs for that. So the source, map layer for structures, the source map layer for the pipes. The target pipe network that the data is being transferred to. And the JSON file. Now, the JSON file contains the definition of the properties. That could have been stored as an Excel file, CSV, file or text file, but we used JSON in this case.

So the first thing to do is just pull all the property definition data out of that JSON file. Put it together as an OD table, object data table, because that is almost GIS friendly format. Then we're going to take all the pipes and structures and add the record to the object data table for each one of those objects. And finally we're going to pull all the values for those properties from the features and transfer them over to the object data records. So I'll go ahead and run that. Both are the input. So there's a map layer input for structures, map player input for pipes, target pipe network, and the JSON file that we're going to use.

So go ahead and select the JSON file. And run the transfer. So once that finishes running, we'll go back to the drawing and take a look at the drawing objects. OK, so there we go. So I'll just go ahead and check one of the structures. And it looks like all the OD table data came in. So that object has been attached to the custom asset properties object data table. And all the values have been transferred over. And same with the pipes. OK so that basically covers the both graphs of sort of GIS to Civil 3D data flow.

Next we're going to take a look at the data flow from Civil 3D to GIS. And Maxime is going to present that portion.

MAXIME CARRIER: Thanks Jae. Now we need to transfer our final design, our as built Civil 3D plan back into our database. The designer might have added or removed some piece of our network and obviously might have updated the metadata on the remaining parts. So to apply these alterations back to GIS, GIS user, or most of the time a database administrator would run these automated actions. He would read our proposed pipe network and any metadata attached to it. He would keep intact in the database all untouched existing components. He would update all components that were identified as "modified." He would remove all components that were marked as to be removed. And, of course, he would add all components that were new to our database, that were new to our pipe network.

So can Dynamo handle all of these actions on a connected GIS database? Unfortunately no. That's because Dynamo, while it can totally take GIS information as an input, has a limited ability to write back some information to the database. It has nodes that read and modify existing elements through the FDO the connector, but it cannot create new elements and erase existing ones. And for that reason, the user will have to develop his own workflow, his own workaround procedure to update his GIS database. Will this be problematic? We don't think so. And at this point, it could be even for the best since most GIS administrators will work within their own GIS software with some automated routines that will validate any plan modification and new data entry prior to applying any direct edits to their GIS database. And remember, especially with this workflow, the GIS database will act as an archive for your utilities for years to come.

So it needs to be protected standardized and properly maintained. In that regard, we can still do some legwork within Dynamo. Most GIS software will read DWG files directly, but it won't detect nor extract any information from Civil 3D objects specifically. But when they do, to our experience, they won't grab any, well, every info needed to fully and properly recreate your pipe network afterward. And it won't reach every possible place where you can put your utility metadata. So for the second part of our demo, we will show you the most foolproof way we've developed with Dynamo to grab everything that you need. Not only to recreate your full 3D pipe network later, but to grab every single metadata you added to your pipe and structures. And finally, digest all of this information into AutoCAD objects that can be easily read back in your current GIS database.

On to the demo. To ensure that we can gather information from every possible angle, we split our workflow into these four steps. In this part of the demo, for time's sake, I'll be using these customized buttons in my tool palette to launch my scripts instead of working directly with the Dynamo UI. If you ever want to create your own buttons, we'll have a simple procedure in describing in our handouts. And these scripts will be, of course, be explained, as Jae mentioned, will be explained in much more detail in it too. So if we look at our current pipe network, might have been modified, might have moved things around, deleted, added things around. And, well, how our user decided to put his metadata under the extended tab, the extended data tab. Which is not truly readable by most GIS software.

So we'll have to do some data juggling with it. First, we need to create our AutoCAD objects from our pipe network. I'll launch the first script. The first script reads all pipe networks in the current drawing and will create AutoCAD objects on the specified target layers. You can even mention what kind of block you want to use that's available in your drawing right now. But I'll keep it as default, close the arrow portion. After running a Dynamo script that creates either AutoCAD objects and Civil 3D objects, you might want to flip between your layout tab and your model tab to make them appear.

A simple region command won't do the job. You actually have to manipulate your tabs to make them appear. They're still there. So if you ever run-- if you don't see them, if you ever run the outer scripts, they will work fine. But if you really want to have them present in your screen, you'll have to flip back and forth between layout tabs. Let's look at our AutoCAD objects. Our AutoCAD objects right now have every [INAUDIBLE] required to eventually rebuild our pipe network. I'll start within map 3D object data table. That can be easily read back to GIS or exported through the simple map export command.

It basically grabs everything from your part. And it should work with any custom part, any custom parts catalog you have. In fact, like this one, which was especially built for underground electric networks. By our team, by the way. The next two scripts in my list will work within the drawing pipe networks to gather all information. Any custom utility meter data, any extra stuff that you added and you've edited that's not by default in the pipe or structure properties. And we will adjust them into object data tables. First, we will read from the external data tab to create some new object data table structures.

So I will launch the script. To build a structure we read from a specific data property set. And then we aim that on the current or a specific or target pipe network. So here I am with an empty but properly structured object data table Then we launch the second script, which will transfer everything, every content that you add in your extended data priority set to your object data table. So those tables should be filled up. And then, last but not least, we finally transfer our object data table from our pipe network to our AutoCAD objects.

So there's many ways that you can kind of map your stuff or transfer stuff between objects. We decided to use the geometric option. So if a pipe network structure shares the same insertion point as a specific block, it will transfer everything from the pipe structures to the actual block. And we did something pretty similar with a pipes, so it's a simpler way to do that. But you can map your objects in all sorts of ways. So I'll read an object data table from a specific pipe network and then I'll send them to pipes and structures that are located on target layers. Let's do the transfer. And it's all done.

Our object data table with custom properties should be in it. So we have our default stuff and our custom metadata asset stuff. From that point, a GIS user or administrator could easily use one of these many custom properties to automatically assign actions to our database. Like using the original component layer name to split your design and change parts, modify parts, edit parts, and parts to be removed. And while this whole procedure was demoed with separate scripts in individual buttons, you can easily string all of your scripts back to back into a single button. Since it's all come in like coding at the end of the day.

Again, please read our handout to properly recreate your own command line. And now if you ever want to export your stuff, everything that's in there can be exported properly through a standard map export command. I'll just overwrite this SDF file that we used as a reference and load a proper export profile to grab stuff from the exportable layers, preset the feature class. And then we click and it's all exported properly into a GIS ready file. Could be either directly from this DWG file, depending on which software you use, or use the exported stuff that's all structured properly.

And with this demonstration, we conclude our workflow presentation. Our main idea was to show you the possible connectivity between GIS and Civil 3D through Dynamo scripts. In this case, choosing complex Civil 3D objects like pipe networks, though you could easily adapt this workflow without any problem. Well, I wouldn't say easier, it would be easy but it would be without any problems too much simpler objects like other simpler objects like AutoCAD objects or much simpler Civil 3D objects. With that being said, we really wanted to open the conversation with our community to multiply those kind of applications. Especially exploiting this brand new connectivity between Map 3D and Civil 3D thanks to Dynamo for Civil 3D.

What else could we accomplish with Dynamo for Civil 3D? Well, we are only limited by our imagination. We could go back and forth with the GIS database to work with stored survey points, stored survey lines, pressure pipe networks-- like if you want to manage your gas networks, your aqueducts, whatever you'd want, existing road feature lines, existing land lot parcels, existing road center lines brought in as alignments and profile lines. Well, we could even create custom quality takeoffs built in your script as an extra output. Name it, we can probably do it. So we thank you all for attending our presentation and have a good one.