Water Project Lifecycle Connected Data with Civil 3D, GIS, Dynamo, Python, and IFC

JUSTIN RACELIS: Welcome, everyone to CES600175. This is the water project lifecycle, connected data with civil 3D, GIS, Dynamo, Python, and IFC. So I'm Justin Racelis. I'm an infrastructure digital practice manager with Stantec. I help drive a lot of the initiatives related to design automation, and computational design, and emerging technologies within the infrastructure business at Stantec. And hey, Mike, do you want to introduce yourself real quick?

MICHAEL DESILETS: Thanks, Justin. Yeah, my name is Mike Desilets. I've been working with Stantec for the better part of 15 years. And I'm a geospatial business strategist. Basically, that's dealing with information systems and all types of geospatial technologies. Happy to be here today.

JUSTIN RACELIS: So safe harbor statement. I believe I'm supposed to say do not make any purchasing decisions based on any content that we provide here, et cetera. So customary at Stantec is to give something that we call a safety moment at the beginning of every presentation.

So I just want to tell everyone to be safe when booking rideshare or participating in any kind of rideshare service. Just because you may be coming from some event and think that you're safe because you're in a vehicle. And that may not be the case. You may need to intervene and direct or implore the driver to do a certain thing. Otherwise, you may end up in a situation such as getting involved in a car accident while in rideshare. So here at-- here at Vegas, people are going to be moving around all the time. So just stay safe out there.

All right, so to give a high level overview of what we're going to be discussing here with this presentation, we want to show how you can utilize transmitting data between various different platforms to create a connected data system within a software ecosystem that is not inherently connected. But we can do a lot of things to push and pull data and make some programmatic and intelligent decisions with our model and attribution based on any number of mechanisms.

So firstly, we'll-- just with design data development, with this example project that we'll showcase, we will be utilizing a model-based-- or information model that is was used for the design and documentation. And Mike, do you want to explain a little bit about the geodatabase?

MICHAEL DESILETS: Yeah, thanks, Justin. So we were going to go through the evolution of some of the change management scenarios that people often find in dealing with this life cycle situation. As we have moved through this a couple of times on some projects, the structures of relational databases definitely have their place. But we've also found that there needs to be more fluid ways of moving information through the life cycle in addition to the actual design process, from the beginning all the way to the end in delivery. So we were looking at an enhanced strategy of using a relational or geodatabase technology.

JUSTIN RACELIS: Yeah, thanks, Mike. So with that, and the design-- the information model that we're utilizing for the design, we're able to connect field data with the design model. And we're going to show how this can be done utilizing any number of techniques. But most importantly, this kind of streamlines a lot of the post-delivery to operations and maintenance requirements that we tend to see on a lot of projects these days.

And in order to protect the information of some of our clients who may have some security concerns, we're going to be showing a workflow that's derived from a number of things that we're doing across multiple projects, but in this case, applied to a sample project that's long been constructed in the past and designed by Stantec, just to give an idea of how one could use these concepts to do-- to develop a connected data environment with a similar type of project, or maybe even a completely different type of project.

So yeah, just to give an idea of a little bit of the technology stack that we're working with here. So we have some of the Autodesk products, the Civil 3D, Navisworks, Dynamo, as you can expect from a design environment. And then we have, for data management and attribution, we're using the ESRI suite and Excel tables when needed. And then for collaboration, we're utilizing a lot of computational methods through Python and packaging information with GeoJSON and IFC.

So this is that sample project that I had mentioned. It's just a small segment of a very large sewer network. And this is right in front of the World Trade Center. So this is something that, again, that Stantec had worked on quite some time ago. So you can see, it's a very large chamber connected to some pipes and some manholes.

And right here on the right side, you can see a lot of attribution has been assigned to these models. So what we're going to be showcasing here is what you can do with, in this case, Civil 3D property sets. But if you were working in Revit, you could-- this would be extensible storage in Revit.

Any way that you can use custom user defined attributes to match the client requirements for their asset schema, and yeah, just up here on the left side, that's the original plan sets and alongside an image taken right from the model. So that we can see a little bit of the context here.

So what we're doing with Dynamo essentially is we're taking all of this information. And we need to somehow package it in a way that it can be read as a GeoJSON by platforms that require the GeoJSON schema to be followed to a certain extent. So the first thing that we're doing is we're getting grabbing the pipe objects. And then some user inputs are needed, such as the target, the location of the GeoJSON files, and the list of the property names that were developed that we're using for the pipe network.

And then with Python, we're going to be doing a bunch of fun stuff with it. And yeah, so as I mentioned again, GeoJSON kind of-- it performs a very central role in what we're doing because effectively, as we're going to show over the next few minutes, we're going to-- we've developed a Python interpreter for Civil 3D pipe networks and GeoJSON, which that same concept could be applied to quite a number of things.

But in this case, we're working with pipe networks. And GeoJSON is used extensively within the industry. And Mike, do you want to give maybe a little bit of background on how this gets used on the geospatial side?

MICHAEL DESILETS: Absolutely, Justin. Yes, one of the good things that we had worked a lot on was looking at different types of proprietary formats. And there's been a lot of work between Autodesk and ESRI to deal with the integration there. So with doing that, looking at the kind of open data structure that JSON format had put out there, some of the more complex topology types that people depend on looked to be finally evolved within the GeoJSON format.

It was great because it's nice and lightweight. And kind of really works with a lot of the different platforms that we're dealing with. And instead of looking at how to fit and squeeze data from one design environment to the next, we ended up finding that this GeoJSON format, not only did it carry the rich attribution in a simple structure. The geometry was also able to be packaged. So very, very elegant solution. Yeah, thanks, Justin.

JUSTIN RACELIS: So yeah, as I mentioned before, we effectively have built a translator with Python. And it's very flexible. I mean, this is really just-- logically speaking, it's any number of ways that you can think of to take civil object data and instantiate it within the GeoJSON schema. So I'll just step through some of the functions that we had to build in order to get this workflow up and running.

So some of the main things that we needed to accomplish with this is the first thing is just identify the pipe network object. So look at what's in the model and identify the models that actually need to be transmitted via GeoJSON and ignore the rest. So that's really just parsing through lots of information.

The next would be to create the GeoJSON itself and then populate it with the geometric and property data. So yeah, we did that through a number of functions that we've defined in Python. So the first is-- it just needs to look through the object properties and values and collect them as dictionary keys and values.

And this portion right here was a little bit interesting because while we are using property sets to represent the asset attributes, there are a number of object properties that are not property sets that are used-- that need to be transmitted as well. So we kind of needed to look through both of those and figure out which ones are necessary and need to be transmitted.

And next, we just need to-- following that GeoJSON schema, we needed to build the GeoJSON header and then create the feature object itself and then go through all of the Civil 3D objects that have been identified and then populate-- format them and populate them as objects within that GeoJSON feature database.

And the result is just between these two right here, you're selecting the model in Civil 3D. That's the original property-- set of properties that I've highlighted here on the left side. And that's significant because we're going to keep-- we're going to keep revisiting these particular properties throughout the rest of the presentation. And on the right side, this is just the GeoJSON opened up in text editor.

So you can see the correlation between the two that the property sets that we needed have been exported and are mirrored inside of that GeoJSON format. So the end goal of doing this is collecting all this information and using it in GeoJSON. And we want to go to the field and do something with it. So that's our goal is we want to take the model and we want to use it somehow using the ESRI field apps and utilize some of the geospatial technologies that we have available to us. Yeah, and go ahead, Mike, you want to go ahead and take over and talk about the geodatabase management?

MICHAEL DESILETS: Yeah, that sounds great. So as I mentioned before, we understand the power of the relational database technology. ESRI relies on geodatabases for its rigid geometric management and attribute management. One of the things we had to do is respect a little bit of that, but trying to move the information from the design environments up into the field collection environments, as well as look at things within the GIS.

We decided to move things through to REST services. And we were able to easily do that with the GeoJSON import functions from ESRI. If you want to move to the next slide, Justin. We found out that we are really at a time where the REST services. And the combination of the GeoJSON structure has really enabled the accessibility to such a wide variety of technologies that by moving data into the REST service from the GeoJSON, now we're able to put in the hands of a wider audience or a wider group of users on a project, handheld devices, desktop applications, web applications, really a lot of independence and freedom there.

And this is just part of the evolution of client services and database technology into web-centric patterns as the graphics are showing. What that really means is we're not using proprietary formats anymore, one size fits all type of applications or platforms. And that's just increasing our collaboration. This gives us efficiencies in the data movement. And again, that's what you've been detailing, Justin, is how we move the geometry and the attribution in the simplest, most lightweight format. If you can move to the next slide, I'll just briefly touch on the quick functions available in the ESRI product.

The idea, if you haven't used REST services before, they've been around almost longer than and WFS standards. And they are a very powerful way to make sure rich geometry is usable in a variety of formats with ESRI's cloud services product, the ArcGIS online environment, which runs some of these lightweight mobile applications relies on REST services. So moving it from import from Civil 3D up to REST services is a very easy process with the GeoJSON format.

Yeah, so then what we do is we deal with web app-- web maps and web applications that can allow a user to go out in the field, edit information based on observed measurements. And one of the things to be very aware of is that if you're doing spatial proximity measurements, to make sure that you're using the right type of instrumentation and the right type of survey professionals. Thanks, Justin.

JUSTIN RACELIS: Oh yeah, sorry, so in this case, what we're showing doing here with the survey team is this is emulating a workflow in the field where, for example, a surveyor would come out. And during the as-built collection, data collection phase of a project, the surveyor would shoot the point here. For instance, if we needed to find a point that represented the insertion of a structure, and that varies differently from what was designed, the surveyor would traditionally would have provided that as red lines on a sheet and then had somebody in the office just draft up the changes in the plan.

But in this case, what we're doing is we're utilizing the ability to capture this information in the field. So rather than that information being typed-- just handwritten into some type of notepad or a piece of paper, here, we can just input it directly into the ESRI field maps. So yeah.

MICHAEL DESILETS: Maybe I'll just add on that as well is that these devices that typically had been used by more advanced survey grade exercises, we can now put some of the data collection that's more relevant to a wider group of project team members into smaller, more affordable devices, and still be able to capture a lot of attribution that might not be part of scope for survey-based operations. Thanks, Justin.

JUSTIN RACELIS: Yeah, sure. Yeah, so here are just some-- showing some of the points that get modified via the ESRI App. So on the left side, that's like the insertion point, the x and y. And then updated on the right side, this is, again, kind of emulating that surveyor as-built workflow of collecting the insertion points.

And then here, these are some attributes that I think are, again, we highlighted earlier when we selected the structure-- or the chamber element in the model. And in this case, these are some standard design criteria that are typical of New York City sewer chamber design. So anyone who's familiar with the area would kind of recognize this.

So the idea is that if-- the as-built of the chamber may have some slight variance from the original design. So if you want to capture what those new parameters are for that determine the shape of-- or the geometry of the chamber that can be done as well and collected as effectively as metadata through the ESRI field maps.

And yeah here, so moving onto model is as-built. So essentially, bringing that information back from the field, so as we showed talked about before, we took all of that information and in the field and updated the web map with it. And in this form, it's still-- all we're doing is we're just updating a simple point and polyline-- or point and line geometry in the map.

But because we're utilizing the attribution to transmit the design change-- or not the design-- the as-built changes to inform the designer, we're no longer dealing with heavy models and lossy conversions. It's just what changed. And then we're looking through what changed and using the Civil 3D functionality to update that.

So this is the JSON that we get back. And one of the interesting things here is that what we encountered is a limitation with web-- with the web services in ESRI in terms of the attribute field name character length. So even though a character limit of 255 is actually supported, in this case, it's only 31. So we did have to adjust for that a little bit in our workflow.

So what we ended up doing with that was basically just building that into the script. So it would understand that some of these attribute fields got truncated and correct that. So now, we're basically just doing the reverse of the workflow that we did initially in the beginning. We took the pipe objects. We needed to identify the ones that are being used in this particular scenario, and then grabbing the updated GeoJSONs, and then using Python to look through them and understand what has actually changed.

So here, yeah, going through some of the functions that we-- as part of our Python workflow for this side of the process is just loading the-- loading the GeoJSONs and grabbing those feature dictionaries, which is the way that they're kind of contained in GeoJSON. So then, it looks at every single structure. And this is something that's going to be specific to Civil 3D pipe network elements.

And so even though, technically, when you move a-- when you change the insertion point of a structure, it's going to move the pipe as well. So in the field, moving the-- inputting the data or inputting the updated coordinates for the structure wouldn't require us to also add the changes to the pipe because we're going to handle that on the Civil 3D side.

So it's going to look at all the structures and determine if the insertion point is different. It's going to just grab-- grab that structure and then move it based on the updated coordinates that are listed in the attributes in the GeoJSON. And then it's going to do the same thing but look at the properties.

So it finds all the properties that have changed. And then if they have changed, we assume in this case because we're going back to the model from GeoJSON that those are newer. And those are the ones that need to be revised. So then we'll go through all the objects once we identify those properties that need to be changed and then apply those functions to all of those Civil 3D objects.

Yeah, so this is just a quick video just showing, running the actual script, so finding those changes and importing them, and then that those properties that were highlighted, again, those were, again, what we're using in our example here. So that's-- you can see that those have been modified. So not only did the structure shift, but the properties that captured in the field also came through.

So this is kind of indicative of what we can do with this type of workflow. Where it's just transmitting parameters through GeoJSON and then utilizing the capabilities of the native modeling or information modeling platform to make those changes for you. So that, again, you don't deal with lossy model file format conversions.

And lastly, what do we do with that information now? So we have this updated model in Civil 3D. And one of-- at this point in the project, what you're doing is you need to deliver either as-built or archival level of model information for the client. So the client may have requirements that they need IFC 4.3. Or they need a geodatabase, which a lot of our clients do need that.

And the useful thing about this workflow is that the geodatabase is inherently part of it. So just by instantiating this process in the first place, we already have created effectively that geodatabase deliverable. And if we need to export to IFC, that can be done from Civil 3D directly once the as-built conditions have been captured.

And also if you, unfortunately, were still developing sheets like many of us are and you needed to update those sheets, updating the model if your views of the sewer were cut with sheets and Civil 3D with the sheet-- the sheet tools, then those would update as well once the model has been updated.

So yeah, moving on to just finally, what do we do with the post-delivery side, just sending those-- transmitting those IFCs or the geodatabases to the clients. And just to recap, what we did is we went over how we're taking design content from information models. We are capturing that somehow in a format that is ideal for geospatial applications. And then we are taking those changes in the field via the ESRI mobile apps. And then we're programmatically taking those changes and using Civil 3D to modify the original model to reflect those as-built conditions. And then finally, taking those-- that updated as-built model and delivering that in the formats that are needed for by our clients.

So just side by side right here, this is showing what the original and updated 3D model in the ESRI environment on the left side. So just showing that there are some capabilities on the ESRI-- on the ESRI platform to view 3D models as well. And then on the right side, this is just the IFC export. So this is in the Open IFC Viewer. And again, looking at those properties that-- all of that-- all those property sets come through. So in this case, we're-- this is asset attributes.

So if you have a client that have dictated a certain set of asset templates that you need to follow, you can use property sets to-- as the foundation for that in your workflow and then use a workflow like this to exchange information with different phases of the project. So construction, as-built, et cetera.

And as far as the business case for this, I mean, again, there are a lot of clients that do have IFC deliverables where they want different schemas. That they want to follow different schemas and have them delivered as IFC models. There are geospatial deliverables that this would align with.

So especially with some of the ways that the ASCE 38 and by year spec would kind of require some of the properties to be instantiated in the field for utility mapping and things like that, I mean, all of those are directly applicable using this workflow. And then of course it streamlines your whole as-built process. So no more red lines-- drawing red lines on a plan.

I mean, if this was a different type of infrastructure project that utilized a different-- was using different types of models, I mean, there are-- there are ways that this workflow can be repurposed for that as well. And I think I'll touch on that a little bit at the end. And yeah, Mike, do you want to give a little blurb about mixed reality?

MICHAEL DESILETS: Yeah, thanks a lot, Justin. So what we wanted to really do is really push the envelope with some of the asset management deliverables that we had as well, which Justin just alluded to, there are standard requirements for some of those. They don't necessarily require the same level of geometric precision or even the structure of it, but the qualification of the assets themselves.

But with the process that we've been detailing, what we are also then finding with something, especially like an IFC, native to a lot of applications that can work with virtual reality, augmented reality technology very well. So what we can do is also introduce the idea of asset management or public engagement opportunities at any stage of the project with the designs. And that in this particular case, this picture on the side, we're just kind of showing that you can utilize some pretty potent BIM or IFC deliverables to allow for reality comparisons.

So really, got a quiz here for everybody. Just throw it out there, and say, based on the picture that you got here, can you tell us which one is the real one and which one is the augmented reality building? And we can definitely work through and go through which one is which, but just trying to highlight that the realism behind these models now is getting so great, the technology is so great that, really, this should help a wide variety of model verification as well as any number of means from marketing to public engagement. Yeah, that was exciting for us as well. Thanks, Justin.

JUSTIN RACELIS: Yeah, thanks, Mike. And lastly, again, I did mention that I wanted to show how you could use this similar type of workflow and apply it to a different-- to a different type of infrastructure or a different scenario. So as a quick example, this on the right side, this is a very simple parametric subassembly that is designed to identify the different retaining wall cases in this theoretical location where a retaining wall is needed.

It will design the foundation for you, essentially. So what would we want to do with this? Well, a lot of times, we have some transit and transportation owners that are developing GIS and assets requirements for even linear features like these, which are a little bit-- something that's a little bit difficult to capture in just a simple point and line geometry.

But the attribution, if that's really where we want to get smart about how we convey that via a polygon, what we can do is we can take this use the same workflow and embed that attribution that represents the geometry and the type of foundation in a similar fashion through a polygon entity within ESRI web map.

So that corridor, for instance, imagine running through the same workflow here by building out a process for grabbing the alignment of that wall and then also the attribution that the subassembly was tailored to be able to output for readability within Dynamo. We can take that same linear feature and then using a different function, just take those points and collate them in a fashion so that each segment captures the start and the end point of the retaining wall design.

And then the same process, export to geoJSON, and then upload that to the web-- to the web map. And yeah, here on the left side, it's just a pretty similar scenario. It's just now you're out in the field. And you have the ability to take a look at a retaining wall that's represented in plan, just a simple line.

But if there's information that you want to capture about it during construction, you could do a similar workflow here. So there's any CAD or based-- CAD or BIM-based system that has the ability to scripting, especially with Python or any other language that has the-- makes it easy to communicate information via external text files, this is some of the ways-- some of the things that you could do with that. And yeah, so that's all the content that we had. So yeah, thanks for listening. Thanks, everyone.

MICHAEL DESILETS: Thank you.