Connecting Substation P&C Drawings to Mesh Models with an iPhone/iPad LiDAR

JACK HOPSON: Greetings, and welcome to Autodesk University's prerecorded case study session of "Connecting Substation Protection and Control Drawings to Mesh Models with an iPhone or iPad Lidar." My name is Jack Hopson. I'm the Technical Solutions Executive for Utilities here at Autodesk, and my primary role is really just making sure utilities, whether that's electric, gas, telecom, or water, is just getting the most out of our solutions. And I support all the account executives across North America and Canada.

A little bit about myself, I've actually been working with Autodesk products for over 20 years now. I started off my career in the telecom industry, doing design work for fiber optic and RF coaxial distribution systems. From there I taught 3D design at a local college in Saint Louis, Missouri, where I'm from, and was the program director for a few years there.

And I just had just an urge to get back into industry, so I ended up going back to utilities, except instead of telecom, it was power delivery. So I spent about eight or nine years working for substation design as a manager of design there. And with me today, actually, from Ameren is Steve Jacquin.

From there I actually left and went to go work for an Autodesk technology partner who builds utility-specific software and ended up here at Autodesk. So that's a little bit about me. Steve, why don't you introduce yourself and tell them a little bit about yourself?

STEVEN JACQUIN: OK, thanks, Jack. I'm Steve Jacquin. I've been a Protection and Control Design Supervisor at Ameren for a little over two years. I've been at Ameren for about five, and I've been working in the substation utility industry for about 11 now.

JACK HOPSON: Thanks, Steve. You know, if you're watching this, odds are you know a lot about Autodesk and us as an organization and what we do, but odds are you don't know a lot about Ameren. So Steve, why don't you tell them a little bit about Ameren?

STEVEN JACQUIN: Sure thing. So Ameren is a medium-sized utility company that provides both electric and gas. We service both the Illinois and Missouri areas. We have about 2/3 of the service territory in Illinois and about half the service territory in Missouri. Between both Illinois and Missouri, we have about 1.2 million customers on each side of the river.

We also generate several different types of electricity. We have nuclear facilities. We have coal fire, hydroelectric, and renewables with wind and solar. Thank you, Jack.

JACK HOPSON: Thanks, Steve. So I have to do this as an Autodesk employee. This is our safe harbor statement, and there is a lot of literature up there.

But really the key point is, if I do make comments about possible features that may be in future products of ours, that you don't make purchasing decisions based on that information, because we tend to pivot and kick features out with various types of software releases, so again, just to impress on you not to make purchasing decisions on that. And in the spirit of that as well, any software that we showcase to you during this case study, we are in no way endorsing. We'll tell you why we chose the softwares we used, but again, in no way are we endorsing any software used for this case study.

So let's go ahead and kick this off. So this story actually starts back in 2015, when I worked at Ameren. I was walking down the hallways one day, and I happened to look over the shoulder of a protection and control designer.

And they were doing a wiring job, and as I watched them go from one screen to the next, they were scrolling through all these wiring pictures and then they were going back to their CAD drawing. So I walked up and asked them, is this how you go through all the panels? You just float through all these pictures?

And they're like, yeah, it's better than the way it used to be. We used to get these markups, you know, the wiring information, but yeah, they just send us these pictures. And it's better than it was before, but yeah, I have to cycle through all these pictures.

And I thought, man, there's got to be a better way to do this, so I went back to my desk and I thought about it. And at the time, like a lot of people in industry, they were playing around with photogrammetry, and within our Autodesk software suite, I knew we had ReCap and some software that had this capability. So for those of you who don't know what photogrammetry is, it's just taking multiple pictures of an object from various angles, and the software brings those together in order to create a 3D mesh model.

Coincidentally, I actually had just purchased a GoPro camera that had a wide camera lens, which was required to do this, so I thought, well, let me try this and see what results I get. So I did what a lot of people did at the time. I went around the office, started taking pictures of various objects.

So here's me in 2015 taking a picture of my desk, the sandbox and my little LEGO set I used to just unwind with a little bit when I was trying to come up with ideas. So once I got those pictures and I ran them through ReCap and processed it, I was really happy with the results I came up with. And so I thought, you know, I bet we can use this for wiring panels. I think if we take multiple pictures of a wiring panel that we can create a mesh model, so instead of a wiring designer looking through all these pictures, they could just bring up the model.

So I knew I was going to need help with this because, in my opinion, it's an "art versus a science" situation. So is the technology good enough that it doesn't matter how you approach what you're trying to do, the technology's just going to take care of everything? Or is it a situation where how you do that dictates the results?

And this is very much an "art versus science heavy" situation, so I knew I was going to need help. So I actually reached out to our Autodesk rep and asked if there was a photogrammetry expert that could come out and help me. And so there was, and they came out and we decided to do a substation that was about 25 years old.

And we went out and we took about 120 pictures of this panel at a small distribution substation, and once we processed the photos and had the model come up, I was really happy with the results. I was able to see all the wiring letters. I can also see all the pin numbers as well.

And then as I scroll through to the open door here, sorry it's a little bit jagged as I orbit this, but you can see that the etching on the hardware is there as well. So my brain began to spin like, is this really useful to, say, engineers that really want to see that asset? So I asked around, and most of the protection and control engineers said, yeah, it'd be great not to interpret a drawing, just be able to actually open up the asset on the screen and see it how you're seeing it here.

So while we were out there, we did a few other things as well. So we went ahead and did a photogrammetry model of the entire wall. So once I zoom in to this third panel here, you can see that everything is very, very crisp. We took some time on this particular panel. And reflective objects tend to cause an issue here, so what was surprising was, we were actually able to see the text through the glass here.

And then as an example of art versus science, as we move down the panels, we actually started getting a lot more sloppy. We sped up the pictures. We didn't take as many, so you can see how the quality drifts off as we move towards the right.

We did a few other cabinets as well, I/O and some relay outs here. So you can see on the board, you can actually see the etching on the board here for this assembly relay out. So again, we were really happy with the results we were producing.

And then we went outside and did a transformer really quick and fast. We maybe only took 25 pictures of this, and we added a scale to it and, again, just to see what we can do with the technology. So it caught the attention of Ameren's internal marketing. They did an article, and a lot of people were approaching me like, is this something we're going to start and do? Like, we see a usefulness for this.

But as I began to think about it, I had really to weigh the pros and cons of an as-built workflow because, as field technicians do, they don't always have time when they're wrapping up a project. So when I was looking at the pros and cons, great, we have a mesh model of this asset, so an engineer can look at it at that stamp of time, what that wiring panel looked like, when that model was created. But that's really all I could come up with.

You know, the cons, the art versus the science, how you take the pictures, that's hard to expect a field personnel to really take time to take those pictures correctly and the amount of pictures that's required for a wiring diagram. The postprocessing of the photos, you have to go through 100, 150 of those photos and take the ones out that are not in focus, because if those are in there, it's just going to tear up your 3D model and you're not going to get really anything usable back.

And then the file size was just particularly really large, and because of the file size, we said, eh, you know, we're going to have to have a network connection. We're going to have to have a shared file. There's just no easy way to do it.

So I decided to kind of put that whole idea and experiment on the side, and my thinking was maybe we'd use it for some really faraway substation that was in the middle of nowhere. But that's pretty much-- I just put the idea to the side. So fastforward five years later, 2020, not necessarily a year of great remembrance, but what did happen during 2020 was Apple released an iPad with a LiDAR scanner, and a few months later I think they released the iPhone.

And so it really reinvigorated my thinking on this, and I thought, you know, maybe we can revisit this. Maybe this is an easier way to more focus on that science part of it, to where it doesn't matter how we're approaching it, the technology's just taking care of it. So I thought, yeah, yeah, let's go ahead and revisit this, and I decided to do it both ways, with both pieces of hardware.

So I have an iPhone 12 Pro and an iPad Pro 3rd generation. And for those of you who are big on hardware specs, as you can tell, the iPad is way more powerful than the iPhone. However, the iPhone has a better camera, so I thought, well, let me just test both of these and see what we come up with.

So I thought the 3D Scanner App would probably be best. One, it's free, and two, the other ones cost money. They do have free trials, but I would've had to put my corporate card on the Apple ID. And it just kind of turned into a headache, so we decided to just stick with the 3D Scanner App.

However, we did do some preprocessing kind of experiments with the other softwares, and we'll talk about that. So of course, I didn't have access to the substation, so I contacted Ameren. And Steve and I, we got together just on a company meeting, and I told him, I said, you know, I have this idea.

It would require me to be on a substation and test some hardware with the iPhone or iPad. Is this something you'd be interested in joining me in? And Steve was really excited and very helpful to join me on this.

So we talked about this, Steve, I think about a year ago almost now, and then we decided to wait till next spring to go out when the weather was nice. So there's us in this-- in front of the wiring panel, getting ready to take this, and when we started, we did the iPhone. So here's me taking an iPhone scan, and really it's just swiping it left to right.

Now, the iPhone was nice because I had more agility with it. I could get into the nooks and crannies with it. I wish I kind of had a selfie stick to get up higher, to be able to get some of the top of the racks, but again, it did work well.

However, I will say, as we were scanning this, it was starting to slow down a little bit, and that iPhone started getting very hot as we were doing multiple panels. So the more scanning you do for the duration of time, the iPhone really was heating up pretty hot. So after we did a few panels with the iPhone, we decided to try the iPad.

Now, the iPad is not as mobile. It's not as agile, but holy cow, was it way faster. We ended up having a much faster processor time. We were able to get the scans back much quicker, and as you can see, as I'm swiping left to right, it's just capturing it much faster.

So from a time-usability standpoint, we were really liking the iPad results more than the iPhone. And really, when it came down to the end results, which we'll talk about here in a little bit, we really didn't see a difference in-- as far as camera quality goes. So the better camera on the iPhone really-- we didn't see a pan-out of a better image quality.

Now, as I said, we did play with some other software here, Polycam and Sitescape, and Polycam pretty much worked in the same way. So you pretty much targeted your asset, you hit record, and it shaded as you were capturing the data. And the pre-post-- I'm sorry, the pre data actually looked the same as the 3D Scanner App, so my assumption is that the post results are probably going to be about the same. I have a lot of colleagues that use the Sitescape app as well, and this is just primarily a point cloud. So with this you're not really going to be able to grab that fine text imagery, but it certainly has a lot of great applications when you're doing big things outside, or rooms or things of that nature.

So after this is done, it previews a texture, and you can see there's some artifacts there that we could go in and erase in the app. I generally tend to do that stuff post moving it to an Autodesk software to do that. But what you do is you process it, and again, the iPad way outperformed the iPhone on this.

So once you process it, you actually see that model right there, right on your iPad screen, and you can zoom in. You can look and see what the quality is, so if you really want to redo it, you certainly can. But the imagery came up really good right on the iPad, but actually when we did move it to the desktop software, it came in even more clear. And again, we'll show that here in a little bit.

So one of the big things is where do we go with this file, right? So when you go down to the right, there's a share button, and once we hit that button, it gives us all these file options. So OBJ, STL, DAE, FBX, those tend to be the big ones that are used.

We decided to use an FBX file as a means to move and work with the file. So of course you can Airdrop it, message, mail, because it's only 7 megabytes, but it dawned on us, why don't we use Autodesk Construction Cloud for this, to move this file? So I was able to save this FBX file to a folder on my iPad, and then I was actually able to just go and open up Autodesk Construction Cloud right in the web browser.

Now, if you're familiar with Autodesk Construction Cloud, obviously we have ways we can upload files. There's reviews, transmittals, if you're familiar with this. Well, the key thing is, is all the players and the people that are members of this team get notified when things are uploaded.

So that little blue button with the cloud, I click that. I click Choose Files from the computer. There's my FBX file.

I select that, and then it starts to upload. So once that file is uploaded, everyone that's attached to that product is notified. So again, FBX uploaded to Construction Cloud, everybody in perpetuity now knows, whether that's a mobile device, whether it's a desktop in the office or a laptop on the move.

Now, what we didn't realize, and I stumbled on this on accident, is within the Autodesk Construction Cloud browser window, you can actually view this FBX file, which shows up right there. Now, what is interesting is the quality. Once you get into the nooks and crannies of the text, it's not quite as good as the desktop, but it really came in really, really well.

Now, again, the artifacts are there because we haven't cleaned it up, but again, don't need special software. The web browser will do that. Now, it did not work with the PlanGrid app, so note that.

So what do we do with this next? Once we had this file, we downloaded it, and then we opened up ReCap with the FBX file. Now, if you all are familiar with this, ReCap automatically transacts an RCM file out of this. I don't know why it does that, but RCM is a ReCap file, native file format.

So once we opened up the RCM file-- and you can see I cleaned up the artifacts on the side here-- I wish I could tell you we had a home run, but not quite. Now, the first pass that we did, a lot of the text was very clear. We can see what pins it was attached to, and as we move down across the equipment, we can read the attachments there. And so again, we thought, as we started looking at this, we might've had something on our hands here.

The problem is, is as we moved further down the rack, the imagery really started overlapping itself. So we did this several times, moving slower, faster, closer, farther apart, and we really weren't getting much difference as we were moving down. So here's a good example.

As we're closing down low, you'll start to see the imagery overlap here. So once I get scrolled down here, you can see now it's starting to overlap the imagery, and so the text is not lining up. And so this starts to become very hard to read and distinguish where these connections are happening.

Also, reflective surfaces didn't give us as much of a problem as I thought they would. Again, we just kept running into the same problem of overlap of the imagery. But I had an additional idea, since we did have this file and there was the first couple of racks we were able to look at.

I said, what would it be like if we moved this into a HoloLens, so what if we went to an augmented reality environment? So I contacted a friend of mine at Automationforce. And the FBX file actually was not working out very well, so we actually went with the OBJ file. And the OBJ file ended up working pretty well, so you're going to get ready to see the results here.

So this is a recording of him with the AR goggles on, and once that loads-- give it one second here. And now there's the rack, so again, the thinking is, with everyone working at home, what's a good way to visualize assets in the field? If we can capture this data and then view them through an augmented reality, we can actually see assets right in our home office.

Now, the gentleman that helped me put this together, he told me that it was coming very clear through the goggles, but the recording was not picking up the clarity. So while it may look fuzzy to you, what he said he was seeing through the glasses was quite clear. And he is convinced he could actually get this model to look even better, but unfortunately, we didn't have time before this recording session was scheduled to take place.

But again, the possibilities are really neat. I love this idea of, instead of going out to the field to look at this asset, I can open up the file right on my AR goggles and then look at all the pin connections and everything here. So again, as we step away, we start to see more of the panel as we look up and down the panel. Just a lot of opportunity here as this technology gets better and it gets more refined.

So again, I mentioned that we did use Autodesk Construction Cloud here, and we kind of retooled the thinking a little bit because while it didn't really work well for finite text detail, if you were doing bigger things that didn't-- you didn't need to have to read, it worked out really well. So we said, you know, this could still be useful if you want to capture, say, underground conduits or wiring.

So this is a landscaping job I have going on in my backyard, and it only took me a minute and a half to take the iPad, scan the area. And you can see I have some underground sprinkler conduit going through there, electric as well. So what we can do is, once we post this the Construction Cloud, whether we want to do it for design or whether we just want it as a system of record to see what it looked like before we put the dirt and the concrete on top of it-- and again, we can create point clouds out of that, too. So whether you do the point cloud in a raw form or whether you just take the mesh model and convert it, we can actually bring that into the AutoCAD user interface or whatever software you use, and then we can use this as a reference for our design. So again, very powerful tools, very easy to get the data.

So again, to recap, FBX, we create that file. We post to Construction Cloud, and then of course, once we post to Construction Cloud, everyone is notified. We can then put it into ReCap and create an RCM file out of it.

But the big thing is system of record. So I'm working with a lot of utilities now. They're piloting Construction Cloud, and the big decision is, do we keep the versioning in the cloud for the project and keep the system of record in Vault? And that generally tends to be the practice that more and more people are thinking we're going to go towards as we're doing things in the field and then tracking the versioning in Construction Cloud and then taking that system of record back to Vault.

So this is where Steve's going to come in. He's going to talk about how we connected the mesh model to the actual 2D CAD wiring diagrams within Vault. So, Steve, it's yours.

STEVEN JACQUIN: All right, thanks, Jack. So yeah, using the Autodesk Vault Pro app, we were able to load that RCM file in there, and we added a lifecycle to it similar to that of our master DWG file that we had. So we were able to match revisions, match properties, and keep all of the same information with the mesh model as the AutoCAD file.

We also went as far as to create an attachment between the RCM file and the associated DWG. That way, whenever you pulled the AutoCAD file, you also got the RCM file automatically. So you can see that here, whenever we click on this RCM file and then go down to where it's used, it'll show that the RCM file has that section under Attachments where it's attached to the DWG down here. So what that did is that whenever you got that DWG, every time, it would automatically allow for you to pull that RCM file. All you had to do was modify this one dropdown setting to always grab attachments so whenever you hit OK here, you would get both files automatically.

So once we get the file onto our local machine, then the next step is, how do we want to link the two together? So we pulled up our AutoCAD here, and we would look at several different ways that we wanted to kind of link those two together. on the next slide here.

All right, so with the AutoCAD file pulled up, here we have the RCM file, and we tried to do an embed straight directly into the AutoCAD file. So it did work. It did allow us to open up the RCM mesh model.

The only problem with this is that it did not allow for the provisioning control, but it alleviated the need to have the attachment. So there were some benefits to embedding it directly into the file. We tried a different method as well, which was using a hyperlink.

So just taking the piece of text here and creating the hyperlink, we were able to add some general information and link the AutoCAD file directly to that RCM mesh model using the hyperlink here. And as I mentioned, whenever we got one file, we ended up with them both, so we didn't have to worry about the link breaking and not being available. Adding some basic information, like when we captured it or what revision it was for, we were able to add that into the hyperlink as well to kind of help more with that document management and the ability to revision one or the other in case, for whatever reason, you had to rev up the drawing but then were unable to do so to your mesh model. That way there is no confusion, no risk of having bad information with a mesh model that was out of date linked to a DWG.

So the other thing that the hyperlink allowed us to do as well was that not only could we open it directly out of AutoCAD with the hyperlink-- so that's what we're demonstrating here, here's our mesh model pulled up-- but we were able to also open two other ones. So this is the first method, which was the hyperlink directly out of AutoCAD. The other way that we can do this is by plotting the hyperlink into a PDF. So using this Hyperlinks option inside the PDFs, you're able to just create a PDF that includes that hyperlink in it.

And here we can also see the RCM file, the embedded image, was still there as well, but it's no longer able to be clicked on or be able to be used inside the PDF. Whereas, the hyperlink still is able to be used inside the PDF. So it really did have some advantages there. That's our second method of being able to open that RCM file out of AutoCAD and the PDF here.

So the last one here is just drilling down in the folder structure and finding where the Vault downloaded that file locally on our machine. So there's our RCM file as well. And whenever we open that, that's where you can see here, as Jack mentioned, there some of the wiring information's not exactly coming through how we were hoping. But there is a lot of good information that's available there as well because we do have the ability to look at some of these test devices and see the pin lists and what pins are used. We're able to get some of these colors on the wires and if there's a single wire or double wire on that, so there's a lot of possibilities, but maybe not everything that we wanted. OK, thank you, Jack.

JACK HOPSON: All right, thank you, Steven. So some of the key takeaways, we have a few of them, but we've really narrowed it down to three for this particular case study and presentation. The iPad really is preferred to the iPhone. While you don't have the agility and, you know, you've got to manage it a little bit more, the processing speed, it just far outweighs the iPhone, so it was much preferred for this case study.

The second, which we stumbled on, really using Autodesk Construction Cloud to manage that file from the field back to the office really seemed to work out and be very beneficial, and again, being able to view that FBX right in there, these things we stumbled upon really got our gears turning. So again, using Autodesk Construction Cloud to manage the files and the data you're gathering really, we feel, is the best practice for that. And again, while the technology really isn't ready for wiring because there's still more of that art than the technology is taking care of, again, the results that we got were-- too much overlapping for the test. But however, for other types of work, it can completely really share in the design ability with what you're looking at with mesh modeling, bringing in a point cloud, or again, if you do just want to capture an FBX model, keep it as a system of record so you know what that was before you put the dirt on it, before you put the concrete over it.

So again, those were really our three key takeaways that we wanted to share with you. So if we were live, this is generally where we would do a Q&A, but if you do have questions, there's our contact information, so jack.hopson@autodesk.com, sjacquin@ameren.com. If you do have any questions about this, feel free to reach out. Again, thank you for viewing our case study session here, and we appreciate it. And until next time, have a good one.