The Power of Reality Capture and AI Benchmarking for Improved Project Delivery

ELISSA FLANDRO: OK. I think we're ready to get started. Good morning, everyone. Thank you for coming. This morning we're going to be presenting the power of reality capture and AI benchmarking for improved delivery. We're going to start out just with some introductions first.

WILL PLATO: So I'm Will Plato, the VDC manager at Hensel Phelps for our southwest region in Texas, New Mexico, Oklahoma, and the surrounding states. I've been with Hensel Phelps a little over 11 years now. Gotten to travel pretty much all over the world.

And really the thing that I think sets me apart really is my experience of starting out in the field for probably the first seven years of my career before making the jump over to VDC. And one of the things I've really brought to the company is not using technology for the sake of technology but making sure that there is an attainable ROI. And so that's what really led me to HoloBuilder.

ELISSA FLANDRO: And my name is Elissa Flandro. I do customer success for HoloBuilder. My background is in civil engineering. I studied at Stanford for my undergrad and my masters. I then worked in construction in San Francisco on two projects, the San Francisco International Airport terminal three for United Airlines and then also the Transbay Terminal, which just recently opened in downtown San Francisco.

I then went on to work for a drone company for a short amount of time. And I've been at HoloBuilder for about two years now. My role really involves working with all of our general contractors and owners, trying to understand how they're applying HoloBuilder on their projects and really getting the most value out of that.

So to provide you all an overview of what HoloBuilder is, we've basically built a platform to organize and store 360 degree images. I like to explain that it's kind of like Google Street View for your job site. So you basically give your team in the field a 360 degree camera just like the one that's here. That's actually a Ricoh Theta V camera.

So for any of you that are going to be wandering around the expo later today, we actually have a booth with Ricoh. They'll have their cameras there. And you can check them out and learn a little bit more about 360 cameras. There are a couple different cameras that are actually compatible with our app, which runs on both Android and iOS.

And the app is meant for just capturing images. So we tried to make it very fast and efficient process so that your teams out in the field can capture those images and then simply upload them to the cloud. And they'll be organized by location on your floor plan and by the date in which they were captured.

So this video is an overview of HoloBuilder. It opens up in the enterprise dashboard so you can manage all of your projects in one location. And now it's opening up into the web editor. So this is basically the platform where you would edit your photos, add text, share them with other people. You'll see in the upper left hand corner that's where your floor plans are. And those little blue circles are each of the 360 degree images that have been captured in the project.

You'll see that the pull down menu is also what we call time travel. So over time, it's doing a side by side comparison of two different images that were captured in the exact same location so that you can see exactly what has happened on your job site. It's now showing that you can add in additional 2D images. You can add in specs. You can even take measurements straight from the photo if you're using a tripod.

And we do also have an integration with BIM 360. This is filling out an issue in BIM 360, which will then be within your BIM 360 account so when you log in you have each of those issues that are listed out. And there is a really great visual that is supplementing your BIM 360 issue.

So who is using HoloBuilder today? Currently, we have about 12,000 different active construction projects that have deployed HoloBuilder and are capturing 360 degree images. That's about 1,000 different general contractors and owners that we work with internationally. And this amounts to about 54% of the top 100 E&R general contractors that are using our software today.

So HoloBuilder is basically a digital replica of your project. It's a document that contains all of the progress that has gone on during your project. It's all organized, as I mentioned, by location on a floor plan and also by the date in which you captured those images. And it provides a visual of your construction site so you can easily send it to anyone that is involved on that project.

We primarily work with general contractors, but we also have a number of owners that will deploy it and provide this as a tool for their teams on site to use as well as architects and engineers. We've built out a couple of different integrations that support BIM renderings as well so you can implement your design along with all of your construction documentation.

So why are construction companies using HoloBuilder? What we've built is really a process for teams to go out and-- go out and regularly capture any change that's happened on a project. We work with a number of companies that maybe don't have any type of process currently in place to capture all of those images. Maybe they're going out with their cell phone, or using a digital camera and just taking standard 2D images. And this is really the process that we've built for providing to the people in the field. Primarily within construction, but as I mentioned, we've started focusing on supporting the design and planning phases.

And now we're starting to see a lot more companies and projects that are deploying this on their facilities, and using it during that O&M phase. It's great-- excuse me-- because they have all of the documentation, and they know exactly how a building was created, and they can understand what's behind a wall or what type of equipment was installed.

So the basic components of HoloBuilder, as I mentioned, includes the mobile device with the job walk app for your teams out in the field. It also includes a 360 degree camera. So the Ricoh Theta V, and there are a couple other cameras out there that are compatible with the app as well. The majority of our customers do have their data synced to the cloud. We do offer an on premise solution as well. We understand that sometimes for some owners, that's what they require-- where they require their data to be stored.

And then on your computer, you can access any of your projects through our web editor. You can add as many people as you'd like, so that they can all work together on a project. And basically add anything, in addition to those 360 images that have been captured.

So we do also have three integrations that we recently released this year, as I mentioned, to support design. We have the Revit plug-in, as well as a Navisworks plugin. That allows you to work within Revit or Navisworks and then export those 360 degree renderings and bring them over into HoloBuilder. And then we also have the BIM 360 integration with issues. So that if you create an issue within a 360 degree image, that will all appear in your BIM 360 account, so that you'll have a list of each of those issues that have been created. Along with a unique URL where you can click and go back to that HoloBuilder project.

So what we've been working on, and basically how this relates to AI, and how we've been developing AI, is that we started with focusing on the fastest and most efficient process for capturing real-- for reality captured data. That includes the job walk app and a 360 camera. We're also going to talk today. I'll share a video with you about laser scanning, and supporting point cloud data in HoloBuilder as well. We can incorporate satellite imagery. So in the beginning phases of a project, when you're doing your site plan, you can add in a satellite image.

We've also built out an enterprise dashboard. So that once you start using HoloBuilder on a project, you can expand to multiple different projects within your organization. And then this last component is what we've been working on over the past couple months, which is the AI. And that is basically-- our focus has been on progress tracking on a construction site. And we do that through object recognition and deep learning.

So here's a diagram that gives a good visual description of what we're doing with AI. So each of these images represents one of those time travel images that your team would be capturing out on site. Starting at the very beginning, you have a BIM rendering that maybe your designer provided. Either in Revit or Navisworks. And then your team in the field has started adding multiple different sets of images.

Most of our teams will go out on a weekly basis and capture updated imagery on their site. And so what we've been working on is basically identifying certain objects and looking at the difference between the two images that were uploaded week by week. So for our work with Will, we identified six different objects on his site. And in the future-- we'll talk about this a bit more-- we'll be including a larger list of materials that we can identify out on a site.

So today is actually-- we're really excited to introduce SiteAI. This is the first time that we've shown or shared any of this AI with anyone outside of HoloBuilder. And we're going to show a video, which is actually what we've done on Will's project. And he can kind of explain a bit about the project itself. And how he's been using it to help with his work.

WILL PLATO: So really, everything Elissa said, it is just that easy. I mean, kids these days, they just don't know how good they have it. Because it wasn't what, three, four years ago. We were out there with our point and shoot Nikon cool pics cameras, taking pictures and logging them all. Well now you're going out there with the HoloBuilder app and the 360 cameras, reducing your picture counts by a factor of eight, and logging those directly to a floor plan. None of that back in file management and maintenance system.

But really, it's just a technology that has enabled us to do more. So this is out in College Station. So those are geographically challenged. That's a college in Texas. Do I have any Aggies in the room? Hey, gig 'em.

So this is part of the A&M expansion up to the Relist campus. There in Bryan, Texas. And it'll be their new academic headquarters for that campus. So we've been taking pictures since day one. Got it all logged in. But really, the power behind that we have found, really, is the integration through the AI with our scheduling. Where we can go link this to our P6 and analyze the individual items in there to go see what is truly happening out in the field. Yes. We have our superintendents, we have our staff out there, every day, all day. But there's hundreds of things going on. They're distracted. They're looking at things. Well now, we can go do, and utilize a process that we already have in place with our photo documentation. And use the AI in the computer vision that HoloBuilder is now offering us to go intelligently link what those field activities are happening out there, back to our scheduled items.

So if you have any inclination that you can already tell, there is a lot of power within that capability. So framing's the easy one, because that's enabling activity then allows trades to go really follow along. With all your overhead MEP installation, you're topping out critical walls. So we wanted to go see OK, we've accelerated with our framing and top out. Now what happens? Did we go communicate back to our contractors and our sub community to make them more efficient with these accelerations in this schedule? As we kind of flip things around the building and how we're putting together.

So here's perfect example with the duct work. Being able to go slide to the left as well. So did that take a superintendent to go pick that up, go modify his P6? No. The computer went and flagged it for us. And is pretty amazing, in my mind, of just having been out there and that superintendent role. And you're trying to go keep everything up with your three weeks or six weeks. Make sure that hips back to the CPM. This really is a powerful piece of communication tools. I mean, it kind of gets back to the old days when we had the job site radios. And you'd go have a conversation on that radio and you talking to everybody on that job site. And then we started narrowing it down with the next tells and the push to talks. And then we all got iPhones, and so now we're having direct communications.

Well this broadens up that communication tool again. Because you're absorbing all these different data sets and bringing it together, and putting it back into the hands of the people that are able to go make these decisions on our job sites. So my mind-- super powerful tool. We're using it, testing it. And so, it has been absolutely-- I mean, it's almost game changing in a certain sense.

ELISSA FLANDRO: And I just wanted to point out as well. Kind of that the flow and how it works with in HoloBuilder.

WILL PLATO: Oh, yes. I'll hit on that.

ELISSA FLANDRO: This is just that the normal web editor as you would use it today. You'll see that on the left hand side, all of those images that have been captured all appear right here on the left hand side. And then you have your floor plan up there as well. And each of those locations is tied to the floor plan. With AI, we have this new feature, or interface rather, that actually shows a timeline. So you can see exactly when a material has been installed, and when that process is first starting. So it makes it really easy to search for a particular material within the project.

WILL PLATO: So we've got drywall framing, drywall squared ducts, circle duct, and CMU. Just some of key activities that we wanted to go test the system, and also be able to-- almost be that artificial intelligence psychologist, and help go train the computer vision and the thinking to go support those different materials. Easily identifiable, and it works with that system.

So the other thing we do on our job sites, is we are a huge proponent of laser scanning. And so, my laser scanning not only starts out with the existing conditions. If we're going in and capturing a space, whether we're hitting a greenfield site sometimes, or if we're doing a renovation. But we also use it throughout the life of construction. Not only as a QA QC tool, but really a communication tool, and a validation check to make sure that our subcontractors are installing correctly. And we're able to go capture anything that the design team may need to use throughout the life of that construction project. So that they're able to go incorporate the data as well.

And so, one of the fundamental things that we launched with HoloBuilder really is the integration of putting our point cloud data truly in a web interface. So that as Elissa said, this information is very shareable. And for us to be able to go share a point cloud within our 360 photo interface to a web, to any of our partners, owners, subcontractors. Especially during buyouts, procurements. And then you retain that same functionality that you would, say in ReCap. And it's a powerful system, really connecting all those different data segments together. Where in my mind, this truly drives that integration that on the construction site, we've been chasing for so many years.

So black and white scans, point cloud. And this was actually one that we're doing-- installed utilities going underground. We've also brought in some of the photogrammetry, deliverables from our UAVs if we're doing a greenfield site. So it really helps put those projects in perspective, connecting all that information together, and really closing the loop on our different sets of data.

ELISSA FLANDRO: So what can you do with SiteAI? So as mentioned, within a HoloBuilder project, with this first test that we did with Will's team, we identified six different materials that were important for him to track within his own project. And then after this next iteration, when we start working with a larger group of customers, and have more data from projects, we'll actually be able to identify a larger number of materials that are out on site.

As mentioned, with the user interface that shows that the bar with the timeline. You're able to know exactly when an activity has started, and when it ends. It also shows the particular location. So floor by floor, you can know exactly when each of those activities are starting in each area. It provides basically just a histogram of that activity.

WILL PLATO: One of the things I like about it too, is the ability to go bring in that model rendering. Like she was talking about from Navisworks or Revit it. Because you have the BIM 360 glues where you can view the models in the field. But what if you just want to go to that room location you have a snapshot. Well you can bring up that rendering, and anybody can, on their mobile device as they have access to that project. And you can go line yourself up with that room, and you kind of get that almost augmented reality type experience. So, super powerful through the construction progression.

ELISSA FLANDRO: So we see this as being helpful in processing progress payments faster. You can also know the quantity for a specific material throughout your project. And then as Will showed, you can also compare your construction progress to your schedule quickly.

So we have not yet released SiteAI. As I mentioned, this is the very first time that we've shared any of this progress with anyone else. But we will be rolling it out next year, in early 2019. And we're hoping that other customers will be interested in working with us, and providing feedback. And basically using HoloBuilder on their job sites, and sharing those projects with us so that we can continue developing this feature.

So if any of you here are interested in actually working with HoloBuidler, and would like to get in touch and actually deploy HoloBuilder on your site, and then start utilizing AI. We would love to hear from you. So you can drop us an email. And with that, I'll go ahead and open it up for anybody that has questions.

WILL PLATO: So I've uploaded three and a half gig point clouds to them. I deliver the RCS files, or if it's coming from the drone, I just deliver the straight LAS. I have not run into any size limitations. As far as the accuracy to your first question of the 360 photos, that's really going to depend upon the placement of your field engineer, or whoever is executing that photo documentation out in the field. As far as where they're taking that picture from, and locating it to that 2D floor plan. So it is subjective.

Now the laser scans themselves, that's all tied back to our site and building control. So we're accurate there. We're not saying we're using the 360 photos to go develop a point cloud. There's other solutions for that. Does that answer your question? Yes, sir?

AUDIENCE: [INAUDIBLE]

WILL PLATO: Just on photos.

AUDIENCE: [INAUDIBLE]

ELISSA FLANDRO: So the point cloud development we're working on right now, that's kind of in the development phase still. We're also actively looking for more customers who would like to give us feedback on that. It was in HoloBuidler, and currently you can take dimensions and measurements within it. But it's not yet live. So.

WILL PLATO: Yes, ma'am?

AUDIENCE: My question is with regards to building the deep learning neural network that you said you use AI for. Can you actually have a large data set for object recognition? So for example, does your system automatically classify the studs and safety walls? Is that based on the neural network? How do you develop that based on [INAUDIBLE]?

ELISSA FLANDRO: Right, so I am not the expert unfortunately on deep learning. But we have been using the projects that Will has shared with us. He has a number of different projects in Texas, I believe 16?

WILL PLATO: Yep.

ELISSA FLANDRO: That are using HoloBuilder. So from those projects, that's what we're using to train our system. I don't know the number of photos that you have in each of those projects.

WILL PLATO: It's thousands. I just speak on my perspective as an end user. I go there and I want to go search for drywall or framing or whatever material that I'm looking for. I go see it show up in the timeline. So however it's happening on the back end, it works.

AUDIENCE: [INAUDIBLE]

WILL PLATO: As far as marking it up. What do mean?

AUDIENCE: [INAUDIBLE]

WILL PLATO: No. That's happening on their back end. And so--

AUDIENCE: [INAUDIBLE]

WILL PLATO: So it's looking for those different materials, and it's recognizing that code that we have in those WBS breakdowns and saying OK, drywall framing, that matches drywall framing. Or duct work, or whatnot. So it's kind of a bi-directional link that it's creating there within that P6 database.

Oh OK, yes. I'll repeat questions. Yes, ma'am?

AUDIENCE: [INAUDIBLE]

WILL PLATO: In a certain way, yes. I mean, that's all part of the training and development that we're working with them on. Is to go bring intelligence to those sheets and that data. Really, kind of think Revit sheets back to your schedule items. Like we've done in other programs. I mean, that is the goal that we're working towards, yes. Because I want to do exactly that. As much as hands off as I can be. Yep.

AUDIENCE: What kind of time and investment are you seeing? On each site, do you need a designated [INAUDIBLE] coordinator on site [INAUDIBLE]?

WILL PLATO: So we-- so with Hensel Phelps we're kind of unique in that everyone starts off on the lowest rung and totem pole, with the field engineer position. So we have five or six individuals that are learning how to be builders. And that is one of their responsibilities every week, as they go out and capture their area of work.

And so, one of the nice features about HoloBuilder is you can have those multiple users, multiple people capturing that data. And it's all coming back to that centralized location. But that's part of their responsibility. And myself and my career going up through Hensel Phelps. I used to go out there and stand in the same locations every single week with my little Nikon and take a picture. And you know, now they're here walking around with their little selfie stick and tapping their iPod-- or iPad where they are. And it collects the picture, they go hit upload and they walk away. They're done.

So the savings that we're seeing on it. I'm probably seeing four to five hours of savings a week, per person. Yes, sir?

AUDIENCE: What's the project [INAUDIBLE]?

WILL PLATO: It is a downloadable file. When you go hit archive, you get all your 360 photos. Basically these same interface that you see on here. Along with any of that documentation, videos, PDFs. QA, QC reports. Whatever data set that you have embedded in there, all that stays with it.

ELISSA FLANDRO: And you can actually make copies as well and share that with your owner, for example, and they would not need a HoloBuilder subscription in order to view that data.

WILL PLATO: Yeah, I'd hand over a jump drive at the end project and say, here you go owner. And it's maintains the same exact functionality. That was one of my primary reasons to go to HoloBuilder. Because we had built our own interface with our 360 photo viewers attached to PDFs, that sort of thing. But really, it was that archiving capability, downstream, where I could truly walk away knowing all that data is preserved.

AUDIENCE: [INAUDIBLE]

ELISSA FLANDRO: It does not, today. The Ricoh cameras actually have GPS. So we have the ability to implement that in the future. At the moment, when people go out on site, they have their iPad and they have their floor plan up on screen. And when you tap on the floor plan, it does two things. It triggers the camera to take a picture, and then it also adds that little circle on the floor plan. So each of those floor plans are just PDFs that have been uploaded.

WILL PLATO: Yes, sir?

AUDIENCE: I'm wondering if this gets rolled out [INAUDIBLE] year. [INAUDIBLE] Will there be other criteria [INAUDIBLE].

ELISSA FLANDRO: Yeah. I mean, I think it's mostly just going to depend upon the feedback that we get from customers. You know, we just look what people-- look for people to partner with. And if there are certain things that would save them a lot of time on a construction site, then we would be happy to try and put those in AI, if it's going to save a lot of time.

WILL PLATO: Yes, ma'am?

AUDIENCE: So follow up on that question. [INAUDIBLE] training in the [INAUDIBLE].

ELISSA FLANDRO: Sorry I-- could you repeat your question?

AUDIENCE: [INAUDIBLE]

ELISSA FLANDRO: Oh. Yes. I mean, same thing. Like we haven't put together a list yet of the materials or objects that would be within AI. Earlier this year, we had gone out on a Hensel Phelps job site in the Bay Area and done a walk with them. And we basically trained this system to identify people on site. So people that are wearing a hard hat and have the appropriate safety gear on. As well as form work and insulation. So.

WILL PLATO: Yes, sir?

AUDIENCE: On your [INAUDIBLE] project, how many errors did you find between the model and the actual installation that the AI found?

WILL PLATO: Well the AI is not looking for model versus--

AUDIENCE: [INAUDIBLE]

WILL PLATO: No, it's not looking for that. That's something we go find manually in our use of the laser scans.

AUDIENCE: [INAUDIBLE]

WILL PLATO: No. It's simply looking for that material being installed.

AUDIENCE: [INAUDIBLE]

WILL PLATO: Correct. Or and-- when it's first identifiable in those photos that we are taking in those locations. Yes, sir?

AUDIENCE: So like that one photo of showing the construction schedule [INAUDIBLE] maybe you already said this-- I don't know-- are you able to [INAUDIBLE] have them side by side [INAUDIBLE]

ELISSA FLANDRO: So what it looks like visually is just that histogram that shows different points in time. And the more images you would capture, the more progress updates it would show for each activity. And that's really the only way that you would, at the moment, be able to go in and look up each of those activities.

WILL PLATO: Yes, ma'am?

AUDIENCE: If you have multiple pictures that capture the same material from another angle, does it double [INAUDIBLE] that material or [INAUDIBLE]?

WILL PLATO: It recognizes the same object because really that capture is based upon the date and time that that photo's taking. So it recognizes that, OK, this material was here, present in this photo. That photo is chronologically older than this other photo, so this was the first one-- an instance that I'm going to go recognize.

AUDIENCE: [INAUDIBLE]

WILL PLATO: Mm-hmm.

AUDIENCE: [INAUDIBLE]

WILL PLATO: Right. It's still going to go key off that first photo that says, that was there on this day. And we haven't broken it down to the minute or anything. We're still looking very broad spectrum days, weeks. Because generally, that's when we go take our photos, as weekly photos. So we haven't broken it down to the [INAUDIBLE] kind of mentality. Yes, sir?

AUDIENCE: So sounds like at your current level, you're using it to trigger [INAUDIBLE] automatically update the [INAUDIBLE] schedule based off of [INAUDIBLE] are you considering [INAUDIBLE] over the next couple of months? Are there safety issues or [INAUDIBLE]? What kinds of use cases are you excited about [INAUDIBLE]?

WILL PLATO: Use cases that I'm excited about exploring really is-- such a broad question, because there's so many different facets I go approach with it. The simple one truly is going to be the integration of more of our documentation coming in with those photos, and having that information kind of permeate itself through the history of those photos.

Lot of times we'll go add in panel schedules and stuff like that for the owner to go attach those two those 360 images. And they're readable right out in the field. So as long as those-- that information stays with that project the entire time-- as far as the artificial stuff, we heard a little bit-- kind of like what smartvid.io's doing. They're really bringing that same functionality, if not more, to it, to go recognize those objects.

And it's really just the fundamentals of saying, hey, HoloBuilder, wouldn't it be cool if-- and they go tinker with it, play with it. And generally, within a couple, three days, whatever, we get of saying, yeah, we'll go do that. And that's one of the reasons I like working with HoloBuilder is because they do help you go champion those ideas.

AUDIENCE: [INAUDIBLE]

WILL PLATO: The system has to be trained just like theirs. We were early involved with them as well, and it was a lot of constant training and really that AI psychiatrist mentality of, why does a computer think it-- thinks that it sees this? And so it's all part of those training algorithms that we worked with HoloBuilder on.

ELISSA FLANDRO: Anyone else? OK. Looks like no more questions. All right.

WILL PLATO: All right.

ELISSA FLANDRO: Thank you very much.

WILL PLATO: Thank you.

[APPLAUSE]