Spot on Site: Maximizing Jobsite Robotics for Evaluating Housekeeping

BROOKE GEMMELL: Hi there. My name is Brooke Gemmell with Skanska. I'm an emerging technology manager. And I'm here today to talk to you about our work with the Spot robot on our project site. With me, I have my co-presenter Evan Reilly.

EVAN REILLY: Hi, I'm Evan Reilly with our Emerging Technology Group at Skanska based out of our Durham, North Carolina office.

BROOKE GEMMELL: So we want to start giving a background to this project, how it started and really how we landed on the specific topic that we chose to focus on with our Spot research. So it all started last summer in August 2021. Autodesk and Boston Dynamics put out a call for proposals. And they were really looking for novel use cases with Spot. There's been a lot of work with Spot in the past and work with construction projects, but Boston Dynamics was interested in how could people use Spot in more advanced applications beyond just photo capture or laser scanning.

What could we really do to start pushing the boundaries and meeting really critical needs on our project sites? One of the things that I really loved about this work is it's a true collaboration between different parties, all leveraging our unique skills and resources. With Boston Dynamics, they're bringing their history of work with the Spot. They have all of the hardware and expertise to help us leverage this the best way possible.

Autodesk has a really special technology center for lab research. And they also have the technical support to help connect us with experts to learn more about machine learning and do really robust testing in a safe lab environment. Skanska brings our construction experience and our jobsite testing opportunities, so we really have the experience building construction projects, having boots on the ground, and understand what those unique challenges are that are ultimately opportunities to leverage Spot on a project site.

When we first started thinking about what to work on, we really wanted to make sure that it was going to be the right problem. So instead of pitching one specific problem to Autodesk, we actually started with a two-day workshop to ideate and prioritize use cases. This was held at the Autodesk Technology space in Boston.

And we brought together about 15 folks from local Skanska project sites, as well as people from Boston Dynamics and Autodesk, to really talk about what are the opportunities to leverage Spot, where are there unexplored areas, what's within the technical capabilities of Spot in the six-month-long research and testing time period that we had. And ultimately, we landed on housekeeping as the primary use case. With housekeeping, there's so many opportunities to develop complementary workflows to automate and improve existing processes. And we really felt like this was going to be a new avenue for Spot that had many different phases of iteration that we could continue developing based on how far we got within the research.

The other thing that was really important about this is we had lots of folks from our project sites that actually told us what they thought about Spot, how Spot could be helpful, and what were their biggest needs on site. Ultimately, one of the reasons that we landed on housekeeping is that it's a really big indicator of actual project health and can be a big factor for our safety records on site. In some ways, it feels like the tip of the iceberg, but it really has such a big impact on how it feels to be on a project.

And right now, there's a lack of consistency in how we actually manage and track housekeeping on site. So we thought Spot could be a really great opportunity to increase our consistency, our efficiency, and just the robust level that we're actually able to analyze housekeeping on site and automate this process.

Ultimately, we landed on our project goal, which was automate housekeeping inspections with Spot, increasing frequency and consistency of inspections to improve job site housekeeping and safety record over time.

We have several research questions that we started with that helped to lead our testing and all of our work with Spot. The first is, can the housekeeping inspection workflow be completely automated with Spot. Basically, we wanted to know if we could make an end-to-end process that would make it so there's no human interaction with our housekeeping workflow. We wanted to know if the quality of photos captured by Spot could adequately identify housekeeping issues for further analysis.

We wanted to know if Spot could cover the same area as a human in less time, if Spot could autonomously navigate a complex job site without supervision in a repeatable way, and then, how do humans interact with Spot on a job site. Do they behave differently? Is Spot a distraction or a safety risk?

As this was really the first time that Skanska was working with the Spot robot, we focused a lot of our research also on maneuverability, just how does Spot function, maybe some of the more basic things that we just wanted to know because it was our first time bringing Spot on a job site. And then we had more complex questions specifically relating to our housekeeping use case to understand what is that repeatability over time, how well can we automate that, and what does that process look like now versus how could we develop and evolve that in the future.

EVAN REILLY: So now that we had a clear research objective that everyone was bought into, we started actually planning our research program, what it would look like, and where we would conduct the actual testing.

We knew that we needed to first test Spot in a controlled environment, obviously, before taking the robot out to the field on an active job site, which has its own unknowns. So on the first day, we use the lab space at the Autodesk Technology Center in Boston for our initial onboarding, getting comfortable with the manual controls, planning an autowalk mission, and capturing photos from a 360 camera that we mounted on the back of Spot. Once we felt comfortable with the workflow in the lab, we also identified two job sites in Boston for field testing the next day. One was an empty office building that was to be demolished, which proved to be a perfect testing environment. And the other was a new high school under active construction.

Again, our goals with the jobsite testing was to evaluate Spot's agility outside the lab on a real jobsite and also to see how people react to Spot walking on their projects. We also recognize that conditions change every day on a jobsite, even in a lab environment. And so we returned to the tech center and the high school project two months later to see if we could rerun the same missions that we used previously.

One of our research goals was to investigate the efficiency gains of Spot navigating the jobsite and collecting photos, instead of a project engineer or a superintendent, for example. So we benchmarked the time it took for a person walking with a selfie stick along the same route at all three testing sites. Next, we have a short video with some action shots from our testing at the lab and out on site. Please enjoy.

[VIDEO PLAYBACK]

[MUSIC PLAYING]

[END PLAYBACK]

So overall, I think we were really impressed with Spot's ability to navigate around the lab and the jobsite and maneuver around most of the obstacles that we put in its way.

BROOKE GEMMELL: With our maneuverability testing, we were really looking to evaluate the difference between human and Spot. And so with our test trials, we were testing the duration that it took for the complete capture of our job site with the person and with Spot. We found that it actually took Spot about twice as long to walk the same path. I think that when you really think about it, it makes sense. Spot's moving at a much more slow and steady pace. And people are able to more quickly move around the space.

So we really found that it actually took Spot more time, but that also impacted the quality of photos that were captured. As far as navigating obstacles, Spot was able to navigate almost every single obstacle that we put in its place. We found that where Spot had a hard time navigating, it was a strong correlation between actual housekeeping issues on site, so places where there were a lot of cords or hoses on the ground, open exposed holes, basically things that would be a housekeeping issue on site where also places that Spot would have issues navigating.

We looked at mission repeatability, and that was a really important factor for us of how well Spot could handle going to the same job site time after time. We had a two-month gap in between completing our first walk and our second walk. And we found several areas where Spot was unable to carry the entire mission through because there were some significant changes on site, whether that was having stored materials in the way of the path or there was actually painting, or drywall, or additional framing. It was helpful to see where were some of those areas that Spot had challenges. And after talking with Boston Dynamics, we learned a little bit more about autowalk path planning and how adding additional [INAUDIBLE] or navigating the space in a little bit of a different way can increase that repeatability.

As far as some of those obstacles, we had a lot of fun trying out lots of different construction challenges. We put Spot through basically everything we could think of on the job site, going upstairs, walking over pipes, metal decking, rocks, steep hills, and Spot was able to get through so many obstacles in really an expert way and navigate the site just like a seasoned veteran.

We did find a few areas where Spot struggled. The first is any glass wall or clear partition. The Spot sensors have a hard time seeing that. And so they'll often walk through clear areas, so it's important to know we need to be careful about that.

Also power cords were something Spot had a challenge with. And as we were doing on of our autowalks, Spot's leg got caught on a cord. And luckily, we were able to stop Spot before it pulled down a light tower, but something to be really aware of. That's another issue where that's linked to housekeeping. We want to make sure that we have our cords tidy so that they're not out of risk of being snagged and pulled.

We also saw plastic sheets were hard for Spot because it felt like an obstruction. Hoses and cords were a really big trip hazard for Spot, really easy to get caught on the foot. Cardboard rolls, as you can see, were probably the biggest foe for Spot, ideally, not something we would have lying around the jobsite. But it's right at the capacity where Spot will try to step over it, and then once that material actually moves, it's really hard for Spot to adjust.

The other issue that we found was caution tape. Spot can see caution tape and won't go through if it's at the right height. But if it's too low, Spot will step over it. And if it's too high, Spot will go under it. So it's really important with caution tape on the site, if we want to keep Spot out, we essentially have to add an extra layer of caution tape at the right height to keep Spot from going into some of those areas.

The other thing that we talked about, that was a really big factor for us with the mission repeatability is the ability for Spot to go back into a space and navigate and complete a mission autonomously without needing human intervention. In this space, we had a really big issue. We were unable to continue the mission. And we had to manually override and navigate Spot through the area. As you can see, there's a really big change of what's going on in this space.

On our March testing, it was completely open and clear. And in May, there's a lot of obstacles and stuff in the way. So the actual path routing went right through this area. And there wasn't a large enough obstacle avoidance tolerance for Spot to navigate around and make its way through this obstacle without needing manual intervention. It's good for us to note, if we want to be having Spot walk consistently through these areas, we need to be mindful of staging areas and where material is going to be laid down on site, as that will directly impact Spot's ability to navigate through those areas.

EVAN REILLY: So beyond just capturing progress photos, we also wanted to investigate the downstream analysis and actual use of this visual data. We decided to leverage existing tools that we already use, like StructionSite, for example, to manually flag issues based on our standard housekeeping scorecard. So as we all know in the construction industry, housekeeping is an important indicator of productivity and safety.

At Skanska, we pride ourselves on maintaining safe and clean job sites. Currently, our EHS leaders and project teams complete monthly housekeeping reviews using a scorecard form, which is a standard. And this includes checklist items, like checking for clean floors and making sure that we're maintaining a clear means of egress.

So according to the workflow that we envisioned with Spot, the robot would run an autonomous mission at the end of every workday and capture photos. After being sent to the cloud, these photos would then be reviewed by our safety staff in order to complete the scorecard and, of course, take action to fix those issues in the field. In another scenario, these photos would be analyzed by machine learning, by an ML model, that flags potential issues for our EHS staff to review or the AI would actually complete the scorecard for them automatically. With an onboard computer on Spot, the analysis could also be done in real time instead of waiting for the photos to sync after the mission is completed. And in the future, you could conceivably have Spot recognize these issues in real time and take action in the field, like the idea of having Spot pick up trash and plastic bottles as it walks by.

So this is what we attempted to simulate, using tools like StructionSite for the manual analysis and New Metrics to do the automatic tagging of photos with housekeeping issues. And we compared the results. From the manual workflow, we basically reviewed all of the photos on StructionSite that were taken from both the human walking with a selfie stick and from Spot, and we added notes to flag issues based on our housekeeping checklist. This took about five to 10 seconds per photo, depending on how busy the environment was. And we found that about 2% to 15% of the photos actually contained a housekeeping issue that was worth noting.

We also compared photos from a human holding a selfie stick to photos from the camera that was mounted on Spot. We observed the same photo quality despite the difference in speed and movement. But both had similar views of the floor conditions.

There was, however, a very clear advantage with a higher vantage point with a selfie stick of being able to see above tables and equipment and also to see if trash bins are empty or full, for example, which is an important thing to check for as part of our housekeeping process. But also, because Spot is slower, there's also more photo locations because the photos are extracted from a time lapse video every two seconds. This is both a good and a bad thing, because there's less gaps in photo coverage, but it also creates more duplicate photos to review.

Here's an example screenshot of how we leveraged the Notes feature in StructionSite to manually flag and categorize different housekeeping issues.

And as I said, we also wanted to test using machine learning for automating some of this subjective analysis. And so we engaged with New Metrics to learn more about their platform and their existing housekeeping SmartTags. Based on our manual review of the photos that were auto flagged with high confidence, we found that all the flagged issues that New Metrics brought to our attention were valid, but actually only included about 20% of the total unique issues that we had identified manually.

Here's an example screenshot from New Metrics which shows how you search for their housekeeping SmartTag and how you can filter for the different confidence levels. Now I'll turn it back to Brooke to talk about the reactions from our craft.

BROOKE GEMMELL: Thanks, Evan. One of the really important things for us with bringing Spot on site is we wanted to make sure that this wouldn't be a distraction to our craft workers and also that they really understood what we were trying to do with this research. So while we were on site, we interviewed a lot of different folks to get their understanding of what they thought about Spot, what they thought Spot could be used for, and we want to share just a little bit of that response with you.

[VIDEO PLAYBACK]

INTERVIEWER: Can you tell us a little bit about Spot and the Spot project here at Belmont?

DAVID WATTS: Yeah, it's been a great experience to see this leading-edge technology coming out of Boston Dynamics and Autodesk. And it was so unique for me, personally, because I got to be involved in the planning sessions for these activities. And how we landed on the topic of trying to deal with housekeeping on the job site, I thought it was really unique, the democratic process that we used to get to that conclusion. And of course, housekeeping is a topic that's very near and dear to me. To be able to see us using robotics to address that concern is something never in my career did I think I would see something like that.

INTERVIEWER: Any other comments or things you want to share about Spot?

DAVID WATTS: I'm just excited. Having been in the industry for 33 years, long time ago, we struggled with getting people to wear hard hats. And the idea of wearing safety vests was non-existent. The only people that wore safety vests, back in the day, were safety professionals and surveyors.

Today, we've got a different culture. I think Spot is going to help us push that envelope even further. So I'm excited about it. I couldn't be more honored to be involved and engaged with Autodesk and with Boston Dynamics and with our innovation group. Our innovation group has done a great job of really lifting this up. And I'm just really excited about it.

BROOKE GEMMELL: While we were on site today, we got to talk with a few different craft workers and just hear what their impression was of Spot. Overall, it seems like people are really excited when they see Spot. It's something new. It's something exciting. But I think that there is just a little bit of fear because it's something that they don't understand.

We kept on hearing that it's really important that we communicate why we have Spot on site and what Spot's actually doing so that it doesn't just appear like a fun robot to have move around the project site. But it's actually completing a task. And really, it's completing a task that can help people out, that can help our people on the job sites by saving them time or by putting them outside of a situation that they could be having more risk or exposed to hazards.

So it was really great to just see all of the enthusiasm from craft workers. Folks thought it was really exciting to see Spot. And they wanted to get Spot to fetch and they wanted to see Spot do some tricks, so overall, a really positive reception. And I think people are really hopeful that robotics like this are going to make their jobs easier in the long run and really contribute to some positive change in the industry.

JUSTIN CRAFT: Housekeeping is important to me because it makes for better morale on the project and it's a safer project and progress can proceed easier without a lot of things in the way. I think maybe some people might be skeptical of what the actual role of the robot is, like is this thing spying on me or am I going to get in trouble if it catches me doing something, that kind of a thing. I think there might be some skepticism with that.

[END PLAYBACK]

BROOKE GEMMELL: So we want to circle back to our research findings and talk a little bit about where we landed after all of our research. Our first question was if housekeeping could be completely automated with Spot. And we found that that can't be done with the current capabilities of Spot. Basically we need to build out our integration in order to fully automate this process.

There's a lot of steps in play as far as how do the photos get created and how does the capture get generated and tied with the autowalk. We've been talking with our partners at StructionSite as to how to build up that integration. And then when we talk about how do we get the photos off of the camera up to the cloud, there's just different steps that aren't currently able to be done an automated way but could in the future once that development is completed.

The next question was about the quality of the photos captured. We found that Spot was able to capture photos at a good quality for identifying housekeeping issues. But the low height is a bit of an issue for seeing the entire site. It's a little bit hard to get an entire vantage point and, also, it's not the view that people are used to seeing. Taking photos at eye level is a little bit easier to absorb and more quickly recognize some of those issues than the low height that we get from Spot.

One of the value adds from Spot, though, like Evan mentioned, is the increased coverage and just having more photos to evaluate on a project. We found that Spot could not cover the same area in less time than a human. Our human operators are faster, able to complete the same area in about half the time. But they're going to have less photos generated from that walk, so it's a bit of a pros/cons there, if we want to have a faster walk or if we want to have more photos.

Then our next question was, can Spot autonomously navigate the job site in a repeatable way. We found that really depends. It really depends on the quality of the autowalk, on how that was initially set up, the number of [INAUDIBLE] that are created, and also just the amount of changes that happen on a construction site.

It was a little hard to definitively answer the question because we were only able to test on site two times over the course of two months. If we were running Spot every single day on a construction site, we would have a better sense of how repeatable those machines were and how often intervention needs to take place to adjust the autowalk and make sure that Spot can navigate onto a site.

As far as how do humans react to a Spot on site, we found that people really had a lot of curiosity and responded with open minds. They really wanted to know why Spot was there. I think there's a perception with a lot of technology tools that they're toys, that it can be a waste of time or a distraction. So we really wanted to communicate to them that we were doing this research looking at housekeeping applications so they know that Spot is there to complete a task, just like any other coworker, any other person on site. But we found a lot of people are optimistic about the future of construction robotics and how they can help save time, reduce risk, and really just increase the operations of a construction site.

So that's all that we have for you today. Thanks so much for tuning in to listen more about our work with Spot. Feel free to reach out to Evan, myself, in the future if you have more questions. And thanks so much for letting us share today.