Artificial Intern: Let the Robots Get the Coffee

DENNIS GOFF: Hello. My name is Denis Goff and I'll be presenting the topic Artificial Intern: Let The Robots Get The Coffee. So just a quick introduction of who I am. Like I said, my name is Denis Goff. I'm a design technology manager for ZGF Architects. I'm a computational designer and what I call a recovering architect. I was trained as an architect, practiced architecture for almost 10 years, but have since drifted into the technological sphere. And that's kind of where I find myself today.

So ZGF as a firm, we're spread out across seven offices over the USA and Canada. Our headquarters is in Portland, Portland, Oregon. And we have offices in Seattle, Denver, Washington, DC, New York, and Vancouver, which is where I am based. ZGF as a firm, we're about 750 employees, give or take. We kind of have our hands in all kinds of project types, everything from higher ed to lab to medical to residential, just a little bit of everything. But kind of our big project areas are health care, lab, higher education.

We also do urban design. We even do a little bit of fabrication. We have a fabrication lab in Portland, so we're a pretty holistic firm that have our fingers in a lot of different areas. The team that I am a part of at the firm is called the design technology group. We're a team of 14 experts sprinkled throughout the firm. We serve as firm, wide resources. We're not tied to any one office, really.

We have expertise in BIM, Revit, Rhino, computational design, software development, kind of a little bit of everything. The group itself is part of our IT group. So we're kind of floating from place to place wherever we're needed. But we do have team members who are in the architecture teams. They're embedded in projects. They're the ones who are really the feet on the ground and bringing the big issues to our team.

So, on to the talk itself. Again, the title is Artificial Intern: Let The Robots Get The coffee. And that title comes from how I look at AI in the context of the architectural industry. So I'm going to start off by talking a little bit about just overall high level views on AI, and then we'll move into more practical examples of how we're actually implementing AI at ZGF.

So the first thing I want to talk about is what happens in the AC industry when one of these big new technologies arrives, what people think it could do to the industry versus what it ends up actually doing. So the first example is BIM. I think the potential and idealized version of BIM is that we'll have a digital twin for every project. The model will be an exact replica of the building and we may even be able to eliminate the need for 2D drawings, because the 3D model is the building and the building is the 3D model.

So, you don't even need 2D anymore because it's an exact replica. You could just fabricate straight from the model. However, we all know this isn't really the case today. We're all still sitting in Revit, generating 2D drawings. Those are still the deliverable that projects are built from. And some of this is, technological, some of it is just contractual. Most of the contracts that we work on at ZGF, we won't go past LOD300 because we're just doing design intent.

So handing one of our LOD 300 models to a fabricator isn't going to be enough to actually build the building. So that's just kind of an industry issue that contracts haven't caught up with technology. So we're still doing 2D drawings. So what it's really done in the overall scheme of things is made working more efficient. It's much easier to create drawings that are coordinated, data that's coordinated with the drawings. So that's really the big leap that I think that BIM has given us from drafting and AutoCAD even.

So moving on to another example, which is parametric design, computational design. When it first started popping up in projects, we're starting to think like, is this how all buildings are going to be now? They're going to be these fully custom responsive projects that are built with mass customization because digital fabrication was taking off at the same time.

So in a way, that's happened in that these big marquee projects across the world, you do see that kind of design. But that's usually projects with extremely high budgets and not your typical kind of building. So while it does take over in some instances, it's another instance really where it gets used the most is for automation analysis. And it might get-- you might see some cool parametric kind of things relegated to smaller kind of feature pieces within a bigger project.

So again, what it's really done for us on a whole is its increased efficiency in how we work. So that brings me to the big buzzword of the last few years, which is AI. And whereas the last two, I'd say the hype around them was probably overwhelmingly positive, in the case of AI, the hype has been positive and also pretty negative. People are kind of scared of what AI means for the world.

People-- it's going to come take jobs. It's going to take creative work away from creative people. And in some industries, I'd say that's true. There are industries where it does pose a significant threat. We've seen it with the Screen Actors Guild going on strike against the use of AI in film. Computer programming, I'd say there's a bit of a risk there. AI may take over some lower level programming jobs.

But in the case of architecture, I'd say that we're probably safe. And that's due to the complexity of delivering an architectural project. There's all these separate pieces that need to be interwoven and coordinated. We have 2D drawings, 3D models, text documents like specifications, just data that's coming out of the models or coming from other places. And they all need to be brought together, coordinated. They have to be compared against building codes, which can vary by zip code almost. So there's just so much complexity there that I don't see in the near future, or maybe even ever AI being able to fully do the job that an architect does.

So just to illustrate that, one thing I tried in preparation for this was I went into ChatGPT and said, draw me a floor plan of a 100,000 square foot hospital. And this is what it came back with. And if you squint your eyes and don't look very carefully, you might say, wow, it did it. But the second you start looking closely at all, you see the problems. It's got three emergency rooms, things are all kind of wavy. There's weird words in there.

It's not actually an architectural floor plan. It's an image in the style of an architectural floor plan. But that's about it. So just to take this further, I then said, thinking like, I'm working in Revit, OK, I need to cut a section through this floor plan. And what it gave me was now this, which again, kind of looks like a section perspective, maybe a section X0. But at the second you look closer, it's all messed up. It's got that kind of AI waviness to it.

And it's not related to the previous image in any way other than style maybe. But it's not something like where it's an actual cut through the previous drawing. And again, I said, OK, from that draw me a wall section. And there it really starts to break down. It doesn't even really understand what a wall section is. You can see it's sort of trying to put some details on the page maybe. But again, none of this means anything. And none of it is related to the previous two images. And so that right there is evidence that it cannot do the work that an architect does.

So the reality of it is that AI isn't going to take your job, but somebody who knows how to use AI well might take your job. And that AI should be treated as almost a tool, obviously, but almost as an employee or the way I look at it is, I treat it like an intern that needs constant supervision. So with an intern, you figure out what they're good at. And then you can start leveraging those skills to help you work better.

So to compare the two, a human intern, some pros about them is they're creative people. They're eager to help. And they're usually more familiar with the kind of new, up and coming technologies in the industry because they're coming out of school where they kind of have freedom to do whatever they want in order to produce a design project. So they're usually exploring anything and everything they can get their hands on versus in a professional office, you've got to work with the tools that particular office uses.

Some cons about interns is they don't have a lot of experience, obviously. That's why they're there. And they don't have context of how the architectural industry functions, really. Sometimes humans can be overconfident. They think know everything already. And they might not want to ask for help. So that can lead to issues if you're not supervising their work. So if we then think about AI in the place of an intern, the pros are the same. AI is very creative. We've all seen what AI can do. It's very fast and it is this new amazing technology that also has this huge amount of knowledge about the world.

But the cons again. It lacks real world experience in context. So it may a lot about what the internet has to say about architecture, but it doesn't know-- it hasn't worked as an architect in the real world. And just like humans, AI can be overconfident and it can hallucinate, and it can make things up. And one of the big issues with the AI is it rarely tells you, I don't know. It'll just give you a really convincing wrong answer. So AI, just like humans, requires supervision to make sure it's not going off the rails or just making things up.

So the big question here is, do you think that either one of those human or an AI could really jump in and just do your job as an Architect And I would say absolutely not. So like I said, it's important to figure out what is it actually good at? You don't just want to say we'll use AI. You want to know what's I good at and how can you use it.

So obviously we've all seen the things that these AI image generators can do. It's very good at image generation rendering. There's a ton of these AI sketch to render kind of companies popping up. So that's a big use case for it. It's a very good teacher. I personally use it all the time when I'm writing code for various tools. It's a way more efficient way to learn than Googling something and kind of having to find that one particular post on the internet that kind of talks about the issue you're having and then hours of trial and error.

So it's a very good teacher and it's very good at language comprehension, understanding the meaning of words, because these large language models are that they're trained on language.

What it's not so good at is math. It is not doing calculations. A language model is basically just trying to generate a sentence that looks correct. So if you ask it to do a math calculation, it's not actually doing any real calculation. It's trying to create a sentence that looks like a mathematical calculation. So, it should not be relied on for things where you actually need to do complex calculations.

It struggles even to count things sometimes. Like, I've had instances where I'm working on a list of values and I need it to do something for each value, and the results come back all jumbled and out of order. And I finally had to stop and say, all right, forget all that. Just tell me how many things are in this list. And it would be incorrect in a different way every time I ask the question. So it can't be really relied on for that kind of work.

And another thing it's not so good at is archi-speak, these niche terms that architects use for things, because in the context of what these models are trained on, which is essentially the whole internet, the portion of that training that applies to these little architect specific words is very small. So it might struggle to understand when you start talking about very specific architectural concepts. But it's getting better every day. These models get updated and get smarter. So it is improving in that regard.

So moving on now to how we're actually using AI at ZGF, we use all the kind of usual programming, usual options that everyone has seen the OpenAI, Midjourney, Stable Diffusion. Those are all great tools. And it's great that they have free options. But they can be limited in terms of what they can do, how private they are. So what we've actually done is we've started to just build our own versions of these tools. And so the first thing I'm going to talk about is something it's called-- we call it ZGF.ai. It's a web page essentially, that can be accessed from any computer in our ZGF ecosystem. And it contains a whole series of tools that we have built.

This was developed by my colleague Jonah Hawk where, by doing it this way and using the API that's provided by these services, we can basically jump over the whole hurdle of licensing for a huge firm. We're just paying for API access and that's it. And then we can hand this to any person at our company versus having to manage, oh, we've bought 30 licenses, but we have 700 employees. So we have to constantly be shuffling licenses around.

It also allows us to customize image generators or whatever tool to our own needs. We can add functionality that we see valuable. And obviously, it's more private when you're going through the API versus through these public portals because the data is secure. It's not being used to train the model. It's just being processed and then thrown away. So that's very important to us.

But it's a website. It works like any other image generator or whatever kind of tool you've used online before. But we've put our own options in. You can select style, what kind of image you want it to be. And it works just as fast as any other web interface for this kind of thing. But again, it gives us a whole different level of control.

So, the next kind bunch of things I want to talk about is how we're actually plugging AI into our Revit workflows. So the first kind of way that we're using it is just to take existing tools that we've built Dynamo or Grasshopper or that kind of thing and make them more flexible because anybody who's ever developed a tool for a project or used a tool that was developed specifically for a project, you'll know that the rules for using that tool are usually pretty strict. It's usually like, all right, you have to label everything in the model using this one parameter, and you can label it in one of three ways.

And you have to train the team to use your tool because it's expecting very specific setup from the model, which can make things really hard when you then want to go try using this tool on another project because they might have built their model completely differently. So it makes the rules very strict. But what you can do by plugging AI into this process is, it allows the rules to be a little bit fuzzier.

They don't need to be as strict because AI can interpret data and you can give it instructions for, take data and it needs to come out in this way. So you can be much more flexible about how things are done in the model because you can kind of leave that translation to a standard up to the AI. So the first kind of version of this, it's a simple use case, AI is used in a very minimal way on this tool.

This tool started out as a Dynamo script. All it does is it takes every room in a floor and generates a path of travel to exit doors. So that's a cool tool. It's very useful. But the first step in it is you have to remove the spaces that are considered circulation from the process. You don't need to generate a path of travel from a hallway, right?

So in the case of this particular project, you can see that they were careful about how they labeled. Their plan is very clear as to what's circulation, what's not. But not every project does that at all. And even if they do, they might use different values. They might use different parameters to label that, which would cause the tool to break if you plug it into a project that used a different parameter.

So what this does instead is it uses AI to just read the room, just read the room names. Every room is going to have a name in a project, and based on that name, determine whether or not it's a circulation space. So it's just a little sprinkling of AI that makes a huge difference in terms of how useful the tool is as a firmwide tool. Another way that we're using it kind of in a similar manner is what we call benchmarking.

ZGF is a large firm. We've been around for a long time. We have a huge archive of projects, which is data that we could be using. But it's hard to grab all that data from years and years and years of projects and try to put it together in a useful way where everything can be compared against itself. So in this case, we have a workflow that's been developed which, when a project finishes, essentially its Revit model is fed into this process which extracts all the rooms in the project, and then it takes those and compares them to basically an Excel list of just paired values of the name that was used in the project versus the standardized name that we want to use for our database.

So it first just kind of checks, is this name already in the list. So if it is great, you can grab the standard name and move on. But if it's not in the list, it then sends that name through AI, which anytime I say sends through AI at this point I'm talking about the Open AI API. So it sends that name up through the Open AI API and processes it into a standard name.

So we say, all right, here's the name and 20 possible standardized names. Which one is best? And then it takes that value, feeds it back into the spreadsheet so that the next time this tool runs, it's smarter and it's gained a new value. So ideally, eventually you wouldn't even need AI for this process because it's generated every possible pair of names.

So then once it finishes that whole process, it grabs all that information and then feeds it back into a power BI dashboard where we can then go in and see every health care project ZGF has ever done and compare them, all kinds of programmatic breakdowns and all kinds of the project information. And it's all standardized. So it can be useful for decision making and all that.

Another one which is again kind of similar to the benchmarking option is name standardization in models. So we have standards for how we think views should be named. However, on these big projects that are just churning through time and have deadlines every couple of days, the standards tend to fall in priority. And it's more about just producing work and you know, get the drawings out. So typically, the longer a project goes, the less standardized it becomes, just because of people prioritizing speed.

So what this tool allows you to do is it goes in, you can filter by view type, filter it however you want, just specific views. And then I will generate a standardized name based on the current name and based on a template that we've fed into it as instructions. And this is a case of supervision being important because you don't want to just set loose on your view names and just hope that it gets it right. So this tool allows you to check all the names that come out of it. You can change it if you don't like what you see, and then you can select which ones you're ready to accept and change.

The last kind of cluster of tools are tools that are based around semantic search, which you'll probably hear a lot over the course of this conference. But semantic search is different from the kind of search that we're used to, which is keyword search. And that is just go into a document, you hit Control F, and you search for the word computer. And it shows you every time the word computer pops up in the document.

But what semantic search is, it's really more about searching for meaning. So it's not just saying, OK, here's where it says computer. It's actually taking a question or a sentence about computers and looking for similar sentences in the document.

So just a brief breakdown of how that actually works is, you extract the text of a document and you break it into chunks of a given size or structure, however you decide. And those chunks are given meaning in the form of a vector. And as architects, we think of vectors as XYZ, three dimensional, that's what a vector is. But in this context, these vectors are 1500 dimensions, which is really hard to get your head around as a three dimensional being.

But computers have no trouble with that. And this is technology that is way more complex than any of the stuff that we're actually showing. And it's kind of in the background of how all of these AI things work. But so it creates the meaning of all those chunks and feeds them into a database.

And so then when you have an app where you're asking questions, it takes your question, creates a vector for the question itself, and then compares it against all the vectors in the database. So it's essentially looking for the best three answers, say, that number is irrelevant. And so then when it gets them, it returns those to the user in either raw form or you could feed it through an AI again to get a nice kind of natural language response.

So, the first example of this that we're using at ZGF is a tool called Ask ZiGFried And you may ask why that's the name. And that's because that's the only name I could think of that contained the letters ZGF in correct order. So that's what his name is. But ZiGFried is an AI document analysis assistant. So the original idea behind him was that when you're working in Revit and you have a question about building code or some standard, this allows you to stay in Revit while you get your answer to that question rather than going on the internet and starting to Google, and then before it, you've been watching YouTube videos for two hours and you've gotten completely off track.

So in this case, you might be working on some bathrooms, and say, I need to I need to know how much what the turning radius should be according to the ADA documents. And so you can just select ADA from the dropdown and ask your question and it gives you your answer right there. And it also gives you the raw text where it found that answer.

So the answer in the answer box above has been fed back through AI to give you a nice natural language explanation of the answer. But it's also giving you the source where it found that information to act as a check to make sure you weren't just getting a totally AI made up response. It's actually showing you, I gave you a nice response, but here's where I got the information. And it contains the big asterisk of you should not just assume AI is correct. You should be checking it and making sure these answers are correct.

Another tool that we're in development with right now is a similar kind of thing. But this is, you're actually talking to the Revit document itself, not a PDF. So with the PDF, extracting that meaning is relatively straightforward because it's just full of text. Text has meaning inherently. So you don't have to do any real legwork to get the text into a form where you can get useful information from it.

Revit model is a little bit different because a Revit model is full of things. It's not just a Revit, a text document describing a building. It's full of objects and data and all these physical things. So in order to create semantic meaning from that, it's a little trickier. So in the case of sheets, you could just say, well, here's a sheet and here's its sheet name and sheet number. That's all I need to know.

But with a sheet like full of details, the sheet might just be called details. But the actual details on the sheet will have more specific names, more specific information about what they are. So what you want to do here-- what I do here is I collect the sheet information, but also information about all the things on the sheet. And that all gets kind of combined into a text chunk, which is then used to generate the meaning of the sheet.

So what this allows for is a kind of more intelligent way of searching your model versus the built-in Revit search, which is just keyword search. You just type in a word and it finds you everywhere in the Revit project browser that has that word. But if you're being if say, you've been assigned to a new project or not a new project, a project that's been going for years, it has this huge bloated Revit model and you have no idea where to begin. You can use a tool like this to search the model in a better way to learn about it.

So you might be assigned to stairs. So you're going to come into this tool and say, all right, I need to find all the sheets that are talking about stair details. And it gives you a list of items that it thinks are a stair details. And this can be used on rooms as well. So you can say, all right, well, where are all the stairwells in this project. And this will give you a list of rooms. And you can find them immediately in the model. So much easier way to search and comprehend a project when you've just jumped in than just actually going through the project browser, keyword searching, and then digging through drawings and chasing through all the different view tags until you find where you need to be.

And this is the final tool I'm going to talk about, which we're calling the BIM Doctor. Because as we all Revit models have a lot going on. It's important to check on them, make sure they're performing well and not having any issues, which we do-- we have done for years and which we usually do in the form of a-- we call them project health dashboards. But the limitation of these is that it's kind of a formal process to do them.

We usually wait until some kind of project milestone where the team actually has a moment to sit and not touch the model for a bit, breathe, and look into the issues that are causing performance drops in the model. So usually we have a tool that runs this data extraction, which can take a significant amount of time. Then we have to get the report together, sit down with the team, have a whole meeting, go through issues and what they can do.

So it's a process that ends up taking, I don't know, several days to actually get through and get this information across. So what BIM doctor is, is it's essentially an AI chat bot where you can talk to your model quickly and it kind of returns all the same information that's in that report. But it's immediate and it's kind of meant for the BIM coordinators on the team who are constantly putting out fires and kind of dealing with the performance issues in the model.

So this gives them a way where they can just at any point in time, regardless of if the team as a whole has time to stop, they can just jump in. They can say like, model's sinking really slow today. Let me check on why that is. And so you can start asking it questions here. Like in this case, I'm asking it about what are the warnings in the file. And then it'll list out every warning that's in the file and how many times that warning happens.

You can ask it about groups, like how many groups are in. model. Which ones are really big groups, which ones are only used once to help you figure out that shouldn't be a group. We should make that into a family or something like that. And then you can ask it to do a whole comprehensive report which gives you all this information.

And you can ask it from that OK, what can I do to improve my model? And it'll spit out, responses which are kind of guided by our own guidelines in the background. It's been kind of fed into the tool as context. So it can read the results of that report and then compare that to our own kind of red flag issues and create recommendations for the users.

So this tool uses something called function calling, which is essentially, the AI is, give the AI your question, but you also give it a list of functions or tools it has available to answer that question. So in the case of just talking to ChatGPT, you're getting an AI generated response when you ask a question. But when you do function calling, the AI response, all it's telling you is based on your question, this is the function I think you should run. And then it returns the arguments that function needs based on the question.

So in this case, I said give me a summary of the model groups in my model and it says, OK, based on that question, you should use the get document groups function. And here are the arguments for that function. Group type is model and summary, it wants [INAUDIBLE] you want a full summary.

So what this is really doing is it's kind of when you see a lot of these apps out there where people are talking to the thing they're working on and it's doing all these amazing things, the AI is doing very little of that. It's essentially like 95% the human programmer behind it and 5% AI. So in these cases, it's not this magical thing. The AI is essentially a switchboard. It's just taking the question and routing your call appropriately to the correct function.

So it's important to realize that when you see these kind of AI tools that seem very amazing, the AI isn't doing maybe as much as you think it is in the background. So it's just important to realize that these tools seem kind of out of reach, but they're actually pretty accessible. So the takeaways of this whole talk are, AI is being portrayed in the media, in the world as this boogeyman that's this scary thing that might take over the world or take over your job.

But it doesn't really have to be that way. We can work with AI and use it for what it's good at. And what it's really great for is it's really enabling us to make better tools. Like the examples of ZGF AI, we're not changing much about architecture itself, but the use of AI in both those tools themselves. But the development of those tools has enabled us to make just huge leaps in terms of what we can do internally for our firm.

So it's at its core, it's almost a cost saver because every time that we can build our own version of something that we usually have to pay for, that's a good thing. So AI can really serve a great purpose for firms in terms of development of internal workflows. So it's just another tool in the toolbox. It's not something to be scared of.

So it's just important to remember that AI works for you. So you should be treating it as such. So thank you for listening and look forward to seeing everyone in person at the conference in October.