Design Smarter, Not Harder: AI-Driven Comparison and Optimization with LLMs and Autodesk Forma

PRESENTER: Hi, everybody. This is Design Smarter, Not Harder. We are going to learn how to use AI-driven comparison and optimization with LLMs and Autodesk Forma.

ANIK ALAM: Hi, I'm Anik. I'm working as a computational designer in Mithun. I have my master's of science and design technology from University of Washington. Most of my work focus is making AI tools especially tailored for AC industry.

VALENTIN NOVES: And I'm Valentin Noves. I'm the CEO at E-Verse and have a bachelor's degree in architecture, an MBA, and a certificate in strategy and innovation for MIT.

ANIK ALAM: Here I'm presenting my firm, Mithun. It is 2023 AIA Best Architecture Firm award winner. So we have a diverse group of people in the team. We have landscape designers, urban designers, interior designers, and also architects.

I, myself, representing a small group of people named CoLab. And in the CoLab, it works as a sandbox for the whole firm. And we provide different opportunities for people from different backgrounds. There we do computation, make software developments, and all other crazy stuff.

We are a very big firm. We have three branches all over USA. I'm talking from Seattle. There are other branches in San Francisco and Los Angeles.

VALENTIN NOVES: And I'm representing E-Verse. We're basically a bunch of architects, engineers, and nerds who have a huge passion for technology and innovation on the AEC. We're an international firm based in Miami, Florida. We're around 30 software developers. And we have projects in the USA, Europe, Latam, and Australia. And most of the stuff we do is either related with digital construction or design automation.

And in this class, we have a couple of objectives. The first one is going to be to learn how to use LLMs and the Autodesk Forma extension. Then we're going to move on learning how AI analyzes design aspects. Then we will help you acquire skills to integrate AI-assisted design. And finally, you will learn the limits of AI.

So let's get going. And let's start learning how to use LLMs and Autodesk Forma extension.

ANIK ALAM: In this class, we'll be mostly talking about pre-design stage. Most of our work is concentrated in pre-design stage. So if I want to explain very shortly what is a pre-design stage, it's the very beginning stage of any architectural design. At this stage, we make different concepts, different massings. After we fix one design solutions, then we make design drawings and go for construction.

This pre-design stage is very important. Why this is important? Because at this stage, we have the most amount of client communications. We have to compete with other architectural firms to get projects in our hand. At this stage, we do the maximum amount of site analysis. And site analysis includes so many good insights for the design.

We do maximum amount of feasibility study. At this stage, we get the most amount of directions of how the design should be moving forward, like what should be our design approach. We define major problems in our design. So this is a very important stage for the whole architectural design process.

If we think about how this stage is done by designers and architect, in the very old period of time, people used to drawing a lot with pencil and pens in their sketchbooks. But with the passage of time, now we have more powerful tools. And with those tools, now we can have a more good and concrete understanding how our design should look like.

Now we have more computer-aided design tools, like SketchUp, Rhino, and AutoCAD. So with these tools, we can now design and explore more and more and have a better understanding how this design is going on. Later, in the realm of computer-aided design, we got Grasshopper and Dynamo. This helps us to do parametric exploration of our design solutions.

So what is a parametric explorations of design solutions? It's about you make constraints and make a parametric design options where you are changing different parameters and getting different design solutions based upon them. So that also expedite the process of exploring more and more design proposals where you can just change the slide and you can try many, many, many other design options.

VALENTIN NOVES: And just in case you have never tried, let us give you a little bit of background on how Forma works. You first select a site. For example, I'm selecting Miami. Then after you select the site, you select a specific area that you're going to cover with your project. In this case, I'm just moving a little bit towards the seafront.

Then the project is going to be generated for that specific location. So what you need to do is select a specific area to get the surrounding buildings, and then those buildings are going to be exported, imported into your project. So as you can see on the screen, those buildings, the surrounding buildings get imported into your project.

The next thing you need to do is define a size limit. So after you define your size limit, you define the basic shape of your building. And that will not only create the shape, but also the levels of your building and will allow you to get surfaces, levels, heights, et cetera, from your building and then also the type of building, like if it is residential, commercial, et cetera.

Then the good thing is you can, for example, analyze and do a similar analysis and then compare it to another project. As you can see, you can get multiple proposals and quickly get the differences between them, or you can do other analysis, like microclimate or wind, as you can see on the screen right now.

And then after that, you can add stuff like vegetation. You can add roads to the project. Or you could even start modifying the internal pieces of your project, for example, defining rooms, defining corridors, defining stairs, shafts, et cetera. So it's not only about external volumetric pieces, but it's also about defining the overall of the interiors of a building as well, and then compare it to other proposals. So that's how it works.

ANIK ALAM: So Forma is a truly game changer in the pre-design stage. And it is coming out with so much benefits that you can exploit at your pre-design stage.

VALENTIN NOVES: So if you think about it, the current proposed processing Forma works in a way that it's amazing, but it still has a lot of steps. So you first, let's say, you define proposal A, B, and C. And you think, all right, let's just go with the shape of proposal B.

So then you will start iterating and maybe working on the facade. And then you're going to decide again on all right, I like proposal B, too. And then with that specific facade, you might start working in the interiors. And the process is repeated over and over again, where you get multiple proposals, get one, and then polish it more and more every time.

So this is a repetitive process. The problem is that you have to define options. You have to analyze and then make decisions and then generate new options again.

In the process, it's missing some automation. So our idea is that you could define, let's say, three options and then automatically get an analysis that allows you to quickly decide, not take a lot of time for deciding which is the best option, but just do it really quick and then be able to give directions on the adjustments on those specific design options. So instead of you doing it manually, we're thinking on a process that allows you to do not only the analysis automatically, but also the adjustments on the geometry as well.

So the first thing we need to do to get this process working is getting more data. The problem with Forma is that the amount of data you get is limited. And it's a lot of data, but it's limited. You can get dimensions and you can get great analysis like solar analysis or wind or daylight.

But if you want to, for example, compare costs of buildings, you cannot. If you want to get zoning information, density of the surrounding buildings, or construction code to understand if one proposal is not compliant with the code or factors on sustainability, land use patterns, traffic, vegetation, nothing of that comes with Forma. So our idea is to find a way to get this data into our extension.

ANIK ALAM: So when we are talking about getting a huge bunch of information and make our design solutions more data driven, the major problem is not all the data are open access. And also, when we get the data, which is open access, it's not also formatted in a comprehensive way that the designers can get insight in a very quick way.

So we need to find out a way that the designers can get a good amount of information that is easily accessible. And also, the information should be super comprehensive for the designers that they can digest pretty fast and go for further design iterations. So we decided to leverage the power of large language models, because the large language models are trained with huge information. And they can answer so many questions of you.

We decided to incorporate OpenAI's large language model with Autodesk Forma. Our main intuition of using this large language models in the design process is if we see this graph, like if in the past, if we needed something good and fast, that becomes expensive. If we need something fast but it's cheap, that compromises the quality. If we think about it is good and cheap, that also becomes slow.

So the AI or the current large language models are filling up all these criteria. And you can get a huge amount of information from those large language models. So that's our main motivation of using or incorporating large language models in the pre-design stage. And we decided to connecting the large language model in Forma.

VALENTIN NOVES: So then the last piece would be to make adjustments to the model automatically. And for that, we need to first get the geometry that we want to modify and then the adjustments that are requested by the user, for example, adding labels or subtracting a piece of the building. So for that, we will need to get code that we can inject into our app and then run the API or Forma automatically. And you will be wondering where we're going to connect Forma with OpenAI, but how we're going to connect it and bridge this gap.

So what we're going to do is we're going to use Forma extensions. And we're going to show you in less than a minute how you can create one. So you go to create an extension, you define the name of your extension, you define the project you want it to be connected to, and you define the size of the embed view. And then you just hit Create. After that, you add the extension to your project. And the only remaining piece of this is adding the code you want to run.

So for that, I'm going to show you some JavaScript code that is going to be on the embed view, that the only thing that it's going to do is it's going to color the surrounding buildings of your project every time you hit Click. So you click yellow, for example, you hit color buildings, and then all the buildings are colored. And the same happens with red. And you can even reset the colors. And as you can see, it's extremely easy to create an extension with Forma.

And that's the way that we're going to bridge the gap. We're going to create our own extension that is going to allow the user to see the data that is going to be processed, make questions, and even send requests to make modifications to the model itself.

ANIK ALAM: So before we make our extension and make it super technically sound, we stepped backward and asked ourselves like, what are the things we should be doing that actually truly benefits the designers. So as I have mentioned before, Mithun is a big firm with a diverse group of people. We have landscape designers, urban designers, architects, all different people in the firm.

So we did a firm-wide brainstorming and get their thoughts to know what are the things we should be doing with this extension. We learned about what are the bottleneck situations in any design process, what are the things they are expecting from different technology, and get in the process.

So we gathered and collected a lot of ideas from this diverse group of people. Someone was asking for making some custom chatbot that actually helped them understand the current situation of the project. Someone was talking about understanding the setback for any site location. Someone was also trying to get some sustainability instructions from the AI.

So after we have collected all this information, we have subdivided or sorted all those ideas in five key factors. So now we have a better direction what we should be doing, how our extension is going to help, especially designers, and what are the things we should be focused on.

VALENTIN NOVES: So with all this in our mind, we were ready to kick this project off. So the first thing we define was the workflow on how the app was going to work. So we thought on a way of getting the information, process it in a way that the user could understand it, and then interact with the chat bot or AI based on that, take decisions, and then request changes.

So we propose a couple of sections. The first one was going to be the project information to get all the information we could from Forma, then weather information of the location, costs of elements, code compliance, sustainability, and a chat bot to speak or interact with the AI.

And the second step was to define the UI and UX experience. For that we use Figma. We define a user flow. And we base our design using Material UI library. So this will allow us to be more efficient when we code this extension. And based on that, we define how each of the different sections was going to look on the project.

After that, we define the software architecture. So the idea was that first the user would interact with the Forma extension. That Forma extension will be a front end hosted on Amazon S3. That would make it really efficient and really cheap as well. And then that front end will connect to Amazon Lambdas. And each of these functions we're going to connect to OpenAI.

So if we speak about the front end, it was just a simple React JS app. The main section of the project was just an app. And then we split each of the different sections on a different component. And then each of those components were connected to a layer that was called Data Fetcher. And this layer was the one in charge of doing all the processing, not only getting the data from Forma, but also interacting with each of the endpoints on the back end.

ANIK ALAM: And now if we talk about the back end and how the things in the back end is getting connections from with the front end, as we have six different tabs, and like all the steps is a different source of information, so to keep things super organized, we have made different custom GPTs and also we have defined different sources of knowledges. And all our different Lambda functions are accumulating these sources, getting information from those sources, getting different answers for different queries from different custom GPTs.

So each of the Lambda functions are connected with different endpoints, which all of are built in Amazon Web Services. So the front end is connecting all these endpoints. So for any tab, like when we need some information about the code compliance, we need some information about the sustainability, the front end is reaching those endpoints. And the Lambda function is processing all the information and sending back the outcome in a very defined template.

So now we are going to talk about all the six steps we have added for the extension. We can see for the very beginning, we have this project information. So we are collecting information for multiple proposals that you are making in your project. So this project information is making a comparison between multiple proposals.

So think about one of your proposal have more floors and another of your proposal does not have that much floors, but it has a bigger building footprint. But when you are seeing all this information in a very comprehensive and digestible way, you can make better decisions super fast.

And think about you are making more proposals, and it's like having more and more. You might be losing track of what's happening. And also Forma is a web-based tool where people from different location can also access and make their own proposals. This can be super helpful in keeping track of what's going inside in your project.

The next one is the weather analysis. And I want to explain why this is important. So anytime you set up your project site locations, you need the information, like what's the daylight situations in that site, what's the precipitation in that site. Is it like, do we need to design for cooling or we do need to design for heating? All these information are super important.

And think about you are a designer, you have to design things. And you are just surfing internets to get all those informations. So we decided to put all this wonderful and amazing information in a very visual way. You just decide which one is your site, and you get all the information.

But most of the information are coming from ChatGPT. At this stage when we ask the ChatGPT to get different information like temperature precipitations, it took a lot of time. So we decided to figure out alternative sources that will not take that much time.

Then we exploited the Open Access API endpoint coming out from Open-meteo. So Open-meteo is a very wonderful site. And if you put any location, it will give you all different information about that location. And it's the public access open API. And we are connecting that API endpoint with our extensions. You're getting information from there and visualizing to the designers in a super digestible way.

We worked on cost estimation. So think about you are making multiple proposals, but at the very beginning stage of your design proposals, you don't need such detailed understanding of what the cost should be. But a very rough estimation can be super helpful in taking more rational decisions for your design. And also you when you have an understanding, like my project or this proposal will be costing around this amount of money, you have more rationales and more strong points to convince your client or more understanding which directions your design should go.

So we asked the GPT to do a rough estimations of different design proposals. And these estimations is justified with many other out information. And whenever you are making a proposals, these cost estimations are done instantly with the help of custom GPT we have created.

Another very important thing, and the designers always look for is understanding the code, the construction code or the rules you have to follow when you are doing your design. But it's super tedious job for any designers to look for all those construction rules and find out all the necessary information out of that.

So in the age of large language models, we can get the benefit in a very good way. So we defined for any given proposal, this is our template, these are the informations you need to know. And the Forma or the large language model from his huge knowledge base is getting those information for us. Think about you just start a project and you learned about what should be the setback of your design solutions, what should be the maximum height, how much green space you should be having in your design. All the informations are super ready for you and you just start doing your design.

One of the challenges we have in this process is when you give the large language model, this is our location, and you should be giving the construction rules, it doesn't know what is the zoning information of that locations. And also, if you think about getting the zoning information and searching all over the internet, sometimes it's costly and sometimes it's super tedious.

So we figured out another way of how we can solve this problem. We used Mapbox API. And Mapbox has an API endpoint for their tiles informations. If you think about you drop a location, it will give you the information of 50 nearby objects.

So think about you just drop the building in the downtown area, it's getting information of nearby restaurants, nearby roads, nearby buildings. And when it's a very huge amount of information of the surrounding and when we have fed this information to ChatGPT, it was super good at predicting what kind of zone the site is located. So if there are so much restaurants and other things, it was very much good at understanding that this is a commercial zone. If there are some more townhouses or single family residences, it was very good at predicting that this is a residential zone. And we also match this information with ground truth and also, that work pretty well.

So the last thing we have to work on is not last but not the least is every time people start working on design, especially in our firm, we look for sustainable strategies to follow in our design solutions. But to get those understanding of what steps we should be taking, what are the requirements to do the sustainable designs, you need a very expert person for sustainability team.

So we decided to make a instructor, which is an AI instructor for sustainability. And this instructor is getting information for you from a defined source. So it's not telling everything super random, but it is trained in a way to get information from particular site and giving you instructions.

We trained the AI or the sustainability instructor to get information from this site, which is called 2030 Palette. This site has wonderful accumulations of so many important information and data. So from all the given information, the instructor have an understanding what sort of suggestions you would need from this site. And it gives you all this information.

And the most important one, which we have worked on, is having your personal chatbot for the project. So think about you are making so many changes in your designs. You are working on your different proposals of your design. But if you want to have a very human conversations about your project, this can be super helpful.

You ask anything, any query about your project, and you get the replies instantly from the GPT. And the GPT is always up to date of what's happening in your project. Any questions you ask, you can get a wonderful reply from it.

This is a information flow framework, because as there are so many complex things happening inside the extensions, there are so many informations are passing here and there. So we decided to make our workflow super organized. It's simple but complicated.

So if you see, we have three sort of information is passing through all the spaces. So one information we are getting from Forma, one information is the predictions made by the GPT. And another information is we are getting from different outsources like Open-meteo or Mapboxes. All these informations are then get to the GPT, and we are getting wonderful replies in a very wonderfully-defined templates. So this is a demo. Now we'll be showing how you want to use that.

VALENTIN NOVES: So the first thing you do is you open your project. In this case, it's a project with three different proposals. So you open the extension. First thing it does, it loads the information of the project, then the weather data, like wind, daylight, temperature, et cetera. And then it loads the cost that it calculates per trade in total cost of the building, so you can calculate it and compare it to between proposals. Then code compliance, where you get specifics of that location, what you need to be aware of, and what's compliance and what's not.

Then for the chatbot, you can ask questions, for example, is this an option with more floors. So you make a question, you get the response, the proposal number two. What is a proposal that might be the cheapest to build? And then it tells you it's proposal number 3. So you can interact with the data of the different proposals real quick and in a more easier way.

So during the project, we encountered some problems. The first one was data synchronization. We had that specific problem dealing with the Forma, with sorry, with the ChatGPT API. We also had problems accessing multiple proposals. That was something that it was not even available on the Forma API.

And we ended up doing was requesting the engineers from Autodesk to allows us to do this. And it was a surprise, because they allows us to solve this real quick. So in a matter of weeks, we were able to get the data and the comparison between proposals.

And then finally, the thread-based continuous discussion, because our idea was that you could see the history when you use the chatbot. And we basically allow you, every time you open the extension, to maintain a thread.

So every time you close it, that thread is lost. But as long as you keep the extension open, you can see the whole history and you can go back to something you already asked. So it was something powerful that allows us to make the tool more easier to use for our users.

ANIK ALAM: So you can do a more continuous conversations rather than a segmented conversations. You can keep talking with it in a very human way.

VALENTIN NOVES: And then the idea is that you learn how AI analyzes design aspects, like make more data-driven design recommendations, make a comparative analysis of proposals and feedback loops and interactive improvements. So we know that everybody thinks that AI is on everything. But believe us, this is a major help with this type of processes.

So we're going to show you an example here. We have two different proposals. And as you see the shapes, the amount of floors and square footage is different between the three of them. The only thing they share is the same site.

So what you will get when you use the extension is, for example, in this case, is Miami. You get the reference point. You get the different building height and a quick comparison. You get the temperature, the precipitation. And you can even get real quick the total cost. And you can see which one is the more expensive or which item makes it more expensive.

And then when you go into the chatbot, you can query all this data to get answers. So for example, which one of the proposal has the best return of investment? So for example, on this case, it's proposal number two, because you get more square footage and the height of the building.

But then you can ask how you can even improve the ROI further on proposal number two. So in that case, the ChatGPT tells us that based on the construction code, you can still increase even more the building height, so you can increase the square footage. So then you can just ask, hey, how large can the proposal number 2 be? And based on the construction code, it says that potentially you can add five more floors and approximately 50,000 more square feet to the building.

And you can even ask questions like this on, hey, based on the trends and all the surrounding buildings and also on the consumers need in Miami, what amenities can you use that improve the profit when you sell the apartments? So it makes some suggestions that are well-aligned with the type of buildings in Miami. And then you can add it to the design.

ANIK ALAM: I would also like to add that GPT was instructed to talk with you in a more-- it was trained to talk more about sustainability. So all the instructions you are getting from it for design improvement, its main focus is making the design more sustainable.

VALENTIN NOVES: And then you can acquire skills to integrate AI-assisted design. You can automate repetitive tasks. You can enable conceptual design feedback. Or you could even integrate design exploration with natural language inputs as we have seen.

ANIK ALAM: So one of the main, another intention of this class is we are going to explain how we are going to make your custom GPT. As I have explained before, we have made a whole bunch of custom GPTs. And those GPTs was responsible to get data in a very given format.

So now here we are showing how you are going to make your custom GPT. And when you buy OpenAI's API, in the very left-hand side, you will have a wonderful user input. And it is pretty straightforward how you make your custom GPT.

There will be an assistant or the model ID, which you have to use when you are getting reply from it. You name your GPT. You put custom instructions. And these instructions are pretty important, because this actually defines the scopes of your GPT.

Then you figure out which models you are going to use and also in the instruction, you define how the information will be transferred. You can define your own template. You can tune different parameters and get your very customized GPT. So the benefit of using the custom GPT is you will get more satisfying answers when you are not using custom GPTs, because these are trained in a very particular way.

When we trained our GPTs, we always give it the format. So if we think about the cost estimation GPT we have worked on, we had a defined format of input. And all these informations are coming from the front end. And when it is asked to process all this data, the outcome was also defined. So it cannot give us only plain text or the result in the other way.

So the benefit of getting those informations in a defined template, we can visualize them. It can be constant all the time. And we can make more digestible informations for the designers.

This is another example for the code compliances. So for the code compliances, we got this input and we got this outcome. And if you see, the template we have prepared is well-organized, so that anytime people are looking for information, they get a wonderfully organized information instead of just getting a bunch of plain text.

VALENTIN NOVES: So another goal of this class was to learn the limits of AI. And when we're speaking about AI, we're speaking more specifically of LLMs, which is the technology that we are using. So if you consider LLMs, it's just a piece of AI that it's part of deep learning. And it's located in between deep learning and natural language processing.

So the limitations are composed for the limitations of these two different branches. And some of them are that, first of all, it's slow. So every time you ask a question to the endpoint, sometimes it takes seconds, which it's a lot when you're dealing with a cloud software.

It's also expensive. It will charge you per tokens per word. And this is not linear. So it's not also expensive, but it's really hard to calculate the end cost for the user. And then it's not super accurate. You will sometimes not get the exact questions or it might even tell you stuff that is not even real.

And then you rely on the latest training set that the LLM was trained on. And this might be a year ago. So if you ask a question of something that happened yesterday, you might not get an answer for that.

And then something that is also really important, it hallucinates. It's improving. OpenAI has improved this a lot, but hallucinations are still real. And that's a problem when you need consistent results.

So how you can implement changes with consistent results when the LLMs are not consistent? So what we tried initially was we make a request to Forma, like, hey, give me the code to adjust a specific geometry of the project.

And we will get that code. We will need to test it out and make sure there are no errors on the console and make sure that the chain has happened. For example, hey, add a couple of flaws. If that didn't happen because the hallucination happened or because there was an error, we will have to ask again on the same thread to ChatGPT and tell them, you know what, I got this error. Can you try again? And we have to do over and over again until we are successful on making the change.

But the problem with that is that it's really slow. And sometimes, even trying multiple times might not happen. So what we decided was to make a flow that was even more efficient. So every time that you make a question and that works, we save it on the database.

So the next time, instead of going straight to ChatGPT on a new thread, it will go to an assistant that we will first check on the database and prioritize answers that we know are true. And this will make that our system will become more and more efficient over time.

So what's the future of our extensions or of our work? So we are planning to improve the efficiency by more testing it by the designers and architects. So there are many other Open Access API endpoints we can collect information from and make the information more digestible for the designers.

There are some API endpoints, which you have to buy, are like subscribe for their information. We are planning to also include those sources of information, so that it gets more information heavy. So also, we will be trying to get what are the suggestions we are getting from the GPT. We will be customizing more and more our instructions, so that we can get more design suggestions that makes the world more sustainable.

Also, for now, it's connecting to the project. But like in the future, we are thinking of connecting or having a user database where the reply can be customized based on the users. So think about when the GPT will learn about different project of yours, not only different proposals of one project, it can have better and better understanding what's your intention and what you want to do in future. Thank you very much.