Automation Thinking for Civil Engineers & BIM Professionals

SHINU MATHEW: Ladies and gentlemen, welcome to my session at Autodesk University, 2021. This session is about the automation in civil and BIM industry, but not specifically about a tool or a programming language.

This session is aimed at beginners who have either started automation or planning to start. The examples used are basic ones, using Oracle problems.

During the next 30 odd minutes, we will be discussing about how to approach an automation problem, what makes it an effective and practical solution, how to structure an automation, and the steps involved in creating a sustainable outcome.

I hope you will enjoy the session as much as I would love you to. If you have any questions related to the content I'm sharing, please feel free to type them in the chat box and I will try to answer them at the end of the session.

A little bit about me. I'm Shinu Mathew, I work as the head of BIM&Allied Technologies in Arcadis, India, private limited. I'm based out of Bangalore.

I've been working in the Civil industry for close to 24 years now. And out of that, nine years with Arquette as a BIM professional and automation champion.

Now, let's crack on. The technology evolution in the design and construction industry is forcing all of us to adapt to the fast-changing digital landscape. Digital innovation into the regular design space is so rapid and overwhelming, so much so that many processes that were robust, tried, and tested a few years ago are now considered obsolete. New tools and technologies, new capabilities spring up everywhere.

While many have been working on automation since long time, the flurry of easier programming tools, such as visual programming have resulted in many people now taking the automation. If the earlier automaters were text coders and have had some background in software development, the new tools allow anyone with the right aptitude and attitude to automate.

Now, let's talk about innovation. A presentation by Harvard Business School professor Clayton M. Christensen narrates the dilemma of inventors. He talks about two types of innovations, sustaining and disruptive.

Sustaining innovation thrives on improving the current product performance by reducing defects, increasing efficiency, and speeding up production time. The product development of these place would mostly revolve around the existing product portfolio and a customer's current need or expectations. Following this will benefit the company in the short term, but failing to adapt to the needs of the future could very well result in the company's failure.

On the other hand, disruptive innovation challenges the existing practices and initially appears to be doing everything wrong. It does not conform to the framework set by the present practices and the proponents of sustaining innovation would often neglect it. Disruptive innovation, often born from the need that exist in the future that is neglected by the current players. It satisfies a niche market's unmet needs or its future needs. Dedicating valuable resources to an unproven and niche opportunity often would be challenging for the large companies that rule the market and it does not make sense for them.

But this niche opportunity could be the future. Most disruptive innovators are small startups that develop the innovation, they work with the untapped and small market, thus gaining time to try and better their offering. Once it is matured, it is unleashed upon the bigger market. This would often result in uprooting the current scenario. Some large players would fail, some would sustain by merging or acquiring the new players.

The examples of disruptive technology replacing existing ones are abundant around us, what digital photography did to film companies and what smartphone technology did the digital photography companies. Many other instances, such as Wikipedia replacing encyclopedia, Google Maps replacing expensive navigation systems, Amazon Kindle redefining publishing, Netflix, Uber, the list is long. A common trait among these new-age disruptors are none of them conforms to the business models of their time.

Automation in civil design and construction is a mixture of both. While it was around for a while, it suddenly exploded onto the larger group of people with new technologies that would help a known coder to create solutions without much fuss.

Automation, before we go any further, I would like to talk about standardization. Standardization play a vital role in automation, as it would help organizations to keep track of all developments in a structured way, paving way for reusing the contents and code, help tracking usage, and scaling up, et cetera. If there is a standard procedure for all the coders to follow, it helps when the team collaborates, everybody is on the same page with no ambiguity.

Some of the things that need standardization is naming, including file naming, content naming, family naming, et cetera, so that the automation can be generalized and can use certain terms as standards. Coding, all the coding practices, such as variable names prefixed with data type denotation, commands and functions, enrollment accesses, et cetera. Commenting, commenting can be standardized so that the collaboration works best. Grouping, virtual code should be grouped together based on their purpose, behavior, et cetera. Repository, finally, the repository, this will help the organization to collaborate much effectively.

Now, what is automation? Automation is the process with which certain tasks or processes completed with minimal human intervention. From the tedious and boring repetitive task to time and brain power consuming complex processes, many things can be automated using computing or machine power, leaving our precious thinking and creative powers to do more creative and original tasks.

What is automation thinking, then? Automation thinking is the process of identifying or recognizing or executing opportunities in everyday life. We'll be talking only about our work life here as civil engineers and BIM practitioners.

From the early days of creating designs on paper, people found ingenious and innovative ways of reducing the human effort in creating designs. From the T-scale to the drafting machine to CAD was a huge step at that time. From the plane table to theodolites to total stations to laser scanners can be termed as the evolution of survey.

Civil engineering we're stuck in the 2D realm for long, and even the BIM initiative took more than a decade to be applied in more streams. As of now, the industry is trying to compensate for the delay with more rapid adoption of digital tools. And after the big BIM leap, we are positioned to take the next one, design and construction automation.

Let's face it. To survive and succeed, one must adapt to the changing landscape. Sticking onto the data technologies and techniques may buy some time, but not long. Change is already here.

What makes automation an essential skill is the rapid pace of changes happening all around us. Machines are touted to be self-reliant soon, self-driving cars are already on the street, a small drone that can detect obstacles and avoid it, balance itself, and deliver packages or capture images. It is a foregone conclusion, but if we do not update ourselves, we, along with the data technologies, will be obsolete soon.

Why automation? In the conservative crowd of civil engineers, I have heard this counterargument many times. Automation will result in job loss, the current method works fine for me, it's efficient, this is a small project or process, no time to do automation, engineering is a creative process, automation will kill the originality of the designs, I have my utilization targets, I don't simply care, the list goes on.

Is there a reason to believe in any of this? Let us go back in time. It was widely accepted that computers, when they were becoming mainstream, will result in lost jobs by the thousands. Yet, can you tell me how many people are jobless today because of computers? Yes, there must have been some jobs realigned, like that of a manual drafter to a CAD drafter. The lost jobs were of those who cannot make that transition.

Now in hindsight, imagine a world without computers. Yes the current method is good, it has been tried and tested, but that is precisely the reason to move to the next level, right? We're talking about anything between 20% to 80% time and effort reduction from those tried and tested methods, reduction of manual errors, reduction of repeated review processes, bringing in client and design confidence, a day to experiment mindset, and raising the efficiency of our people.

All this results in less stress, less review, less burning of the midnight oil, less graveyard shifts. And if that does not allow our people to work in a better atmosphere, complete more tasks in less time, leaving office on time, having a better work-life balance, then I do not know what will.

Now let's talk about how to approach automation, how to do automation. To create a proper automation that can be used by a wider audience, it would require proper planning and execution. Below are some of the major steps in developing. Ideate, create, test, debug, beta, and release.

Ideate, to find an opportunity and determine its viability takes years of experience, logical thinking, and reasoning. We start by asking these questions ourselves, is your team spending too much effort to accomplish a certain task? Design to model, Excel to model, mode to Excel, and design, model ordering, separately or in combination, are you spending too much effort? Is a certain repetitive task taking too much effort? Is a certain task result in human error most or many times?

Is a certain task seemingly impossible by manual means, but a requirement by the client would benefit all the stakeholders? Is problem-solving required, complex mathematical calculations, and is used multiple times? Is a certain task required to talk to multiple tools, database, tabulated data, web, et cetera? Do you see a pattern in a certain task, even if it is not very uniform, but the variables can be controlled to produce the varying results?

These are some of the example questions we should ask ourselves. If the answer to any of this is yes, then there is a potential to automate it.

Next up is brainstorming and collaboration. OK, we found some opportunity to automate. Now it is time to discuss that idea with your learned colleagues or community of practitioners members, discussing it with people whom you think can help, programmers, domain experts, tool experts, et cetera. This collaboration part is most important. Most times, automation beginners would think every opportunity is unique and it needs to be created.

Furthermore, there is the how do I learn if I don't write code part, right? While one need practice to learn, think about the possible pitfalls. Maybe a better or similar solution already available, maybe you spend too much time arranging it, maybe your knowledge level is not sufficient to crack the puzzle. It is better to seek opinion from experts, and with their help, you would learn more and the right way.

Worthiness, how do we determine the worthiness of a certain case? Again, this is where your automation community can come to your rescue. If immediate help is not available, ask the basic questions to yourself. Is this something we could use over and again, the usability part, or is it just a one time use? If it is a one time use, how much time to develop and what are the savings? Does this have wider applications?

Applicability of it, how much time would you require to create the automation, a task, even if it is tedious and boring, which normally takes six hours can be automated by spending four hours.

If the effort to automate it it is more than the actual time spent to do the task but it is something we use quite often and can result in considerable savings, sometimes we need to create complex and difficult to use automations to achieve a certain result, which otherwise, is not possible to produce. Automations, in that case, is the way to go.

Assess the possible reasons why you would automate it. Write down the pros and cons of it, weigh them. Who can do it? Do we have the capability? Collaborate with other teams to see the expertise to guide you is available. Go through the aforementioned questions, and if pros outweigh cons and we have the capability, then we have a go ahead.

In order to succeed in getting the automation approval, you will need to show numbers. Your line manager may not be as tech savvy as you are, and it is imperative to make sure that LM understands the whole process and its consequences. How much time we will save using this? How much quality improvements? Always play with the best and worst scenario so that you set the right expectation for your management. Do not exaggerate the result just because you are eager to do this. Err on the side of caution.

Now let's talk about geometrical versus logical thinking. As designers, we tend to think in geometrical terms. Simple example is when we create a rectangle, what we see is four lines connected at the ends at right angles. But for programming, we need to see the relation between coordinates to be able to draw that. That from this example here, the P1's y and P2's y is same. And P3's x and P2's x is same. So we need to derive patterns.

If the user giving two corners, opposite corners, then the programmer will derive the rest of the coordinates and easily can create this rectangle.

Another example is the creation of a polygon. The input will be in the-- is the center of the polygon, number of sides, and radius or diameter. The programmer will have to calculate the degree of each triangle connecting the sides to the center, calculate each point from the center, and finally, connect those points to form the polygon.

Now let's talk about term called API, application programming interface. What is it? Most times, there will be a function method within the programming language tool to do most common calculations if the tool is specifically built for the modeling platform. Examples are Dynamo, GC, or on a 2D level, LISP. But if the tool is designed not exclusively for the authoring platform, then we might need to write the calculations. Examples, VBA VB.NET, C#, Python, et cetera.

APIs are specifically designed for this. Application programming interface will enable the coding platform talk to the target application by exposing the target's internal functions to the code language. For example, dot NET is a core framework with VB.NET, C#, ASPX, F#, et cetera, as the supported code languages. If we have to create an AutoCAD plugin, out-of-the-box dot NET framework doesn't understand what AutoCAD is.

So we add the AutoCAD's API to the dot NET code by adding the DLL that comes with AutoCAD, which contains all the methods, functions, properties, et cetera of AutoCAD, explaining them to the coding or compiling language. Once that is hooked into the dot NET code platform, all those AutoCAD-specific functions, methods, and properties will be understood by the core platform.

Intelligent decision making is important in coding. If a certain value meets a criterion, then it should execute a branch of the code. And if it does not, a different one should be executed. A programmer must accommodate all scenarios for the code to cater to the need. The if/else condition is one of the most used branching logic in many languages. There are others, select case and so many other things, but if/else is the most used one. This allows the program to determine which block of code to be executed, depending on the condition. There are other methods as well, but the conditional branching logic needs to be well thought out.

Most programming tools use data types. One of the basic need is declaring the parameters or variables, but the proper data types. Data type is the type of a certain value or parameter with which the programmer intends to process. A program that calculates a person's age based on the date of birth must accept or convert the input into a date data type so that the programming language can process it accordingly. If it is not explicitly declared, the language will not be able to perform the intended operation.

Some of the basic data types I'm trying to explain here, this may vary from language to language, and I'm not including the AutoCAD specific or Civil 3D or Revit specific data types. This is a generally data types explanation.

Strings, all alphanumeric values accepted, but it cannot perform any mathematical operations. Integer, whole numbers ranging from minus 32,768 to 32,767. Long, whole numbers ranging from minus 2,147,483,648 to, you know, it's a huge number. Double or real, some programming languages call it double, some call it real, but any decimal numbers. Boolean have only two values, true or false or 0 or 1. Datetime, datetime is normally used in dot NET and VB. It accepts the date and time in one variable.

There are many others, but for basic operations, these are the most used.

Let's talk about events. Earlier programming languages, such as LISP, had a structural process. Meaning executing one code after the other one. With the advent of object oriented languages and event-driven running of blocks of code brought in more efficiency and ease of execution.

Some examples of an event is a click of a certain button on mouse or keyboard, click on a button or control on a user interface, change of status of a controller or object, addition, deletion, change status of an element. The programmer can write codes for each of this, resulting in execution of the associated code for that event and producing results.

Now we need to think about the end result. What are we trying to achieve with this program? And in that one, there are many ways of producing the outcome. Deciding the platform to do the automation is also really important, unless competing platform would result in not achieving the full capacity and mission.

On the other hand, using a complex platform for mundane tasks would result in unnecessary complexities and more effort for small outcomes. When we create a complex automation, which talk to the databases, design tools, authoring tools, et cetera, and the end user may not have access to some of these target tools, it would be good to create a standalone, which means an executable or any standalone that users can click on it and then bring up the UI, and then start processing.

On the other hand, small tool-specific automations can be built within the tool, even if it require minimal interaction with other tools. Such as, you wanted to process some objects or lines and circles in AutoCAD and export the whole thing into an Excel file. We can use AutoCAD's APIs and use one of the programming languages, Dynamo or VB.NET or C#, and then write something specifically for AutoCAD. In that case, you wouldn't be needing an executable.

To decide which platform is required, the programmer should have a clear vision of the result, interaction sequences, capability of the tool, et cetera. If the programmer is in doubt, consult experts, COP members, anybody.

Now we move on to the Create phase, that we have done our ideation, we have arrived at all sorts of conclusions, and we decided which platform to use. Now we move on to the Create stage, and we have everything needed to produce the application, and we going to start putting together the code now.

There are a few more steps if you want to build a very good, sustainable code. Scalability is one of them. Perhaps the solution that you think of is a unique one, perhaps not. If it has a wider application, or if you think it can have a wider application, then we must think and provision for that.

For example, I have got a couple of screenshots here from a solution that I built some time ago. You're developing a standardization script for one of your projects. You need to check layer names, check text style names, and compare it against a standard document, or an Excel sheet, or some set of rules.

Since this is for a particular project, you can either hard code these values inside your code and check it against the layer names. Like a layer name should be layer1. If it is layer-1 or if it is layer 1, then change that to layer1.

That layer1 name can be hard coded within the code. But a more appropriate approach would be creating something like a configuration file, like the screenshot that you're seeing on the right. In that one, I wrote some key elements like, OK, I'm going to check the layer here. And I'm going to check-- if there is a layer named $0$Defpoints, then that layer's content should be transferred to a layer called Defpoints. Or if it is possible to rename this layer, do that.

If I'm working on the Tstyle, text style, then I need to check the font name. If a font name called SIMPLEX8 is the font of that particular style, then change that style to ISOCP.SHX.. So this approach will help you scale up the whole application. That can be used in any projects, not only for that particular project.

Then we need to think about decomposition. At first glance, most problems would look intimidating. You know, you have a problem statement, and you think that, oh, it's too complex for me. It raises doubts in your mind-- too many complexities, et cetera.

To effectively manage coding or graphing in terms of [INAUDIBLE] programming, we need to break down these challenges into simpler, smaller chunks of code or problems. Large, complicated problem can be broken down into multiple pieces. And with that approach, multiple teams can work on that problem and create a solution easier and in the most effective way.

Once we have the smaller pieces of problem, preparing a high-level flow chart would be of great help. Relying on a person's memory, the developer's memory for how the data should flow, what branching logic we are thinking of, et cetera would not be an ideal way to work on a complex problem. Ideally, the flow chart should consist of the first input, validation of data, a decision point and Yes/No branching, processing, repeat of validation and decision until we have the desired outcome.

Let's take the above mentioned example of [INAUDIBLE] standard processing. Some solutions require a friendly user interface to accept user input, some not. In this case, we need one since we must provide the user a way to define values and select drawings.

And in that program, we can check whether the user has selected drawings. If the drawings have been selected, then move to the next level, that is opening each of the drawings. And then the next line is reading the layer table records or the layers available within that drawing.

And then if the layer name of the first layer name matches with the configuration file's entry, then do something. If it doesn't match, move to the next one, move to the next one-- something like that. So that is the flow chart. That's where the flowchart will help you.

And while creating the program, capturing and trapping errors is a very integral part of programming. Without proper error handling, the program could crash at the smallest error and can corrupt databases. Or it simply doesn't work the way we intend it to.

There are two ways of looking at trapping the error or capturing the error. One can pre-empt it. Pre-empt an error from happening by evaluating all the incoming values before it is being processed. This method is necessary to ensure the smooth flow of execution as well as making sure calculation results are right and the resulting value is as per the expectation.

But how much ever we try, there are certain errors that cannot be pre-empted and thus have to be captured as it happens in the run time. If error trapping is not done properly, then the values that the programmer changes within the environment will cause problems for the user even after the execution of the code is over. Therefore, making note of the values the program is changing during the run time, and in case of an error, set them back to what it was before the run.

And this will happen. When you trap an error, there are two conditions. One is smooth run. And if an error happens, there is another block of code that will get executed.

In that block of code, you will have to change back all the environment values. In terms of AutoCAD, it will be like system variables and registry entries that you are changing. The programmer has to change that back to what it was. Otherwise, the user will have problems.

And the last one is, don't create a full-fledged project first up. As a prototype, a minimum viable product, we could produce a working skeletal frame. It is to see the flaws in the logic, the process, and structure.

At this stage, we don't need to create fancy user interface elements. Some of the elements we might be hard coding inside the code, even though we have provisioned for reading it from the configuration file, et cetera. But this approach will help us not wasting any time in a problematic approach. And if something is wrong, we could go back and easily change that. The idea is to have a testing copy with all the necessary inputs and outputs, and make sure this will work.

Now we come to the last part of the coding-- testing and debugging. In order to produce a working solution that works seamlessly, it needs to be tested vigorously. After completion of the code, we don't really release that to the general public. The first phase of testing is done by the developer himself or herself. It involves running the solution with all permissible range of parameters and systems.

For example, a solution that takes a design input and alters the geometry of a Revit object needs to be tested with the whole range of parameters supplied to it. Starting from whatever lowest level to the highest level, it needs to be tested within the solution acceptable ranges and work in an expected behavior. The geometry form is right, whether it works with intended versions, like all the versions of Revit, whatever you have intended.

And how is the memory usage? Are we using a lot of memory that slows down all the other packages? Are we running an indefinite loop? Checking the crashing, error trapping, almost all those things.

Once the developer is satisfied with the output and process, the tool can be released for beta testing. Beta and release-- in the beta version, only a limited group of people who have domain expertise will test the tool. And this is to ensure that it works on all systems. Once the testing team is satisfied and gives the report that, OK, this is all working fine, then you can release your tool to the larger public. With that, I would like to demonstrate a few of the things that we discussed here in a small video.

[VIDEO PLAYBACK]

- In this video, I'm going to demonstrate some of the basic fundamentals that we've discussed during our session. This is a small automation I have created for Civil 3D. It primarily deals with the property data sets and how to extract the data available on parent Civil 3D objects to a solid once the solid is generated from the Civil 3D element.

It has a way of creating solids from the parent element, whether you want to put it in a certain layer and then copy the parent property data set to the newly created object. Also, it can take object data from the element and then convert that into a property data set definition and attach that same data set to the object.

The whole thing is designed in WPF, which gives us a lot of flexibility in organizing these controls. And then all these elements can be controlled using the XML format below here. Everything, every control on this form is controlled by these codes.

We're moving over to the Subfunctions tab, where I keep all my reusable content. As you can see, I have created a function for attaching Psets. And every time I want to call this, I will use these as my parameters and then call that in my code, so that this particular block will get executed. There are many codes I have created like this.

Using these functions, I can write some of the key elements of this program to the registry. And using this function, I can read a certain value from that registry entry so that every time the user is firing up the window, some of the elements are stored there. And it will be read from there just to keep the previous configuration intact.

In this example, you can see that I use a lot of try-and-catch blocks within the program in order to trap errors that are unforeseen. So what this does is, if there is an error while executing this block of code, it will go here and then show a message to the user that there is a problem with this particular block of code, and compiler or the system is not able to run that code. We can terminate the program after that. Or we can go to the next block of code based on the severity of the error. And that is a programmer's choice how to continue after the error happened.

The API's play a vital part in opening up the host applications objects and properties to the coding platform. I'm going to demonstrate how this is done. You can see that I have called a lot of resources or references added to the program. This is the API Dynamic Link Libraries that I have linked to this platform, so that my coding platform understands what I'm talking about.

If you look at this one, ACBaseMGD, this is the DLL that contains all the codes and functions that will open up the AutoCAD or Civil 3D's base functions to the coding platform. This is the DB-managed one, which means all the database properties, objects, everything will be managed by this one. You can see pressure pipe managed. All the functions and parameters related to a pressure pipe network will be included in this one.

Now let's come here. A database is a data type that AutoCAD understands. In normal parlance, a database will be sequential, language database, or relational database. But here, a database is a drawing file.

Now, this comes from one of these references. And I have to use this import statement to make this coding platform understand, OK, I'm going to call in all the functions from this particular file. Now you can see that there is no error marker here.

I'm going to comment this particular line out. And immediately you will see all the databases and transactions and then opening an object for write, everything is showing as an error. Which means this particular word, or a transaction, or a database, this thing is not understood by the coding platform unless you have this reference in place. As soon as I remove the comment there, you can see that all these things are in color again and is recognized by the coding platform.

All these elements in this drawing are brought in from a GIS package. And it has got a lot of object data attached to it. But there is no property data set definitions present in this drawing. And nothing is attached to the objects as well.

We're going to bring up the program that we've developed. And I'm going to tell the system that the source of my property data definition is from object data. And I'm going to Create. You can see that there is nothing here. And there are no property data set definitions in the system.

I'm going to click on Create and select one object so that I can create the structure of the property data set. You can see all that information from that particular object is converted into property data set definition. We're going to attach the property data set to these objects.

Just click on that one. And there is an option-- One to One or One to All, One to One meaning every object. It will read every object's object data and then attach those information to single elements. That means One to One. And One to All is just a base object's data will be attached to all the objects selected.

We will stick with One to One. And this is the property set definition we have created using this tool. I'm just going to click on Attach. And I'm going to select all these tools. And it says all the property data set has been completed.

Now, if you go here and check this property data set, you can see that everything is attached. You look at the unique ID that is FeedID, 205. And if you check that, it is 205 here.

We go to another object and check. It's 360. And you can see that it's 360. That means every object's object data is converted into a property data set record and attached to the same object.

The next feature I wanted to demonstrate is creating a new property data set definition using the tool, attaching that to the pressure pipes here, and then converting that pressure pipes into solids, and copying that information from the pipe to the solid. Let's bring up the interface. I'm going to pick a predefined configuration file.

So I have saved it as Civil 3D pressure pipe parameters configuration. And you can see that I have added a custom field, which is a string. And I just gave it my name.

And we've got so many other properties directly taken from the pressure pipe properties-- Name, Description, Style. All these things are available in the property of that pipe. Or it is embedded data by Civil 3D.

Now we're going to create this one from the configuration. And we will give it a name called AU2021-Sample-Pressure. And we just click on Create PD Set. Now the definition has been created.

Now what we will do is, we will check this Create Solid and assign a layer, any layer for that newly created solids. And I just wanted to click on the Copy Parent Data so that whatever data we are extracting from the Civil 3D element will be attached as a property data set to that particular Civil 3D element and the newly created solid. So here we go. I'm just saying Attach, and I'm going to click one, two, and three pipes. And it says that it's been completed.

And just a note that to convert pressure pipe networks into solid, we need to change the viewpoint in Civil 3D, and it has to be in 3D. When we started, it was in 2D. So the program will detect-- if it is in 2D viewpoint, then it will automatically change the axes and put it in 3D. Now you can see that we've got a green color element here, which is a 3D solid. And it has got all the elements that we prescribed in the configuration file and its associated values.

With that, my presentation is over. I would like to thank everyone of you for attending this session. I hope it was value for you. And thank you very much for attending my session. Have a great day.