Control and Program a Real Robot by Manipulating Its 3D Model Using Forge

PRESENTER: And so what are the implications of this kind of device, it's two or three degrees kind of plus or minus. Well, not very great. What about the delay of 30 minutes even? Not such a greater problem.

Now, put the same example into a context of IIoT, our industrial one. When you have the assembly line and you have robotics I'm doing different things. A slight variation of, I don't know, 0.1 degree could spoil the entire assembly line or entire production.

Or the even greater example is the medical equipment. So now I mention that you have an apparatus that actually is measuring the brain temperature injecting some medicals based on these data. 0.01 degree it's kind of the limit between life and death. So this is why I keep warning everyone that IoT is not IoT. And what I'm presenting here is IoT, it's not industrial one.

In the end then we will see if we have time what are the implications of the problems. Because everyone is like, security of course. Security is everywhere the problem, but it's not only that. So this is the main concern.

And, to be sincere, I had huge concern in coming in front of you with this kind of [INAUDIBLE] robot and this idea. But I was encouraged by a lady from Caterpillar who was presenting in an Automation Conference last summer. She said that, well, we engineers do great things. But we still have problems with user interface.

Our users their interface is stuck in 90s. And in order to kind of solve this problem is, well, we have to attract those kids from gaming industries. They know everything about 3D, human interaction, about the presentation, menus, and so forth.

So this kind of encouraged me. And we will discuss about that. But the oldest question is how 3D can be useful. To be sincere, if I will ask any engineer who has 20, 30, 40, 50 years of experience in automation or other fields of engineering they will say that-- they will tell me that, kid, there is no point in using 3D. We've been doing this for ages and without any 3D.

How about at the same time the number of these devices is increasing kind of exponentially I might say. So you have-- you need a way to-- if not to control-- at least to diagnose them, or remotely diagnose them. And there is a saying that if a picture is worth a thousand words, how many words could worth a 3D interaction of this thing? After all, we are 90% visual beings. But even from this information that we are visually receiving there are different quality and kind of acceptance of it.

So this gauges. I give-- I keep giving an example of this gauges or an analog clock versus a watch-- versus a digital one. You may see that they are presenting almost the same information. But how much time you need to understand what is showing the one versus another one? If you look at the analog one, just a glimpse, it's already enough. OK, it's zero.

The electronic one, well, you have to spend time to read it and make sure that you do read it. Well it's 1031 or it's 103.1. You have no idea. So see, these kind of things are starting to get into-- kind of into troubles. Even if you have these kind of visuals things there still needs to be-- to have a sort of quality.

Another example is-- OK, so you imagine that I modeled this watch. Yes, I want to share with you. I cannot expect that you have this kind of application. So the best idea is kind of embed this thing into a 3D environment-- or even better the web 3D environment-- and share with you the link.

But even there you might see that, OK-- even here you might see that, OK, I have this kind of clock and showing the time and everything. But there is a problem. So I don't see the-- where these kind of things-- the arms, the seconds, and everything. So would be nice to put in a more kind of visually appealing thing, just for illustrations?

So, you see, a very simple trick with linking this with the current time and everything. It's already showing you something. And if you start by adding some interactions just to illustrate what it's capable of and so on and so forth, it's already give you an impression of what to expect from this thing.

So why I'm showing you that? Because, you see, if I were just showing you the picture of that clock you would not get this amount of information that-- by having this 3D interaction. And this is just a small example of having the 3D model that was used maybe for fabrications for different things. But I just took the very same model and put it on the web. And to either attach some JavaScript things-- and you already have more information.

And even better to illustrate kind of the implications I'm using this pathetic robot that I just purchased on-- I don't know what's the site-- AliExpress or something. But now let's look at the information that presented on this website. OK, so in our menu it says, well, maybe it's plastic with aluminum finish.

Well, what kind of servers it has? Has plastic server I mean, with a plastic CAD? I have no idea. What's the size? I have no idea.

So you can compensate this information by going-- by showing a lot of images or kind of usually in the form of [INAUDIBLE] or different thing. But wouldn't it be nice to kind of have a 3D model of it that I can visualize and maybe even manipulate? So, of course, I order it. And it came in such way. OK, very nice. No assembly instruction, no nothing.

And I had to look very, very closely at all these kind of pictures and everything. But in the end, I took the caliper, measured the things, and assembled this 3D model. So could it be-- couldn't I replaced this model on this shop page with something that is 3D that I can isolate, see how it was assembled, maybe even create an animation about the assembly instructions? Because, believe me, this is a pain. A very huge pain.

I assembled and it resembled it several times, just because they were all kind of prematurely assembled, some parts. So, you see, it's very nice to have these kind of things. So first of all, why we need in this context the 3D model is that we can embedded it into a [INAUDIBLE] and show the different things like the material used for the server [INAUDIBLE], or what is the material of this one, different components, so on and so forth.

And if you ask me, well, maybe it because of the use case of this [INAUDIBLE] or a website or something. No, this is even on a professional level. So I especially choose the American one, Universal Robotics. I had the chance to touch it at the Automation Conference. Very nice looking.

It's like Apple made, and it still has a problem. You see, aluminum and [INAUDIBLE]. What's that? What is aluminum? Is this plastic or is it aluminum, painted aluminum? Is this rubber or plastic or something? I have no idea.

So what I did is that usually these kind of robot providers they present you the 3D models. What I did is just take this model, assigned it materials. Put it in and kind of environment, and you already can of query different things just kind of by clicking and pointing and rotating and exploding and everything. And this is done basically automatically. We will see how-- what effort you will have to undertake in order to get to these things.

But in this case, it's a ready model, 3D model. Well, in my case I had to model it. And through all this modeling procedure I stumbled upon some things that are very kind of interesting, one that I would like to share with you, even if it's somehow not related to this thing, but it still worked.

And for my illustration I will be using Fusion 360. Not because I like it very much, but it's because the Fusion 360 technically encompass every technology that Autodesk is exposing to public through-- as is the case-- that we called Forge. And this Fusion, by the way, is a very good kind of indicator of what to expect from Forge.

So, for instance, in Forge you don't have yet re-entering in the Cloud. But here you have the option of re-entering the Cloud. So it means, OK, you should expect something like this. Simulations, OK, not yet ready but it's there. Because, again, Forge by itself is just exposing to the public the general thing. And the Fusion360 is a very good example where this piece of Forge are used.

So I'm merely using it for illustration. But with Forge you can embed this in [INAUDIBLE] or SolidWorks or whatever internal even thing that you are-- you'd like. So one thing-- apart from modeling and different things-- one thing that I like the most is versioning, so a sort of version control system. Those who use it with the code development and everything they kind of sort of know. But in engineering and in cart design this is in

When you actually keep saving different versions of the same thing just by clicking save. OK, if you're saving it will create a new version. So you always come a row back or come back. And there is-- there are even some attempts to create branching. So if you want to experiment some ideas without deleting everything.

The classic way is just make the copy of the folder. But you see in several iterations you will end up with gigs of files with the same project and you have no idea where is where. And who can notice that also it's using the versioning for the xrefs, which is also convenient. And we'll see along the way that it notifies you that this one is using the version 13. But technically there is a newer version. So would you like to update?

And this is the code snippet-- kind of this is the formatting-- but this is the code snippet needed just to get the very same information about the versions, about the time, what's done and everything. And this is just the kind of extractive information. But there is a lot that is coming just by query through code. This very, well, artifact-- I called it an artifact or item.

And at the same time, it's not just that. It's also kind of encompassing everything like foldering, creating carts projects. So if you want to create something like-- OK, I want a project. All that is related to electricity should be handled only by those people with that electric responsible. So I gave them rights only to these things.

All what is handled by- I don't know-- artists, they can have only the read-only and put it an artist context. And so on and so forth. So you have this kind of access control. And at the same time, as I say, the versioning. And any time when you are putting the same file over and over it just will create versions. That, with automatic kind of rollbacks.

This is-- again, the API that allows you to do kind of the same thing as the Fusion360 does within its own infrastructure. But also it's interesting that, well, we have versions. Those familiar with the code versioning will say, OK, can I see the difference? Yes, you can.

Technically if you let's say mix the thing like data management with the viewer, you basically have almost all the information needed to create this kind of [INAUDIBLE]. What was added was modified and so on and so forth. We will see illustration that it's our only-- also showing you kind of the visually what this difference.

But, again, this is done-- to reach this stage you have to pass through bringing into this viewer. And this stage is done by the model directive API. That will prepare you the model for the viewer. Fusion360 is doing this kind of automatically. But at the same time, there is no problem to use this API to do it by itself. So you can extract its support.

By the way, it supports about 60 formats. So it's not only Autodesk one. But it allows you to do this kind of file translations, created thumbnails, so on and so forth. So now imagine that you have of thousands of assets. And you say, OK, let's prepare them every one for 3D printing. So we need STL.

OK, so just call the API. Show the folder. Get to kind of the results. And along the way say, OK, along the way prepare me the thumbnails. And maybe prepare for viewers so I can visualize or pre-visualize them in the browser.

And, again, as I mentioned Fusion, by itself, is using these. So when you are visualizing your model in this kind of Fusion hub, you can extract it or save it in a certain format. So when you will invoke this, under the hood it will invoke this server. Again, you can embed this into your service without any problem.

So in order to take this from your application or a cart application into something that it's viewable in just a browser-- and anybody use browser, so there is no problem with sharing or everything-- just a couple lines of code. And mostly here it's just because of formatting. And notice here that I'm invoking the formats that I would like. And the [INAUDIBLE] by itself is the format used by the viewer.

Why it was need of a special format? Because-- I like this joke-- like, well, we have 50 formats. Yes, we have all these kind of zoo of formats. Let's create another format that encompass everyone. So now we have 51 formats, and so on and so forth.

So this is needed. Because if you are using the OBG, what kind of information do you have in OBG? Well, vertices, faces, normals, and so on and so forth. Do you have the hierarchy of the objects, or basically have you components? No, you don't. So there are a lot of informations to be translated.

And every application-- be it CUTTY, Solid Works, Fusion, [INAUDIBLE] or whatever-- are using their own format in order to store this kind of metadata, we call it. And the model derivative by itself-- when creating this size [? vf-- ?] it allows you to put these kind of metadata along the objects. So we'll see how it is useful. And this is why actually it's possible, after having this model, just to click on components and see the type of materials and everything. This is the information that was put there by the designer.

And also if you have a house and you are clicking on the window, you're not presented with the-- let's say it's two or three angles. But you're presented that this is a window. Maybe about apart from this size and everything, maybe with a link to the supplier or what information you would like.

But notice here that after sending this job you are receiving this URN. URN is technically your link to the resource. And it's in such a format because it has to be embedded into something, but also with the control of access. So in order to bring it to the viewer, this is the code snippet. You put the URN here. And this is that-- kind of the talk in giving you access.

So now if you-- I will show you some models in the browser and whatever. You will see that, OK, everyone can see them, but nobody can download them. Why is that? Because it has-- this token that I put here is just the one that allows you to visualize this model.

Security measures-- so technically it will give you something like a default viewer with every controls, these kind of rotations, and a toolbar. This is why I say all is default. Of course, you can sort of hide it or even change it. And there is what I call orthodox way of changing it through extensions.

And every element that you can see here is basically an extension. And the source code for those are kind of exposed to public. So we can use it for your own kind of-- to modify or to adapt to your own needs. But also on the website there is nice tutorial how to write this extension.

However, I was thinking beyond this one, because I was interested in sort of animations. Yes, Fusion [INAUDIBLE] by default, it's just to illustrate the components. But it would be interesting to create these kind of elements that's showing the mechanism, how it works or different things. Kind of maybe bring some interactions.

So I was looking at an orthodox way, and there is one. Because technically the Fusion viewer is based on 3GS, a very known [? collaborate ?] that, of course, is based on WebGL But this 3GS is very kind of flexible.

And since-- you can use the Fusion viewer kind of tools and functions. But you can go even lower to the core of it and implement things like-- well, this is a very simple graphical user interface created in couple of lines. The most problem is that-- how to command this or how to connect these things to this model. Because, by design, it was like this.

And I had a couple of tutorials explaining the idea behind it. Well, you can do the hard way and multics-- metrics multiplication and do those sort of things. Or you can rely on 3GS and child/ parent relationships and so on and so forth. I highly recommend to look at the blog posts. Because you will see here a lot of orthodox and unorthodox way of doing-- kind of solving this stuff.

But why we are here is about connecting these robot, OK? Yes, we have the 3D model. We have the kind of physical thing. And at least we want to know, let's say, its position. Or maybe not to control, it could be used for diagnostics.

Maybe I have some sensors, temperature rotation on something like this, and I want to expose this. Well, it's very simple. It's under the hood. It's just JavaScript. So everybody is using now JavaScript.

Even the-- you find more C++ developer and I had JavaScript. I have to kind of say that, yes, it's very simple, flexible, and doing nice things. But remember that box that comes with pieces and everything? There is no microcontroller there, so I have to choose one.

And this is my small collection. I like them all, except this one, the Intel Edison. I have no idea why, but I hate him. And, however, I stumbled upon this small device, the small one.

It's a very nice one. And the cool thing is it's coming with it's own sort of infrastructure. Allowing this kind of control, again, through the Cloud of devices. This Cloud thing reminds me of that joke, like what is the difference between mainframe and Cloud? Well, before we were paying bills to IBM. Now we are paying to Amazon.

In the end, it's the same thing, just kind of changing the name. Anyway, so a sort of infrastructure that you don't have control on it, devices, and the application. Now, again, remember that I'm telling you about IoT. I'm not telling you about IIoT.

Because from IIoT perspective this is a nonsense. I could never allow anyone besides me to control this kind of link. But it is very nice. And I will show you just a small illustration. But before that, just do a small recapitulation.

So as I mentioned, we have these kind of-- the model we added some interaction here showing the time, and also checking that I have five minutes left. This is the Universal Robotics, the model that I had. So I can query here different things, and even see the structure. Kind of a dual sort of explode and everything.

This is not my doing. This is automatically taken care of by the Forge Viewer, because of those-- that SVF file. SVF file contains that hierarchy. At the same time this is kind of the new thing that comes from the Fusion360. And I see-- you see this is kind of a thing coming by default, but later we'll see how it looks the one that I created, not in context of Fusion360, but just a part of it or independent.

As I mentioned about the versioning and also the referencing. So you see I have a new version of this available. So I could update. So if we look at this 3D model, I want here the components that I sort of updated. So I can just update them. But they are taken from here.

The versioning by itself, we have these kind of changes. So what changed in this case? Well, in this case it was moved. I see. OK, so the versions-- the difference between these-- I see now. So you see, this is the visual aspect.

Again, the engineers will say that, well, there is much need of it. Well, yes, when dealing with one model, yes. But if you have a sort of three versions and you want to compare them, it's very handy. And, again, this is done in-- this come by default in Fusion360. But we are not interested in here, in this one. We're interested in creating our own.

So our own come with different things like-- OK, this is the information. This is the metadata. And, believe me, I put in a lot of effort in order to make it one-to-one. Sort of every nut and bolt is modeled here. Well, except the internals of the servers.

And you have tools like, OK, so what is the size of this one? Let's use some measure tools. From this point to the top, let's say. OK, so I have these kind of things. Yeah, very handy.

Again, these extensions that you have the code so you can add it to or modify for your own purposes. But I also interested in creating or adding some controls to see how it works. OK, maybe to illustrate how-- what is the degree of freedom for my robots. What it can reach. What it can show. What it can-- how it can bend or what are the limitations and so on and so forth.

And, of course, the main interesting thing is that can I link this to the robot? And, you see, I'm refreshing this page. But this page is adjusting. And the model is querying sort of the robot. Well, here we have a defect. So we have-- something died.

So you see, I'm controlling it. Useless, maybe. But apart from that it's very fun and everything, it's also can serve as a way of programming it. Let's send it-- OK, I'm sending these instructions to program it. What is-- basically what is animation? It's just a sequence of keys.

So what I have to do is add a library that will handle these kind of keys. So at this stage, I would like to make this. OK, I'd like this to be sort of in this position. And this one to be in this position. OK. So let me put this in [INAUDIBLE].

So if I'm having this sort of animation, I already can check it. Well, not very impressive. After that we will see one very impressive. But these keys can be saved as JSON. And this JSON can be interpreted in the code. So there are several approaches to do that.

When you are controlling kind of the device or you want to program, there are even hard control when you are programming the microcontroller and attaching it to a device. Or you can do a sort of like old G code-driven apparatus are doing, or devices are doing. It's basically the thing with having this logic already created. And the memory region that you are putting the instruction is just reading these instructions. So these are the connections.

So in the end, this is just kind of the position that you have in. And in case of this robot I have very huge problems, because it's off this accuracy. And the second is because-- how to say-- well, I might want some very fast animations. But at the same time, there is a problem with how frequent it can do.

So, for instance, a rotation of 180 degrees of the server is taking under-- well, about 360 milliseconds. And this is kind of creating huge problems. I have some tutorials at a smaller case showing with the server how to control, and how to investigate, and how to change, and everything created from the scratch in explaining this. Because we certainly are out of time.

But recent-- well, this robot has already [INAUDIBLE]. But I had fun time in creating something like-- well, something like this.

[MUSIC - RAMMSTEIN, "DU HAST"]

Again, this--

[APPLAUSE]

This link will be available. But I hadn't planned to sync them then sync all this, too. Oh my god, this one broke. Well, because it's already had under its weight as well something like this. But it would be nice to have tough servers. I was in search of several of them, and everyone-- well, at this size you don't have the precision and you don't have the control.

But the idea was quite nice. And you see, again, I have everything is kind of keys. Of course, this Rammstein and music is very predictable. So I have sort of patterns that we're kind of created automatically with Python, or filled with Python. So I don't have to do a lot of kind of manual keying.

But in the end you end up with I think it's 248 keys. And you have six severs here. So sort of for the amount of work. But, again this is a way of programming it.

Because what are the ways currently? The first is writing the code. The second is putting in a learning mode and manipulating. And after that is just repeating these things. And this is sort of the third way, that just by animating the 3D model you extract the keys and put this or convert this into instructions.

Again, in that tutorial for this server I'm showing kind of the idea behind it in more details. And another aspect that I kind of-- one minute-- is that you see here you have two micro-controllers. One controlling the upper part, and this controlling the bottom part.

From a hardware perspective, it has no idea what it is doing. But from a software perspective, you have just one object. So you see, this is a way of saying that-- well, an excellent example of collaborative robotics. Two pieces working together to achieve a task. That's it. Thank you for your attention.

[APPLAUSE]