Large-Scale Data-Set Transfer into BIM 360 Docs with AWS Snowball and S3 Service

MARCO RAMOLLA: Yeah, thanks for joining. My session is in competition mode to the main stage, you know. Thanks for joining, that you're here. It's your last chance to exit if you want to go to the main stage. But this session will be recorded and that's good because it's a very-- I think it's an important topic to load mass data, large-scale data-set into Docs and a lot of customers struggle with this, and to continue to move forward using Docs because they have a lot of legacy data they want to migrate.

And this is what I want to talk about in this hour session, here. If you have a question, I think we are a small group of people. You can raise it directly and we will try to work directly on it.

So my name is Marco Ramolla. I'm working for Autodesk Consulting. I'm now with the company since 13 years. I'm from Germany. I'm also responsible for BIM 360 service offerings or packaged service offering we are creating. Maybe you heard about we have some accelerators in the program, some standard offerings to deliver 360 services. Yeah, I think I can name me as a BIM 360 expert. I'm now with the tool since two and a half years.

Before, I worked a long time in the data management business, Autodesk Vault, AC business manufacturing, basically. So a lot of experience also with large-scale data sets because my background is a geoinformation programmer and we had worked with a lot of data here.

Learning objectives. You know what we will talk about today, how we can import large-scale data sets into BIM 360. Docs, we will not go through this.

Quick view to the agenda. First, as you read, maybe, the class description, I will share some project experiences I made with a 2.3 terabyte project we have to migrate into BIM360. We talk about the process in general, how I did this, what are the components, how to run the process in detail. So if you want to do it on your own, I will a step by step introduction here. Then, I will give you a quick view about performance, some summary from the performance of the specific project, and at the end, recommendations. What are the limitations that we have to be careful about?

Good. The project-- quick view on this. The customer I'm talking about had a plan to migrate 2.3 terabytes of data into BIM 360 Docs. The decision for Docs was made. They were looking for a common data environment where they can store anything. So the legacy data as well as the design data, and for this, they had-- you have a quick view here what kind of files this set of 2.3 terabytes contains. And the idea was to have a one to one folder migration. So no mapping was required between the folders.

Important point. It's a German based customer, so they have regulations to store the data in a European region. So that was their requirement, not outside of EU. So BIM 360 EU environment is a must. And also for all the migration approach we will use, that should not leave the European region.

They have a low, medium internet connection from their offices, so an upload-- manually uploading to Docs was not an option. And they cannot set a person 10 days to drag and drop the data into dark sets. That's not ideal. And important point, they want to manage control and log and everything. It wants to monitor everything on their own. They want to have a detailed log functionality about success or failure. And we got a duration of a month, so the data should be ready after a month after project start, because they want to use BIM 360 Docs quickly. It's there and they want to use it.

So based on this, I defined the success criterias for the solution. It should be fast. Remove the limitation of the bandwidth from the offices. It should be scalable, so large scale data sets. 2.3 is only an example. They have much other projects in the pipe where they have to manage more data on it. No limitation for files and file size. It should be reuseable, as mentioned. They have more projects in the pipe they want to migrate into it. And no custom solution, so we should try to use as much standard products or services as possible. So that was the four success criterias.

And based on this, I defined a process how we can do this. And there are two major components here. I will talk about more data, which is the AWS Snowball technology and Cloudsfer migration. So let's have a closer look, step by step.

That's the current situation. They have on-premise file server. All the data are in the current system. Large-scale. There are static data because legacy, as I said, but there is no limitation so it can do it also with design data or with data which are changed very often.

An important point is pre processing. So they have to clean up a little bit of data set. They sort, label the folders, they clean up the folder structure before. The second step is the AWS Snowball Upload Technology. I will talk about this in a second. This is our bridge from on-premise to cloud so that you don't need your own bandwidth to upload the data. So you copy the data to a device, send to Amazon, they will load into the S3 data bucket data storage. The good thing here with Amazon S3 is it supports European storage as well. So they have data center in Europe. The data will stay only in Europe. That's really important. Everything is encrypted, also for Snowball, also for Amazon. So the storage itself, the transit to storage, it's encrypted. So high security at the end. And when the data are in S3, they are accessible by many ways. So they have some browsers, you have the AWS Console where you can access the data, or other tools.

And another important step is the Cloudsfer tool we're using. It's a service. I will come back also to this in a minute. It's the migration from the Amazon S3 into Docs. It's a service. You pay per data volume here. We'll talk about more. And the service also it's possible to run in European region, so we also don't break the rule that they have the European region. And it's really easy to use, but that's for the same process. The whole process is really easy to use, easy to configure. It's not really rocket science, honestly.

And yeah, last but not least, Docs at our solution to store all the data as a common data environment. No difference between legacy data, design data. And you can do here the post-processing as well. So we can move the folder, the files. You can set up permissions, securities, whatever.

Components. I will only talk about Snowball and Cloudsfer. I assume you're familiar maybe with S3, so we're not going to talk so much about this. So what is Amazon Snowball? What it is-- maybe you never heard about it-- it's a data transport solution from Amazon to upload mass data, large scale data sets into Amazon S3. You can what you see here is a video, official video from Amazon. It's available on YouTube, so if you want to see it again, then.

What you will get is this kind of gray box you see on the right corner there above. This contains hard drives. You copy the data from your own network on this device, you create a job, you get this device, you copy the data, you send it back to Amazon Data Center. Normally, you send it back to a partner of Amazon, so not directly to Amazon. And then the data will be imported as a job into your S3 data bucket.

The whole solution is scalable. It can manage, as I said, large scale data set. At the moment, you have two options, 80 terabytes of data and 100 terabytes devices are available, you can order. It's simple, it's compatible, it's easy to connect to your network. And, important point also, bidirectional. In my example, it's only a one way into Docs, but you can use it also for download or any other. It's only the way from on-premise, to Cloud, to S3, not into Docs at this moment. But again, bi-directional.

I think it's really low pricing. It costs approximately $200 per 80 terabyte job. This cost does not include the storage cost, I have to say. I mean, if you have access to AWS Console, you can calculate or see the cost of S3. There's a good cost description.

So as it was a quick view on Snowball, I think we have later a Q&A. If you have questions now, raise your hand. We can go through. But next one. Cloudsfer is the second important component. It's a cloud based migration service from a company called Tsunami. The application is available in the BIM 360 app store. At the moment, the app store, maybe you're aware, is only available in the US environment. For European environment, you need to add a Forge custom application, so please contact Tsunami and ask for the key to add, because maybe they change. I have the key, but will not share because I don't know maybe if they changed. So if you need a key to add this application to a European BIM 360 account, then please contact Cloudsfer.

Cloudsfer supports more than 20 providers, so it's not limited to BIM 360. So they have migration from S3 to picture, services to Facebook. So a lot of solutions, cloud solutions, they support. It has a browser based UI, so the web service. You don't need any installation on a device. It's really easy to use. It's well explained where you can plan your job, your migration job, you can schedule, you can maintain, you can monitor logs, everything within UI.

The data are encrypted during the process, and there is no bandwidth required during the migration, because it's from cloud to cloud. There are limitations for performance I will talk about later, but in general the application doesn't need any bandwidth from your network. You can create your schedule, a scheduled migration, if you want. And this supports as well then data migration.

So based on the schedule, for every eight hours for example if you want to do it, you can check for changes, check for modification date. And that's also helpful here using filtering, or you can filter specific file types, substrings of your data, which are in the source there. Yeah. And there are three additions from the business model point of view. It's a free personal business. I would talk about only the business one, and at the moment the costs are around 1 terabyte for $250 to transfer data. Yeah. And this includes everything.

Good. How to run the process? There are some prerequisites you need to consider. The first one is, speak with someone from Autodesk if you plan to import large scale data set. In general, there are no limitations for BIM 360 docs, not from product point of view, not from technology point of view, but we recommend to speak with someone from Autodesk. Maybe that changes. If you plan to update, upload terabytes of data, it would be good that somebody from Autodesk knows what to do. Yeah.

You need an S3 data packet to upload the data first to the cloud. So you need an AWS admin account to set up, to configure, to order the Snowball device, so that's also prerequisite. Check with the IT the Snowball requirements, because they have some requirements about network connection. They support DHCP network, but if there are some limitations in your organization, what is allowed or firewall blocking or proxies, whatever, then please, please talk with them before you order.

You have to enable the Cloudsfer app on your BIM 360 account. It's for free. Keep in mind, this is per account. So if you want to address many accounts, you have to do it per account, so that's important. But then it's valid for all projects per account. Yeah. You need to close a business account for this example here. Speak here also with someone from Tsunami. They can help you to find the right data transfer quota, what do you need, maybe for a test so they can maybe give you more free space to test. Yeah, it's up to them.

And that's optional. I recommend the application S3 Browser I will talk about later a little bit, which helps also to review the data which are in Amazon S3 before you transfer to docs. These are the prerequisites, yeah. Next step is about the Amazon Snowball upload technology, so I will explain a bit in detail what happened. Honestly, I don't like presentation where I have to click a lot, but I use this style because I want to give you a step by step introduction of the technology, yeah. So I have to click a little bit here.

So first one, what you have to do is, as I said, you have an AWS account, an admin account. So from this point, you can create a Snowball job, yeah. I assume that credit card data payment data are part of your account, but then you can create a Snowball account. It's really easy to do. You can click here, then on the Create Job, and then you will guide it through the whole process. It's really easy.

So then you have this main decision. What do you want to do? Do you want to import into Amazon S3? You can see that you have other options. You can export as well, you can sync, so there are a lot of other possibilities. We will focus at the moment on the import only. So then you enter your shipping details and the shipping speed. Some advices for my site, for shipping details, I recommend to name a person, not an organization, that really this device is handed over to this specific person.

Any enablement or unlock of the Snowball device is available after you receive the device. So during the transit, if you didn't sign the UPS delivery, the device will be locked, yeah. You can only unlock with the manifest, I will come back in a second, when you have the device in hand. So please put a person who really works with the device. Its only recommendation.

Shipping speed also does not mean the ship did the preparation of the device. It's really the shipping by UPS, by whatever. DHL. It's all different vendors here. Then you have to select the capacity. What kind of job do you want to do? I took an 80 terabyte device or smaller isn't. So I have only these joists here, and this depends which device you will get, how large the box is at the ends.

And then you have also to define the S3 data packet details. What is your target data bucket if you want to import, or Amazon should import, when the device is back at Amazon? Next one is notification service. Here you can set up a notification service. Keep in mind, when you setup the notification service you will get a first confirmation email, and you have to confirm this email before you get all these messages. If you don't confirm, you will not get the messages. And check your junk email folder, because sometimes it lands there.

And then, summary, it is, as I said, easy to set up. It takes you five to 10 minutes if all the data is ready. You can review it, and then it's ready. The job is created, you see the state of the job, sorry, as a whole, status overview, and now when the jobs [INAUDIBLE], it's in preparation for delivery. Until this point, you have a-- when it's not in preparation for delivery, you can cancel the job. So that's, what, 30 minutes, two hours, depends on availability. But you're possible to cancel if you made a mistake, for example.

And this status overview gives you a detailed overview that it's prepared, as it's shipped to you, and then you can see also the tracking ID of the parcel. So it's really very, very transparent to you. Yeah. And then it's in transit to you, and you can see where are the devices. So job is created. What's next? It's in transit, so then you will receive the device.

So as I talked, the Snowball is this device. It has a size like a desktop or larger desktop PC, and it's dust proven, so very protected. I mean it's easy to open. The good thing is it has an e-ink display, and that's really nice, because you don't need to have care about where to send. Automatically when you close it, when you close your job when you're done, you close the device and the ink display will change automatically to an Amazon partner address where you have to send it.

Yep. So this is how it looks like. You arrive, you unbox it, and then you have to plug it into the network. In my example here, I did it at home, I connected to my router. You can connect it to a company network. You need to discuss with IT as to where to plug-in. And on my home network I'm using here DHCP support, so that's important. You have to start-- it takes approximately two minutes to start. It's kind of as the boot of the machine.

Keep in mind, this device is very powerful, creates a lot of heat and a lot of fans inside, so it's very loud. So if you plan to do the job in your office, don't place it near to your desk, yeah. Look for a room where you can place it. It's a recommendation, because it's really loud and not comparable to a PC. Then as I said, you connect, in my case I got a DHCP address. You can configure here at this point, if you have IT requirements, to set up a specific range, IP address, subnet mask, whatever, all of the details you need.

So when it's connected, it's a device. You can ping it in your network. It has IP address, which is displayed on the e-ink. So you connect, you can check if it's there, and then you get also the information. And from this point, as I said, when you receive the device in the Amazon Snowball in the console, there is a manifest. And you unlock the device with the manifest, and this will be available only when you receive the device. So during the transit, it's not available. You cannot access, so that nobody can use it. So it's very, very secure.

So you download the manifest from the Snowball page and then it's a simple command to unlock the device. And then the device is ready in your network to copy data on it. Copying data, easy. In this example here, I used a recursive pass, so I copy all the data on all the subfolders from this specific folder here. And you define what is the in your data bucket, direct to where the data should go.

Yeah, that's important. So if you don't name here a directory, it will be directly copied into the main folder. In my case here, this was a test I did with 53 gigabyte. I created 10 times the directory and, yeah, for this reason I have this folder structure there. So then when you copy the data, yeah. Some recommendations here from my site. Don't use machines which are connected with Wi-Fi to your network, because it slows down the copy process connector machine where you copy from, or which you use for the copy.

With the LAN, physically cabled, really it's a factor by four or five faster than using minus LAN. So like you see here, I place the laptop on top of it, I connect it and I'm done, yeah. Good. When it's done, yeah, you stop the Snowball with the stop command. You unplug it, you close it, and automatically has the e-ink display changed, switched to the address we have to send.

And then you have to call your package service here. Coordinate to pick up a UPS. In Germany it was sent by UPS. I'm not quite sure what happens in your countries, who's the provider, who will pick up. And that's it. That's pretty easy, yeah? What happens next? You send it, and as I said, as a full transparent what happened, so when you send it, switch the state, delivered to you, and then it's shipped in transit to AWS.

And there are more stages here, and that's what we will talk about. And it's shipped to AWS, it is at AWS, that means it arrived there, and then they import it and then the job is done. Yeah, I will come back to some performance metrics later on this. When the data are in S3, you have many options to access the data, to check if everything goes well, yeah? You can use per standard AWS S3 console. There is a tool where you can directly go through the folder structure and do some smaller manipulation.

It's not a lot you can do, so for this I recommend maybe some third party tools. You can use this one. There is a free edition if you want to test. If you use it for business, you have to pay it, but it's very cheap. But this tool helps a lot if you maybe want to rename some folder before you migrate with Cloudsfer for some smaller changes. I don't recommend to use this tool to copy data between folders, because what does it mean? It downloads and it uploads, so not recommended.

It's not a direct copy, so not very perform enough. But here for view, to check whatever, you can do it. You can create many, many buckets, all listed. You have specific connection security details. Yeah. And with your S3 bucket details, you can access them here from the product. So data are in S3 now. If you have huge data sets, really huge data set, there is also called Snowball trucks. I'm not kidding. It's really the name of it.

So it supports hundreds of petabytes. So really customers-- there comes a truck to your data center. They plug-in these kind of big cables on it. They have a specific own energy power system in it, and then they drive to Amazon data center and plug-in again. No, I'm not kidding. If you have really large, large scale data sets, you can use this as well. But the principles are the same. There are a lot of customers, like the insurance industry, for example, where they move all their applications to S3 and they have tons of data. Yeah, they use this one.

Good. Now data is in S3. Let's talk about how we can migrate them to Docs. We will use here, as I said, the Cloudsfer application tool. And same system. It's an easy process to do it. So if you have your business account ready, you order your quota, your data quota, and it's a web service. You log in with your account from Cloudsfer and then you get this browser UI, and there you can define your own personal migration.

There are a lot of other options here. You can use-- there is an option on premise to cloud. This is a kind of beta status at the moment. They are working on a solution at the moment to migrate data directly from on premise. But again, this uses your bandwidth, so it will not solve the problem by uploading the data. Yeah.

Here you have also the activity log. You see some pricing details, so I will come back to this in a second. So here we want to personalize a migration, and then it's simple. Because you have to define the source and the target. As the source, we select here the Amazon S3. It's important that you-- there are two options. There is a personal one and there is a business one. That's really important. So the connectors are different between.

So if you, for example, use a personal Facebook connector, it looks completely different than the business one. Yeah, it makes sense, because yeah. We're not talking about one account. We talk about really the details behind the means. You're connected then to the back end, and not to the front end like if you have a personal one.

Here, for the business one, we use the S3 account, and then you have to enter your details, your access ID, your security access key, your bucket names, all the details which you have with your S3 packet automatically. And that's important. You can choose, as mentioned, which data center, yeah? In this context here, we have to use the EU data center, so that's important. So in my case, I use the data center here in Frankfurt.

The whole service at the moment runs in US environment and US AWS. But if you need, for a specific customer request, that Cloudsfer should run in European environment, they can set up environment for you. Yeah, that's no limitation here. So here we defined the source and the target folder. Really you click through your folder structure, you select your subfolders, your main folders, whatever you want or need, and that's it.

And the same you do for Docs. You have to log in with the Docs account. Again, keep in mind that the app Cloudsfer is enabled here. If it's not enabled, you will not be possible to connect. You will not get listed any account. You will only see the list of accounts which has these Cloudsfer application enabled. So if it's empty, go back, check if everything goes well in your account settings. And here you have to allow the access to Docs. You have to give Cloudsfer access to this application.

And then you also define the target folder structure, where the data should be imported. Could be completely different, but from this all subfolders will then be migrated. Next one is you have advanced option filters, so you can include subfolders. Yeah, you can see here. You can read, you can override existing files or create a new version, so that's really important, yeah? You have to make decision for once to reload maybe or do a re-initial load. Then you have to override or, if you don't mark this, then you will get a new version. So as I said, you can use it also as a recurring transfer method that's really important.

And you have also filters, so you can filter out when was the last modification dates, some substrings mentioned. When you define everything, you have to review the plan, for sure, and then you can start the migration. Now you can schedule the migration, you can save the migration plan for later, and you can also analyze.

That's a pretty cool feature, because it reads all the data from S3. It shows you how many files, what is the data size, so you get a better feeling how long it will maybe take. Because when I share some numbers with you and you have an upfront calculation about the duration. Yeah. I recommend, if the data are in S3, you should start to define your migration plans. You don't need to start them, but you should start to define them.

Because sometimes it takes really time to define, but you can use a copy reuse of existing jobs. We'll come back after this in a second. So when it's reviewed, yeah, you can start. You get a quick view, OK, is this really what you want to do? And you start the job, and then the job is in the queue. And during the run, you can monitor an activity. You see the results here. The screen looks really bad quality. Yeah.

But you can review, you can monitor, and you can review the results as everything goes well. You get a detailed report, an Excel spreadsheet or CSV file normally, and then we can use and check if any files migrated correctly. Some background information behind the scenes. I think I mentioned a lot, but summary here is clear. The Cloudsfer is an Amazon Web Service, and the environment of the location today US. Speak with Tsunami if you need a European.

You can use an S3 secure layer connection. It reads the data. During the whole transit, the data is encrypted. Also during the processing they are encrypted. And the important point is, the data will not be stored permanent, yeah? So don't worry. They will be directly after processing deleted. Yeah, this is this here. And same, we have use the standard security layer connection to Docs, which is used by a lot of other applications.

Good. Performance. Some numbers about performance, about the job I created. This is the test data set I used, more than 53 gigabyte data set. I have at least three or four numbers. The duration of the whole process was four days. That means, from the order to the end, when there was in Docs. So that's really important to understand, it's not the runtime, it's really duration.

I had other results as well. There was a good example. I ordered the Snowball, I imported to S3, I transferred to Docs using Cloudsfer and it takes only four days. The copy of this data set to this Snowball device with a LAN connection with an average speed of 70 megabytes per second takes me 30 minutes. The 70 megabytes a second is limited by my machine, limited of my hard drive. It cannot support more speed.

I think if you connect to your data center you will get much higher copy rate than I had, yeah. Input into AWS from a pure duration point of view, from the runtime, should be approximately three hours. It wasn't possible to calculate. I only viewed the job, the status lock, and the job status lock showed me at 9 AM the devices on shipment. And at 12:30 PM the data was on S3. So that was my idea, OK, should take three hour between when it arrived. And so that was really fast.

I ordered, next day it comes. I copy the data, I closed, I ordered UPS. On the second day in the evening I sent back, on the third day it was at Amazon, and on the fourth day then I run the migration into Docs using Cloudsfer. And this took me 4 hours and 30 minutes. This is an average transfer rate of 20 gigabyte per hour. I run only one job, so no parallel running.

When I started this, my assumption was, OK, we talk about cloud to cloud, should be 100 gigabytes per hour and not 12. So the limitation comes from Autodesk, to be sure, because we want to keep the system healthy enough. So we don't want to crash a system by loading too many data in it. So that that's limited by all that at the moment. And that's the reason why I mean speak with someone from Autodesk if you plan something. Maybe we have better numbers than we have today, yeah.

But at the moment, the average speed is 12 gigabytes per hour. And now let's have a look to the big project. As I said, we talk about two to three terabytes of data. I separated, I split up into two packets or two jobs, because the smaller package includes all design data, access spreadsheets, PDFs, a lot. And the bigger one contains point load data. There was only 60 files of point load data, so not so many files, but the biggest part of it. So we tried to separate a little bit.

The 14 days is for both parts, for one and two, because I didn't split it. I ordered one time the Snowball. I copied to the terabyte on it. For this reason, it's 14 days. And 14 days is really a bad example of maybe Amazon service, because we had a problem that there was no device available. We ordered the job and it took only seven days when we got the device. And you cannot control it. That's really a negative point in the whole process here.

So please keep this in mind, if you plan similar, you never can plan with Amazon ability. We contacted the support of Amazon. They said they feel sorry, but today there are no devices available in Germany or in European regions they can send to us, yeah. That was really bad example. I'm not quite sure if there's maybe other person or group you can contact at Amazon to make it have a better schedule, but yeah. That was the experience we made, honestly. Yeah.

But when we got it, we copied this two to three terabytes to the device. It took 18 hours. And there was copied by a colleague, so we used your standard home network performance not yeah big IT performance here. Yeah. And then that was really fast. We sent back and only took a day when the data was in S3. So there was good performance, I think. We sent back, a day after data was available.

And bring the data to Docs. Also high performance. It took us in sum for this one nine hours. As I said, we had the same average. We run three jobs in parallel. At the moment Cloudsfer supports three jobs in parallel, which is also limited by Autodesk unfortunately at the moment. Again, contact someone from Autodesk if you have questions here. So we have at the end 33 gigabyte powers run the three jobs in parallel.

And the good thing is, when you have all the jobs, defined if you have many jobs, I recommend to separate into chunks. That's important. And if one job is done in Cloudsfer, the next one would start automatically. So you can start all of them, but only three will run in one process, yeah. And we have no errors here with the huge set. And that was here. That is the data set.

And here it was the same, 14 days, 18 hours one day. But then we transfer this data also into Docs using Cloudsfer. Same performance. It took us then two days and 20 hours in sum for the duration. Not a run time, really, from duration. So I think it is a valid performance, yeah? It's not if you're copying data between one bucket to another in S3. It's for sure, because there are a lot of processes running in the background if you upload data to Docs. Yeah, that's important to know

Good. Recommendations and limitations. Any questions so far about the process?

So does the Cloudsfer work completely inside the cloud as opposed to bringing anything back and going back?

Their servers run in a cloud, in their own cloud service. They load it to their cloud service and pass them to Docs.

OK, but it doesn't have to come back to your Davidson or anything like that? It stays in the cloud?

It stays in the cloud, yeah. And not for permanent. That's really important. Only for the transit. They will delete after. So they pass the machine, the Cloudsfer service will push to Docs. And during the pass they needed to process, but they were not store it, and they would be completely encrypted during this process.

So inside the Docs, you don't have direct access to your S3 project?

No, you don't, no. That's the reason why we used this. Yeah, that would be awesome if you have direct connection. But I think the idea here is, I cannot speak for product management, but I feel that that's not the idea of Autodesk to create these deep integrations so use [INAUDIBLE] vendors. They know their business, and one of these companies, they're really skilled to processing this data and build these connectors.

But I agree, that would be great. Because at the end you know the debates layer of Docs is in Amazon S3 storage, yeah, so why not [INAUDIBLE]? But this is an important point. There are a lot of operations. We are not storing flat structure, object structure, and the 3D data needs to be processed to be readable, you know? The easiest way to explain is the translation service to render the preview for fights. But we come back to limitations. They are disabled for these kinds of migrations.

That's what we'll talk about now. So what are the known limitations? The Project Files section is supported only. So you cannot import into the plans section. Maybe now you have two sections, one to plan, one to project, so plan so plan a project five sections supported only. Maybe for the future, but it's the same. We don't have any integrations at the moment to the plan section. Maybe you're aware of Autodesk Desktop Connector, which is a desktop tool where you can copy data. It also doesn't support planned section, because the API isn't ready at the moment, yeah.

We don't support any references. So if there are references, if you have Revit files, DWGs, which reference linked files, they are not supported. If they stay where they are, so if you link to a C or D drive folder, yes, they will have the same result, but we will not transfer a map into cloud folders. So the original data will not be touched, so that's very important. They were transferred, processed as they are.

Then there are some unsupported file types by Docs. Yeah, that's system requirements from Docs. You can find a list here, and there are some specific types which are not supported. And then that's my biggest pain point at the moment, I can see, is the reduced transition rate into BIM 360 Docs. Which limits to the approximate 12 gigabytes per hour, and this is also based on the automatically created. So we have really here a limitation built into the product that you cannot blow or crash the whole system.

But you can run many jobs in parallel, yeah? So if you're clever, maybe you can do it. But again, speak with someone from Autodesk. And also, the next biggest one for me is translation is disabled. If you have files like Revit, whichever, you wants to preview, this per default is disabled. So when you upload to the API UI, drawing, you know it will be rendered automatically. You have a preview ready after upload. And this has not happened, so the translation needs to be done by the user, for example.

They can select many files and there is a specific item, start translation now. Yeah. And yeah, the file will lose some file and meta information like the original data and modification data, because they will get automatically the date from the upload in Docs itself. But it's the same if you upload manually today to Docs, yeah. It will lose also the original meta information. So let's come back to-- that's the limitation, make some recommendation , because that was the intention of the class, to share limited experiences and best practice.

I can recommend, so in general, yeah I said many times, stay in contact with Autodesk. I think for the second point, I recommend this approach only for huge data sets. If you have maybe 20 gigabytes, 50 gigabytes, for example, it doesn't make sense. You can use the UI used to connect whatever it is. I'm talking really about terabytes of data. This will help. This was the solution. It's really for large scale data sets only. I recommend it.

I definitely recommend, run a test the same as I did. Select a smaller data set that you believe will get a feeling how to do it, how long it takes. So make it-- yeah, you have to test it in your own environment, in your own country, whatever, with all the details you have. I put here on recommendations. In general there's no storage limitations. When you hear this is 12 gigabytes limitations off the upload, OK, but Autodesk with Docs has no storage limitations, yeah.

But again, this will be discussed then in the conversation with Autodesk with you. And for your own saving costs, when you do tests, when you run the data, don't forget to delete all the buckets. I mean, it's not very expensive to store data in S3 in a micro sense. But in the other hand, if you do many tests, if you do many data migrations, they are permanent stored and you pay per storage. So clean up your S3 and that helps a lot. For AWS snowball, as I plan a buffer time you never know how long it takes and how all of that excitement was honestly seven days when we received the device.

Maybe there is a possibility of a specific Amazon contact support, or whatever representative. Contact them to talk about this. Preprocessing. I think this is more related to the on premise, so I recommend to do the preprocessing. Rename folders, delete the files you don't need, rename whatever. Do it on your on premise environment, but you can do it also in S3 using the browser, using scripts, secret script, whatever. Yeah.

And copying data to Snowball, don't use wireless LAN. Use the LAN, as I said. Cloudsfer, I think I said it. My recommendation is to split up the whole migration into smaller chunks. It's better for handling, yeah. It will not help you if you have a two terabyte job in Cloudsfer. Will it run the same? Maybe you have big, big log files to review. It's possible, nothing against, but it's for honesty, for better ending.

And again, if you have 20 or 30 jobs as I did, they will start automatically when one job is done. So don't give Carol what, don't worry about it. Oh, I have to start them off, but you can better schedule for example. Yeah, it's up to you but that would be my recommendation. Schedule and pre-configure. Yeah, I prefer the schedule mode.

And pre-configure is a good thing. Can copy jobs. You configure one job and then you can use this job to use all the configuration settings like the AWS connection details and the Docs connection details. They will be in a job. And you only need to change maybe you folder source on the target folder, but all the other details, it takes time to browse. And this helps you a lot. My recommendation, create a template job and then use a copy of it. Yeah.

On Docs, as mentioned, translation service is disabled, so you have to inform the users when the data are ready. They are maybe a little frustrated because they don't get a preview automatically inform them that they have to hit the button. Maybe it takes them half a minute or whatever. yeah, I did it I did it on my own. I selected-- and here is also at the moment a limitation. This translation service, which creates the preview, you can only select 50 files in one run. Yeah, that's hopefully changed.

And yeah, as I said, use the whole approach only for real large scale data sets. If you have smaller sets, use the UI to upload. Use the desktop connector. And you can also upload a parallel data, so nothing against. You don't need to stop working. That's a good thing, you can do it in parallel when the people are working you can load data into it. Yeah, nothing against. So that means the system is open for work during the migration days. There is no-- you don't need to wait until the migration is done.

So, yep, that was a quick run overview. I think, as I said it's not rocket science. It's easy to use, it's flexible, it's scalable. You can really easy adopt this approach here in your organization if you need to or if you want to do a service, to provide a service to upload data. This is an example. There are maybe other vendors. I had a quick contact with another company, but there wasn't possible to share me some information on it.