Generative AI as a Visualization Tool for Artists and Designers

ANA MANCERA: Hi. I'm Anna Mancera, and I'm the chief design officer and co-founder of at HUM, a generative AI startup focused on interior design and real estate. I'm also a consultant for designers and creative studios who want to implement AI in their current workflows. And before that, I was the co-founder of Flow, a 3D animation studio focusing on architectural visualization. I've always been interested in the intersection between creativity and technology, especially in the ways that technology can help us optimize our creative workflows. And I see a lot of potential in generative AI for this no matter the industry that you're in.

Over the past year, there's been amazing advances in the field of generative AI. And there's been a lot of controversy about AI taking over creative jobs. Now, personally, I don't think that this is what's going to happen. But I do believe that people who take the opportunity to learn about these new tools and include them in their workflows are going to have an edge over those who choose to ignore them.

There are so many AI tools around these days that figuring out where to start learning about them can feel a little bit daunting, especially since some of them have very steep learning curves to get really useful results. A good way is to think about what we want to create with these tools. For text, we have large language models like ChatGPT, which I'm sure by now everyone here has used. And there are some of these tools that can even know how to code. But we're not going to be focusing on those for now. For images, the most popular one is probably Midjourney. But there are a lot of other options, each with their own advantages.

For example, Firefly and Veras live inside software packages that we already use. So implementing them into our current workflows can be a lot easier. Others like Stable Diffusion offer us more control over the end product. When we talk about video, there's everything from AI-created avatars that can read the text that we give them to AI-powered editing tools and even generators that can create completely new video from text or images that we feed it.

Finally, AI can also help us create 3D models, either from video that we've already captured or just from prompts. These tools are currently in beta versions. But going from the advances in text, image, and video that we've seen over the last year, these are definitely something to keep an eye out for.

We'll be focusing on images and video, mostly using Stable Diffusion and Runway. But regardless of the tools that you end up picking, a lot of the techniques that we're going to be going over are going to be the same. And in any case, the important thing that I want you to focus on in this class is how you can implement this on your actual workflows, not so much the technical aspects of it.

So the way creative workflows are structured is almost the same regardless of the industry that you're in. We start with the research phase where we gather information, we find inspiration, and we immerse ourselves in the topic that we're going to be exploring. Then on the ideation phase, we turn all this research into a concept. And we have to explore different ways of how we could present that concept. Then in the development phase, we really flesh out that concept that we came up with and fully develop it into a product. Finally, in the revision phase, we make sure that product actually works, and we edit and correct as needed.

AI can help in all of these phases. For example, in the research phase, we can use chatbots to answer questions, organize our data, keep track of our references, and even use image creators to help us in our visual searches. In the ideation phase is where I truly shines, as it can help us explore many different variations in a very short amount of time. It's a great brainstorming companion, especially for solo creatives. And it helps reduce creative block by giving us a starting point instead of having to stare at a blank page. It's great for clarifying concepts as well, either with external clients or even within our own team to make sure that everyone is on the same page.

And the development phase, it can also help us create production-ready assets. And it can manage these assets in a great way, making it easier and faster for us to find them. It's also very good at a first pass for visualizing references in our specific context. And, finally, in the revision phase, there's a ton of AI-powered editing tools that make this a lot faster now.

We're going to be working on the ideation phase for the first example that we go through. And we're going to be using Stable Diffusion for this. Stable Diffusion is Stability AI's foundation model. Because this is an open source model, there are many different front ends that you can use with it. I'm going to be running a local install of AUTOMATIC1111. You can find installation instructions for this one on the handout. But if you don't have a Windows machine or if your graphics card is not enough to run it, then you can just as easily use DreamStudio, which is the web version.

The first thing that we need to pick once we get started is the Stable Diffusion checkpoint that we're going to be using. You heard me mention that Stable Diffusion is the foundation model from Stability. But what does that mean?

A foundation model is a large scale model that has been trained on enormous quantities of data. And it has learned to recognize billions of parameters. Because of this, these are expensive to develop and train. Midjourney, DALL-E, and Adobe, they all have their proprietary foundation models. The difference with Stable Diffusion is that because it's open sourced, anyone in the community can go in and fine tune a model on top of that. This is a model that has been retrained on a much smaller data set to improve on specific tasks.

For example, you can have one that's really, really good at creating photorealistic images. And this has allowed a lot of domain-specific models to come out. On top of that, we even have smaller models that we can use. These are called LoRA models, and they can be used to train specific concepts. For example, if you're an artist, you could train it to replicate your art style with just a few dozen images or you could show it how to interpret a character or even a concept that the original model didn't know.

These are very tiny data sets compared to the original. So for Stable Diffusion for the foundation model, it took billions of images to train it. A fine-tuned model could use thousands of images and a LoRA can take less than 100. So this means everyone can train their own models now.

For this class, we're going to be using one of the fine-tuned models. But I want to go over the latest foundation model that Stable Diffusion has. This is SDXL. And it now generates images at a much larger scale than the previous base version did.

The problem with this is that it also takes a much more powerful computer to run. And because at the time of the recording, there were also some extensions that hadn't been implemented in SDXL. Instead of using that one, we're going to be using two of the fine-tuned models that the community has created. These are Realistic Vision and Deliberate.

The other thing that we need to pick before we get started is the right samplers. Samplers essentially tell the model how to interpret the noise from one step of the generation to the other. And if you change the sampler, you're going to get very different images even if you're using the same model and prompt.

Euler A, which is the default in AUTOMATIC1111 is the fastest one. So it's very useful to speed up our workflow. But if you want to have more detail in your scenes, I recommend choosing one of the DPM ++ ones. Now we also need to make sure that our size matches the images that the model was trained on, which in this case is 768 pixels. And we're going to up our batch count to 10 so that we can really see if our prompt is actually working or not.

Finally, if you're following along, you're probably going to notice that your results are different from mine. This is because of the seed value. If you have it at minus 1, it's going to use a random value in each generation. And in order to replicate an image, you need to know that seed value.

All of the seed values that I'm using are on the handout. So if you want to replicate that exact image, you can do it that way. Also, if you ever need to check the seed value or any other setting of an image that you created, you can always go to the PNG Info tab, and you'll be able to see all of the settings there.

So now that we've gotten all of the technical portion out of the way, let's get started with the real creative stuff. So for this first example, we're going to be doing an environment concept art for a game. And here's the brief for our game. A puzzle game that takes place in a fantasy world with surrealistic elements. The architectural elements form mazes that the player must navigate through.

So how do we take this brief and turn it into our first prompt? The basic structure of a prompt is having an object and the style that you want to represent it in. You always want to start simple so that you can really see how much impact any new keyword that you add makes in the model. So using that as our starting point, let's pick two keywords, a labyrinth in the style of surrealism.

And keeping it this simple, let's click on Generate and see what we get. So even from the previews that I'm starting to see right now, I know that this isn't what I wanted at all. But that's fine. Something that happens a lot is that people will look at images online that have been created by AI, and there's really cool images.

And then the first time that they try out these tools and they come up with something like this, they'll get really frustrated. And they'll feel discouraged because the results don't look anything like what they saw online. But this is normal. And even if right now this image doesn't work for us, I want you to take a good look at it because we're going to come back to it later in the process.

So let's think about our prompt and how we could improve on it. We want this maze that we have to be made out of buildings. And those buildings are going to be forming something like a village of some sort. So let's try a maze-like village in the style of surrealism. This time, we're getting something a little bit better.

One thing I want to note is how something as simple as a synonym that we could think would result in the same exact kind of images can change everything in the generation. For example, using labyrinth gave us all of this sort of spiraling structures. And now when we're using maze, we're getting these rectangular structures instead. And that's the sort of thing that you need to be aware of when you're trying out different prompts because sometimes the model is going to interpret concepts in a very different way than you would expect it to.

So coming back to our images, I like that we're starting to get a little bit of the building design, even if it's just like a very little amount of it right now. But it's still too focused on the maze, and it's still looking a little bit too blocky for what I want. So for now, I'm going to completely forget about the concept of the maze. We're going to come back to it. But right now, let's just focus on the style of the buildings that are going to be creating this village that we're working on.

And one thing is I really do like that on the image on the right the village looks kind of in the middle of an ocean. So let's stick with that. And let's try seaside town in the style of surrealism. Only, let's be more specific. Let's try Salvador Dali surrealism because there's many takes on surrealism.

So let's see what we get. And, immediately, we start getting things that are a lot more interesting than the maze images that we were getting before. We have this flame shapes and all of these elements flying around in the sky. And it's starting to look really like something that we had in mind. So let's try some different artists and see how that changes the generation.

Here we have Dali, Magritte, Carrington, and Remedios Varo. Once we have a palette of options like this one, we can really start digging into the specifics of each image that we like. For example, I really like that in the Varo ones the sky and the sea have a lot more texture, instead of being just like flat colors like we have on something like the Dali ones.

But I still feel like the houses and the buildings are still missing a little bit of texture. So let's add one more keyword to our prompt to focus in that direction. Let's try seaside village in the style of Remedios Varo and then Victorian houses. And now this is really starting to look like the mood I wanted for my images. I love this tower on the left with its wing structures on the sides. And I love the wooden piers running all along the edge of the village.

So at this point, I think I'm happy with my prompt. Something that you'll often see is that many people recommend using very, very detailed prompts. And you end up with this huge walls of text with negatives, and emphasis, and all sorts of different things.

If this works for you, that's perfectly fine, but I'm a very visual person. And I find that many creatives are as well. And going into this much detail on a prompt to try to get it to work with the composition that I have in mind is kind of cumbersome and time consuming.

Now using negative prompting and emphasis can be useful in specific cases. But, overall, I prefer driving my generations based on images rather than just the text. And this is what we're going to do now in order to bring back our concept of the maze to the style that we're already getting from our prompt.

So there's an amazing tool in Stable Diffusion that allows us to do this, and it's called ControlNet. So what we're going to do is we're going to open the ControlNet panel. There's information on installing this in the handout as well. And we're going to be uploading our maze image.

And now we're going to click on Enable. And we need to tell the model how it's supposed to interpret this image because there's many different ways that it could do so. These are the preprocessors.

So, for example, if I pick Canny, it's going to do some sort of line drawing on top of the image that I gave it. This is a great tool if you want to focus on specific areas that have a lot of detail or if you're starting out with a hand-drawn sketch as your input. If you choose Depth, this is going to look familiar to anyone who's worked with 3D because it's an estimated [INAUDIBLE] of the image that we gave it. This is great if we want to focus on volumes instead of detail, and it's very good for architectural work.

If we pick T2I Color, for example, it's going to turn our image into a color palette. And this is a much easier way of getting specific color shades than trying to say very light pale and saturated yellow with greenish hues, which is going to be impossible for Stable Diffusion to interpret correctly.

Then we have semantic Segmentation, which uses computer vision to know what kind of objects we have in the scene. So, for example, if we wanted to have a car but we don't really care about what model we have, this could be a useful tool. And if you're working with characters, you also need to know about OpenPose, which can take a post from a reference photo and apply it to the character that you're working with.

So for this example, we're going to pick Depth because we want to focus on the volume of that maze image that we have. And we're keeping our same prompt that we already had. And now when we click Generate, you can see that it's starting to balance out between the prompt and the style and the actual shape of the maze. It's still looking quite blocky right now. But there's a couple of parameters that we can adjust to make this better.

First, we have the control weight and then we also have the ending control step. So for example, if we lower the control weight to 0.5, we see that we start getting a lot more detail in the buildings, and we don't just see the blocky shape of the maze right now. If we go too low, however, we can end up losing the maze shape entirely like on the image on the right, where we can only see the spiral on the curves of the waves.

The other option, the ending control step-- the first few steps in each generation are the ones that are more important in terms of composition. So if we only use the ControlNet on those first few steps and stop it with the ending control step before the end, we're going to see a lot more random details added to the composition, but it's going to keep that circular composition that we have, sort of like in this images.

These are a couple of examples that we did with the other maze-like village prompts that we had initially. And as you can see, you can really start deciding on the style of image that you want based on this. Something else that you could do at this point would be to model a very basic outline of your maze in any 3D software and then use a rendering or even a screencap of that model and use it as your control.

Now I'm not going to do that at this point. But I'm going to go back and take one of the images that I really liked, and I'm going to keep improving it to show you some other techniques that you can apply.

So I really like this image that we got over here. Not only is it capturing the mood that I'm going after, but it's also giving me elements about gameplay that I could implement in an actual game. For example, I really love those three floating lighthouses in the back. And I can see them being part of the puzzle that the player has to solve. Like maybe he needs to align them in a specific way in order to be able to navigate through this maze or, otherwise, maybe the ship that he's navigating will sink.

So this is the kind of thing that you can really get out of this generations at this point. I want to add a couple of elements from the other images that I had. For example, I want some more houses on the left. And then there's this little sailing boat that I'm going to use to represent our player. And I don't really like the central part of the mids that we had. So I'm going to try with this other structure that we have over here.

So what we do at this point is we can just go into Photoshop and do a very rough collage of all of the images that we want to put together. The amazing thing about Stable Diffusion is that it's really good at cleaning up horrible collages like this one. So I don't really even need to focus too much on making my masks perfect around each of the objects. So let's see how it does that.

We're going to go to image to image. We're going to type in the same prompt that we've been using all along. We'll input our image. And then we'll also use it on the ControlNet, enable it and use [INAUDIBLE].

And the really important setting for this is the denoising strength. The higher it goes, the more it's going to change the image. For now since I only wanted to clean up my collage and I don't really want it to change the composition, I'm going to put it at 0.3. And then let's see what it does. As you can see, it did a pretty good job at integrating my crappy Photoshop job.

So at this point, this image is only 700 pixels wide. So this is about as much detail as we're going to get at this size. So the next step is to upscale this. There are two ways we can think about upscaling when we're talking about AI. One is to keep this exact image the same and just make sure that it's not blurry or noisy, so something like this.

The problem with doing just this is that now we can really see that there's not a lot of details in many of these areas. So a more interesting way of approaching upscaling is using a script. Let's try this out. We'll add our prompt and then use our base image both as an input and in ControlNet. But this time around, we're going to pick tile for our control type.

And then under the scripting section, we're going to pick ultimate as the upscale. Again, there's instructions on the handout for how to do all of this. And now on target size type, we're going to pick scale from image. And we're going to leave it at 2, which is going to double the size of our image. And then we're going to pick 4x ultra sharp as our scaler. And on type we're going to pick chess.

And so what this is going to do is it's going to draw a grid on top of our image. And then it's going to treat each of these tiles as a separate image. This allows us to create much larger images without running into out of memory errors.

Now that we have this set up, we can also change the sampling method to one of the DPM++ ones to give us a little bit more detail since that's what we're looking for right now. And we're going to lower the denoising strength to 0.5 so it balances between adding more detail and keeping the composition that we have right now.

So now when we click on Generate, we're going to see our tiles processing and then we're going to get something like this. As you can see, this is a lot more detailed than the first upscaled version that we did. It even added these cool-looking dragon shapes on the back and a lot more details on the windows and all of these parts of the houses. This also allows me now to draw over it or add more of our previously-generated images to take the control of where we want this image to go.

So here's the first Photoshop that I did over this one. I've added the tower that I really liked from our previous generations and used it to give some detail to the lighthouses. I broke up the maze in some pieces so that it had a more interesting shape, and I also added the very, very basic version of the wooden piers that we had in some parts of the maze. Now we can use this as an input for another upscaler, and let's see what that gives us.

This is the first version that I got using the same prompt that we had been using all along. And it does a pretty good job in most of the picture, but I don't think it's really getting the detail that I want in the wooden piers, and it didn't do a really good job on the rocks that I had on the right corner. So let's try guiding our prompt a little bit. This time, adding old wooden pier, rocks, and lighthouse, it gives us a much better result, especially for the wooden pier and the rocky causeway that we had.

I also did another version with a lower denoising strength because the water was getting a little bit too choppy for the look that I want to have in this image. So I wanted to have a flatter version so I could play with it in Photoshop with all of these different images. At this point, I also realized that I still didn't like the central portion of this image, but those spiky structures that we have there right now-- they kind of gave me an idea.

What if instead of having this tower, we had a shipwreck there? And this actually makes sense with the game that we're playing, right? If we have the lighthouses and we're navigating through this maze, then the shipwreck could be a way of telling our player what could happen if they don't get the puzzle right. So let's go back to Stable Diffusion, and this time we tried out Shipwreck Remedios Varo surrealism and we got these really cool-looking ships and shapes.

So we went back to Photoshop and did another quick collage with those different images. And something that you can do once you have a larger image like this one and you're only focusing on one specific part is, you can crop out this section and use only that section to run it through Stable Diffusion. So here we go.

And here's what we got as a result after running it through another image-to-image generation using the prompt over here-- Shipwreck Remedios Varo surrealism, rocks, and waves. And this is our final image. And just as a reminder, this is the image that we had started out with. So as you can see, we definitely need a human creative guiding the process and telling the AI exactly what we want to accomplish.

Now, I want you all to think about how you could apply this to your industry. Some of the ways that you could do this could include maybe asking it for help when you're drawing a storyboard or creating concept sketches for fashion design or architecture. Now, all of these images were created using only prompts as the start, but a lot of the time you're going to have existing assets that you want to be using in your generations. For example, if you're in film production you could have a frame from your storyboard that you want to take and turn it into the backdrop for your scene. Or if you're a costume designer for film as well, you could take your sketch and render it out in a realistic way.

And this is a collaborative tool as well. So if for instance, the location team already had a photograph of the place where this specific costume is going to be used, you could have AI create an image putting it all together, and you could even bring in the cinematographer to give instructions about lighting or the director to give instructions about how it's going to be framed. And some of the larger studios are already implementing this into their workflows because it's very good at saving them a lot of time and money.

If you're in content creation, you could use it, for example, to turn your photos into cook-looking watercolors for your foodie blog. And if you're in product design, you can use it to iterate over a concept that you have and try out different materials and finishes and try many prototype versions of it.

It's also really good for creating personalized content for your users. So if you're in the fashion industry for example, you could have your users upload a photo of themselves and then apply the makeup look that you'll have on one of your models. You could also have your users in interior design take a photo of their living room and then test out different styles and color palettes. The possibilities for this are endless, and for the rest of the class we're going to be working on new images based on existing assets.

In this case, we're going to be working with a real building in Atlanta that's being developed by Portman Residential. My company worked on the renderings for them and we wanted to test out some of these AI tools. This initial model was done in Revit. Now something that's very common when we're working on large projects like this is that creating the environment is very time consuming. Right now on this rendering, we only have the base model that we got out of Google as a reference and the satellite textures that come with it. So the challenge is to get AI to give me better results than this.

I also want the AI to give me some ideas about the lighting setup for my scene. So right now I'm only using a white environment. So what we're going to do is, first of all, we're going to switch our model to Realistic Vision which is better for photorealistic stuff, and then let's take a look at the structure of our prompt. The closer to the beginning of the sentence that we have a key word, the more importance the model is going to give it. So in this case, even if it ends up looking a little bit like word salad it can be useful to keep this structure.

We're starting with the type of image that we wanted to give using RAW photo which is one of these weird keywords that the model understands to mean high-quality photography. Then we have the subject of our image building in specific-like condition in Atlanta. And then finally, we have our style keywords, architectural photo and drone cinematography, which is how this photo would have been taken in real life.

And now we're going to upload our base rendering here, and we want to make sure-- and we're also going to upload it in ControlNet using Depth as we've been using. And we also want to make sure that we match the image size from our rendering to the generation that we're going to be doing. We're going to up our batch count to 10 to give us a little bit of room to see some choices, and let's see what it does.

So here are a couple of images that I got from this one. As you can see, even just with this it's already giving me a much nicer environment, right? Some of you are also going to notice that the building changed quite a bit, and we don't want that. We want to keep the same design that we have. But at this point, I really just want to see how the light reacts to a building that's more or less the same shape and size as the one that we have and it uses more or less the similar materials that ours does. So don't worry about it. We're going to get our design back. But for now, this is good.

Now just changing the blue part in our prompt, let's see what are the light conditions we could use. Let's try sunset golden hour. These ones are looking really cool. I like this. But let's try something completely different. Let's try dramatic stormy afternoon. And I like that in these ones, even the pavement looks wet. What if we wanted a night view?

So these are interesting. The bottom part of our image actually looks like a night view, and it's looking nice. But the sky is still looking kind of like a sunset, even if I'm prompting for night. So why is this happening? So the thing is, my base image has a very white sky and there's only so much that the model can do balancing my prompt, which is night, with this white sky.

So if I only change the background to this dark blue, if I run the same prompt I actually get true night views. So that's something that you can take advantage of, and you can also use it to direct the light. So if you paint in the light coming in from a specific side of your screen, it's going to do that in the generations.

Now, I really like the sunset views, especially this one with the sun coming up behind the building on the left. So what I'm going to do is I'm going to try to match these light conditions on my 3D model and I'm going to render that out again. I don't need to be so concerned about matching it in terms of color, but I do need to be very careful that I have the direction of the light coming in from the same side and especially that I get the softness of the shadows the same way as it is on the reference.

Now thinking about how I'm going to be using those images, I'm going to be compositing this with our background, right? So for the background, it's going to be useful to have a blank sky plate. It's going to make my life easier when it comes to compositing. And then because at this point I no longer want our building to change, we're going to be using a technique called Inpainting. So Inpainting takes black and white masks of-- and we can use it to tell the model what to modify and what to keep the same.

Based on what I said before, I do want to change the top of my building into sky, so I'm going to delete that from my mask. And then I want to keep the buildings that are closer to my building and that I already had modeled in a little bit more detail, because those are landmarks. Now when we go into Stable Diffusion, we can click on the Inpaint Upload tab. And then we're going to bring in our rendering without the top of the building on the top part, and on the bottom part we're going to add our mask.

And for now we're going to leave the rest of the Inpaint settings the way they are. That's going to work for us. And make sure that your size is the right size, and you're also going to bring in your control image over here and choose Depth. And here is the first result that I got from this, and then here is the upscaled version of that rendering using the same techniques that we used for upscaling the maze image.

So now we're ready for our compositing, and for that we're going to use one of Photoshop's new AI tools. So if we go to Photoshop we'll see that we have our two layers-- our generated background and our masked rendering. And right now you can see that our rendering is looking a little bit too yellowy-green compared to the background, so we're going to go to Filter and we're going to pick Neural Filters. And on the panel that opens we're going to pick Harmonization and we're going to toggle it so that it's active.

And then we're just going to pick the layer of our background over here. And just like that, we'll see it process, and now it matches. So now we just need to tell it whether we want it to use this as a smart filter or if we want to apply it as a new layer. I'm going to pick New Layer, and then when we click on OK we should see our new layer on our stack. Here's what that final composited image looks like. And as you can see, AI took care of about 90% of the compositing work and it created a much more interesting background than-- in a much shorter time than it would have taken us with traditional techniques.

So since we're using Photoshop for this, let's see one of the other tools that they have now. When we're working with marketing photographs for real estate, we often start with something like this. And this would take a very long time of retouching to make it look really good, right? So Generative Fill, which is one of Photoshop's new tools, speeds this up quite a bit. Let's take a look.

We're going to start with this area on the right with all of these dead-looking trees. So we're going to make a quick selection around it, and then on the context bar we're going to click on Generative Fill and we're going to type in what we want-- in this case, sidewalk with trees. And when we click Generate, in just a few seconds we have three options that we can use. So let's take a look at the second one and the third one.

I really like this third one. I think it kind of makes sense with what's actually in that part of the city. So let's do the same on this other side. And Generative Fill, sidewalk with bushes and trees, and we'll click Generate and we have our three options. And if we don't like any of these, we can always click on Generate Again and we'll get three more options. I'm actually going to stick with one of the first ones, and I'm going to be using this same technique to get rid of some of these structs. So let's try street and street with no cars. And there we go.

Another thing that we can do is we can make our selection, but instead of typing in something we'll just leave this blank. And then when we click on Generate, it's going to do a contextual fill. So as you can see, it cleaned up the side of the building really well without even having any prompts. Let's try it on this side. And we have a clean sky. So we're going to use the same technique. And we're going to keep cleaning up our image.

And one thing that you need to take into account is the size of the selection that you make. So you don't want to make it too large around the object that you want to change because it's going to change everything within it. So for instance here, when I type window, instead of having the three windows it gave me just one big window.

So let's try it again with a smaller selection, and now it's going to match. We can always adjust our prompt if we want it to be more specific, and of course we always have the tools that we've been using forever like Color Correcting if anything doesn't exactly match what we want.

Generative Fill works on the layers beneath it, so you need to be careful of where you're standing in your stack when you start your selection. Because if you are on the wrong layer, you might end up with something like this where you type in what you want and you see nothing happening. And then if you move it up your layer stack, you see that it doesn't really match what you had. So if this happens, just delete that layer, and then you can make your selection again. And this time when we click on Generate, we should have something that matches.

OK, so another thing that's common when we're working with architectural photos is that if we adjust for the vertical distortion of the image, we end up losing a lot of information. And if we want to keep the proportion of our images the same, this can present a challenge. So Generative Fill is also very useful for this.

Let's start with our marquee tool, and we're going to make a rectangular selection around this area where we want it to fill it up, and we're going to pick Generative Fill. And we're not even using a prompt for this, and it actually does a pretty good job. Let's see what the second option looks like. I think I like that one better.

Now let's do the same on this side. Again, I'm not using any prompts for this. And this time it didn't do such a good job because that building is actually a landmark, so I don't want to change a lot. So what I'm going to do is I'm going to make a smaller selection just on the edge of the building, and let's see if the Contextual Fill works better this way. And this looks very nice. Very close to what's there. So now I just need to finish with this portion on the bottom, and we should be good to go. Let's see. Generative Fill, and there we go. So as you can see, this is much faster.

Let's try one last tool. The last thing that we're going to be doing is sky replacement. So for this we're going to go to Edit and we're going to click on Sky Replacement. And we don't even need to mask our sky manually. It uses AI and it automatically detects the sky. So we can pick one of the preloaded skies that it has, or we can go in and add some new skies that we have. And that's what we're doing here, and we're going to pick this blue cloudy one.

And it usually does a pretty good job of adjusting the brightness and the temperature to the existing light conditions in our photo, but we can adjust it more if we want. And then when we click on OK, we're going to see our new layer with all of the adjustments added. And then we can use our Opacity to make it mix a little bit more with the original sky, or we can even add a new mask and paint out some of the parts to bring back some of the parts that we like most in our original sky. And here's our final image. As you can see, this can speed up photo retouching quite a bit.

We've seen that we can get very good AI images in the first part of this class, but now we're going to be working on generating video. For this we're going to be using Runway, and you can try it out for free in their website, runwayml.com. And you get a fair amount of credits to test out all of their tools. The only thing that you can't do without a paid plan is get the highest resolution video.

So if you go into their page, you're going to see that they have tools for 3D generation, image generation, and videos. That's the one that we're going to be focusing for this class, and if you click there, you'll see three tools. You have Video to Video, Text or Image to Video, and Frame Interpolation. We're going to start with Video to Video.

For this one, we created this white box really simple model and we did this small animation of a cabin in the woods. And I want to turn it into an animated concept sketch, so we're going to go to Video to Video and then we're going to upload our animation over here. And on the right we can pick an image that we want it to use as a reference to style our video.