How To Use Runway’s New Director Mode and AI Custom Camera Controls

In a turn-of-phrase that might be the epitome of 2023, we can now say that this new release from Runway ML might be the best tool for any aspiring “AI filmmakers” out there. Known for its AI photo and video generation, RunwayML has quickly become one of the better tools for AI video editing.

If you are someone who is interested in exploring this new AI world, Runway has been hard at work pushing out new innovations and improvements. With some new AI color grading features as well as a mobile app, it is quickly becoming a staple tool for filmmakers.

With Runway Gen-2 in particular, there’s been a stream of new features, the latest of which might be its biggest yet. AI custom camera controls promise to be a real game changer in AI filmmaking as it allows for new directional camera movement capabilities.

Let’s take a look at these new AI custom camera controls, including Runway Gen-2’s director mode, directional looping, a motion slider feature as well a host of other AI magic tools, to see if these AI features might be worth checking out.

---

Runway Gen-2 Director Mode

Probably the most interesting new feature added to Runway’s AI offering in this version Gen-2, Runway’s Director Mode unlocks several new custom camera controls that add directional movement. These movements cover horizontal, vertical, zoom, and even a roll control feature. After generating your director mode control, you can then choose to either extend it further or add a different one.

While still pretty rudimentary in terms of actual direction and camera movement, this is indeed a major step forward for generative AI as it allows for much easier and quicker controls of these basic camera moves.

Here’s a summary of how to use this feature in the Runway app:

• Add an image to Gen-2, click on the “Motion” button, and click “Custom camera control”
• Choose one or multiple directions to move your camera
• Then, adjust “Speed” for your motion
• Clicking the “Extend” button will reuse the existing camera direction for your extensions to continue the motion seamlessly, up to 16s
• For even more control, try using the Image + Description feature, which will allow you to dial in a more specific prompt

Runway Gen-2 Directional Looping Video

On top of this new director mode in Runway, the AI app has also unveiled a directional looping video feature which is a quick trick to use to turn your short AI-generated clips into longer, looping videos, which you can then export either as a video file or as a GIF.

Here’s a summary of how to use this feature in the Runway app:

• Add an image to Gen-2
• Navigate to the advanced camera controls section to use Director Mode
• Turn on Upscale
• Choose a camera path and generate a video for each direction that you want to present in your video
• Pro tip: the next step is easier if you rename outputs with keywords or directions
• When editing video files, play the clip forwards; when it reaches the end, duplicate the video and play it backwards, so that it resets back to the first frame of the video
• Play with keyframes to make things feel more dynamic and snappy
• For even more loop-ability, try exporting your video as a GIF

Runway Gen-2 Motion Slider

There’s also another new feature called motion slider which allows users to control the length and intensity of the motion in their videos.

Here’s a summary of how to use this feature in the Runway app:

• Open Gen-2 and input a prompt
• Increase or decrease motion value by typing or dragging the slider
• The default is 5; the range is 0-10 (0 = less motion, 10 = more motion)
• Click Generate

Runway Gen-2 Workflow w/ AI Magic Tools

Finally, Runway has a small array of other AI magic tools that, when combined, can allow users to create generated videos that have a range of camera moves and frame interpolation.

Here’s a summary of how to use this feature in the Runway app:

• Start with Text to Image
  • generate an image you want to bring to life with Gen-2, then add it to your assets
• Open Gen-2, and use that image as your reference image alongside a text prompt
• Try bringing your video into Super-Slow Motion to extend the clip time
• Or drop your video into Text to Color Grade (LUT) to give it a color treatment
• Bring your video into Inpainting to clean up any artifacts or parts of the frame
• Toss it into Green Screen to extract your creation from the scene or create alpha mattes
• The possibilities are endless with 30 AI Magic Tools!

And that’s pretty much it, at least until a new Runway Gen-3 comes out with what could undoubtedly be more advanced features. For now, these tools are a nice introduction to generative AI and a safe way to explore how AI can be used for video or filmmaking.