In my recent experiments with AI integration in filmmaking, I've focused on adding background extras to existing footage while maintaining creative control.
AI image and video generation tools have tremendous potential, but filmmakers need more precise control than current systems typically allow. Text prompting alone creates a significant gap between directorial vision and AI output, making it inadequate for professional filmmaking workflows where nuanced control is essential.
My workflow leverages ComfyUI's tools including depth mapping and pose control to achieve greater artistic direction over AI-generated elements. These features allow for precise placement and integration of virtual extras that respect the original footage's perspective, lighting, and atmosphere.
This approach bridges the gap between AI's capabilities and the filmmaker's craft, enabling us to harness the technology while preserving our creative authority rather than surrendering artistic decisions to algorithms.
The process combines several tools:
- ComfyUI provided essential control mechanisms
through depth maps,masking, and pose control
- fSpy extracted perspective data from original footage
for accurate 3D integration
- Blender allowed precise blocking and placement of AI elements
- Photoshop facilitated seamless blending of the components
- Runway transformed the composite still image into video.
- DaVinci Resolve Fusion used final compositing
through depth maps,masking, and pose control
- fSpy extracted perspective data from original footage
for accurate 3D integration
- Blender allowed precise blocking and placement of AI elements
- Photoshop facilitated seamless blending of the components
- Runway transformed the composite still image into video.
- DaVinci Resolve Fusion used final compositing
I discovered that AI-generated background elements were particularly effective when positioned out of focus.
Technical challenges were, Runway's 10-second video generation limit and difficulties producing video without unwanted camera movement.
Thanks to Valofirma for camera equipment for the test.
