Posts

Frame Interpolation Large Motion (FILM)

A comprehensive guide to setting up and using Google’s Frame Interpolation Large Motion (FILM) TensorFlow implementation, enabling the creation of smooth animations by generating intermediate frames between existing images using deep learning techniques.

Python #ai #video #animation #tensorflow #machine-learning

"The official Tensorflow 2 implementation of our high quality frame interpolation neural network. We present a unified single-network approach that doesn’t use additional pre-trained networks, like optical flow or depth, and yet achieve state-of-the-art results. We use a multi-scale feature extractor that shares the same convolution weights across the scales. Our model is trainable from frame triplets alone."

Prerequisites

Miniconda

https://docs.conda.io/en/latest/miniconda.html

Git

https://git-scm.com/download/win

Setup

Get Frame Interpolation source codes:

git clone https://github.com/google-research/frame-interpolation.git

cd into the cloned git repository, for example:

cd C:\Users\trima\Documents\GitHub\frame-interpolation

Create the Miniconda virtual environment:

conda create -n frame-interpolation pip python=3.9

Activate the Miniconda virtual environment:

conda activate frame-interpolation

Install requirements:

pip install -r requirements.txt

If FILM is not running on your GPU it’s because cudann is missing. Install it with:

conda install -c anaconda cudnn

Usage

From the Conda Shell, cd to the FILM directory:

Open File Explorer at this directory and copy the frames you want to interpolate to the “photos” folder.

start .

Place the images you would like to interpolate in the “photos” directory and run this command to begin interpolating them:

python -m eval.interpolator_cli --pattern "photos" --model_path pretrained_models\film_net\Style\saved_model --times_to_interpolate 1 --output_video

Rename frames

When rendering thousands of frames, the file names will unfortunately be formatted as frame_001.png, frame_002.png, frame_1000.png, frame_10000.png, etc. ffmpeg can’t make sense of this when the frames are being assembled into a video, so every frame needs to be renamed. Here’s a script that does this:

import os
import re

folder_path = r"C:\Users\Tristan\Documents\GitHub\frame-interpolation\photos\interpolated_frames"  # replace with the path to your folder

# Define a function to extract the number from the filename
def extract_number(filename):
    match = re.search(r'(\d+)', filename)
    return int(match.group(1)) if match else 0

# Get all files in the directory
all_files = [f for f in os.listdir(folder_path) if os.path.isfile(os.path.join(folder_path, f))]

# Filter only the .png files
image_files = [f for f in all_files if f.endswith('.png')]

# Sort using the extract_number function as the key
image_files.sort(key=extract_number)

# Rename each file
for idx, file_name in enumerate(image_files):
    new_name = "{:05d}.png".format(idx)
    old_file_path = os.path.join(folder_path, file_name)
    new_file_path = os.path.join(folder_path, new_name)
    
    os.rename(old_file_path, new_file_path)

print("Renaming complete!")

Assemble frames with ffmpeg

ffmpeg -framerate 60 -i %05d.png -c:v libx264 -pix_fmt yuv420p interpolated-60fps.mp4

Batch Processing

Enter this For loop in the Anaconda Shell to iterate through a folder of folders containing video frames and batch interpolate all of them.

FOR /D %i IN ("C:\Users\<user>\<some>\<directory>\*") DO python -m eval.interpolator_cli --pattern "%i" --model_path pretrained_models\film_net\Style\saved_model --times_to_interpolate 1 --output_video

Use this batch script to copy all “interpolated.mp4” files to the same directory as the script and rename them in sequential order.

@echo off
setlocal enabledelayedexpansion
set /a "count=0"

for /r "." %%a in ("*interpolated.mp4") do (
    set /a "count+=1"
    set "filename=00!count!.mp4"
    copy "%%a" "!filename:~-6!"
)

echo Finished copying !count! files.