Frame Interpolation Large Motion (FILM)
Overview
"The official Tensorflow 2 implementation of our high quality frame interpolation neural network. We present a unified single-network approach that doesn't use additional pre-trained networks, like optical flow or depth, and yet achieve state-of-the-art results. We use a multi-scale feature extractor that shares the same convolution weights across the scales. Our model is trainable from frame triplets alone."
Prerequisites
Miniconda
https://docs.conda.io/en/latest/miniconda.html
Git
https://git-scm.com/download/win
Setup
Get Frame Interpolation source codes:
1git clone https://github.com/google-research/frame-interpolation.git
cd into the cloned git repository, for example:
1cd C:\Users\trima\Documents\GitHub\frame-interpolation
Create the Miniconda virtual environment:
1conda create -n frame-interpolation pip python=3.9
Activate the Miniconda virtual environment:
1conda activate frame-interpolation
Install requirements:
1pip install -r requirements.txt
If FILM is not running on your GPU it's because cudann is missing. Install it with:
1conda install -c anaconda cudnn
Usage
From the Conda Shell, cd to the FILM directory:
Open File Explorer at this directory and copy the frames you want to interpolate to the "photos" folder.
1start .
Place the images you would like to interpolate in the "photos" directory and run this command to begin interpolating them:
1python -m eval.interpolator_cli --pattern "photos" --model_path pretrained_models\film_net\Style\saved_model --times_to_interpolate 1 --output_video
Rename frames
When rendering thousands of frames, the file names will unfortunately be formatted as frame_001.png, frame_002.png, frame_1000.png, frame_10000.png, etc. ffmpeg can't make sense of this when the frames are being assembled into a video, so every frame needs to be renamed. Here's a script that does this:
1import os
2import re
3
4folder_path = r"C:\Users\Tristan\Documents\GitHub\frame-interpolation\photos\interpolated_frames" # replace with the path to your folder
5
6# Define a function to extract the number from the filename
7def extract_number(filename):
8 match = re.search(r'(\d+)', filename)
9 return int(match.group(1)) if match else 0
10
11# Get all files in the directory
12all_files = [f for f in os.listdir(folder_path) if os.path.isfile(os.path.join(folder_path, f))]
13
14# Filter only the .png files
15image_files = [f for f in all_files if f.endswith('.png')]
16
17# Sort using the extract_number function as the key
18image_files.sort(key=extract_number)
19
20# Rename each file
21for idx, file_name in enumerate(image_files):
22 new_name = "{:05d}.png".format(idx)
23 old_file_path = os.path.join(folder_path, file_name)
24 new_file_path = os.path.join(folder_path, new_name)
25
26 os.rename(old_file_path, new_file_path)
27
28print("Renaming complete!")
Assemble frames with ffmpeg
1ffmpeg -framerate 60 -i %05d.png -c:v libx264 -pix_fmt yuv420p interpolated-60fps.mp4
Batch Processing
Enter this For loop in the Anaconda Shell to iterate through a folder of folders containing video frames and batch interpolate all of them.
1FOR /D %i IN ("C:\Users\<user>\<some>\<directory>\*") DO python -m eval.interpolator_cli --pattern "%i" --model_path pretrained_models\film_net\Style\saved_model --times_to_interpolate 1 --output_video
Use this batch script to copy all "interpolated.mp4" files to the same directory as the script and rename them in sequential order.
1@echo off
2setlocal enabledelayedexpansion
3set /a "count=0"
4
5for /r "." %%a in ("*interpolated.mp4") do (
6 set /a "count+=1"
7 set "filename=00!count!.mp4"
8 copy "%%a" "!filename:~-6!"
9)
10
11echo Finished copying !count! files.