Automate video highlights with Ruby and FFmpeg
Finding useful moments in long footage takes time. Ruby can automate scene-change detection, extract candidate clips, and assemble them into a review reel with FFmpeg. This is a visual-change heuristic, not a system that understands which moments are important.
Introduction to automated video highlights
The workflow detects large changes between adjacent frames, selects timestamps far enough apart, and encodes short clips around them. A cut, flash, or slide transition may be a useful candidate, but a goal or spoken insight may have no scene cut. Review the generated reel before publishing it.
Setting up Ruby and FFmpeg
Use Bash on Linux with Ruby, FFmpeg, and its companion ffprobe on PATH. This walkthrough was
tested with Ruby 3.4.10 and FFmpeg/ffprobe 9.0.1. These are tested versions, not exact installation
requirements. Use a current patch of Ruby 3.4 from the
Ruby installation guide, and check its
maintenance status.
On Ubuntu/Debian, install FFmpeg and ffprobe together:
sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
The example uses Ruby’s standard libraries without a third-party FFmpeg wrapper. Before saving the scripts, check that Ruby can load them and inspect your FFmpeg build’s filters and encoders:
env -u RUBYOPT -u RUBYLIB -u BUNDLE_GEMFILE ruby -v &&
env -u RUBYOPT -u RUBYLIB -u BUNDLE_GEMFILE ruby -e "require 'fileutils'; require 'json'; require 'open3'" &&
ffmpeg -version && ffprobe -version && ffmpeg -filters && ffmpeg -encoders
Version compatibility
Your FFmpeg build needs the select, metadata, atrim,
asetpts, and concat filters and the libx264 and aac encoders. The same recipe also runs
with FFmpeg 6.1.1; use a patched package for your distribution. The invocation clears inherited
Ruby loader and Bundler
settings, so running from an enclosing Ruby project does not select that project’s gems.
Choose one known-good local video with a fixed frame size and SDR color. The script selects its first video track and, when present, its first audio track. Audio should cover the selected video intervals. Untrusted uploads, damaged recordings, and files that change resolution midstream need separate validation and an isolated worker; this CLI does not supply that boundary. FFmpeg can recover from a damaged packet, print a warning, and still return success. A completed batch does not validate the source’s integrity; review diagnostics and the generated cuts.
Extracting key moments using FFmpeg filters
FFmpeg’s scene score estimates visual
differences between frames. For a manual probe in a fresh working directory, this command writes
selected frame timestamps to timestamps.txt:
ffmpeg -nostdin -n -threads 2 -filter_threads 1 -i input.mp4 -map 0:v:0 \
-vf "select='gt(scene,0.3)',metadata=print:file=timestamps.txt" \
-an -f null -
Keep scene detection and clip extraction on FFmpeg's input-relative timeline. Do not reset the
video stream's timestamps independently: audio may start before the first video frame. Detection
and extraction both select the first video track explicitly. The filter's
text file is separate from FFmpeg's regular output-file checks: do not reuse an existing timestamps.txt.
The Ruby implementation creates a private job directory for it.
Implementing Ruby scripts for automation
Save this class as highlight_extractor.rb. It keeps generated filenames fixed and passes input
paths as separate process arguments, so paths do not enter shell or filter syntax. A failed stage
raises an error rather than silently producing a successful batch report.
require 'fileutils'
require 'json'
require 'open3'
class HighlightExtractor
def initialize(input_file, output_dir)
@input = File.expand_path(input_file)
raise ArgumentError, 'Input must be a regular file' unless File.file?(@input)
@output = File.expand_path(output_dir)
end
def generate(threshold: 0.3, clip_duration: 5.0, pre_roll: 2.0, min_gap: 10.0)
unless threshold.finite? && (0.0..1.0).cover?(threshold) &&
clip_duration.finite? && clip_duration > 0 &&
pre_roll.finite? && pre_roll >= 0 && min_gap.finite? && min_gap >= clip_duration
raise ArgumentError, 'Invalid scene or clip settings'
end
# Refuse earlier job outputs before taking ownership of this directory.
Dir.mkdir(@output, 0700)
begin
duration = video_duration
selected = detect_scene_changes(threshold).each_with_object([]) do |time, result|
next if time >= duration
result << time if result.empty? || time - result.last >= min_gap
end
# Cap output count so a noisy video cannot create an unlimited clip set.
lengths = []
clips = selected.first(20).each_with_index.map do |time, index|
start = [time - pre_roll, 0.0].max
length = [clip_duration, duration - start].min
lengths << length
name = format('highlight_%03d.mp4', index + 1)
execute('-i', @input, '-ss', start.to_s, '-t', length.to_s,
'-map', '0:v:0', '-map', '0:a:0?', '-c:v', 'libx264', '-preset', 'fast',
'-crf', '23', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-threads', '2', name)
name
end
create_highlight_compilation(clips, lengths) unless clips.empty?
clips.length
rescue StandardError
FileUtils.remove_entry(@output)
raise
end
end
private
def video_duration
output, status = Open3.capture2('ffprobe', '-v', 'error', '-show_entries',
'format=duration:stream=codec_type', '-of', 'json', @input)
raise IOError, 'Cannot inspect input video' unless status.success?
info = JSON.parse(output)
@has_audio = info.fetch('streams').any? { |stream| stream.fetch('codec_type') == 'audio' }
duration = Float(info.fetch('format').fetch('duration'))
raise ArgumentError, 'Invalid video duration' unless duration.finite? && duration > 0
duration
end
def execute(*arguments)
unless system('ffmpeg', '-nostdin', '-n', '-threads', '2', '-filter_threads', '1',
'-filter_complex_threads', '1',
*arguments, chdir: @output)
raise IOError, 'FFmpeg processing failed'
end
end
def detect_scene_changes(threshold)
execute('-i', @input, '-map', '0:v:0', '-vf',
"select='gt(scene,#{threshold})',metadata=print:file=timestamps.txt",
'-an', '-f', 'null', '-')
File.foreach(File.join(@output, 'timestamps.txt')).each_with_object([]) do |line, times|
match = line.match(/pts_time:([0-9.eE+-]+)/)
times << Float(match[1]) if match
end.uniq.sort
end
def create_highlight_compilation(clips, lengths)
list = clips.each_with_index.map do |name, index|
"file '#{name}'\nduration #{lengths[index]}\n"
end.join
File.write(File.join(@output, 'filelist.txt'), list)
arguments = ['-copyts', '-f', 'concat', '-safe', '1', '-i', 'filelist.txt']
if @has_audio
clips.each { |name| arguments.concat(['-i', name]) }
filters = clips.each_index.map do |index|
"[#{index + 1}:a:0]atrim=duration=#{lengths[index]},asetpts=PTS-STARTPTS[a#{index}]"
end
inputs = clips.each_index.map { |index| "[a#{index}]" }.join
filters << "#{inputs}concat=n=#{clips.length}:v=0:a=1[audio]"
arguments.concat(['-filter_complex', filters.join(';'), '-map', '0:v:0', '-map',
'[audio]', '-c:v', 'copy', '-c:a', 'aac'])
else
arguments.concat(['-map', '0:v:0', '-c:v', 'copy'])
end
execute(*arguments, 'highlight_reel.mp4')
end
end
The concat demuxer needs matching streams and time bases. These clips share a source and encoder settings; the list contains only generated basenames, so safe-path mode stays enabled. Explicit durations place each video clip on the reel’s timeline. The reel copies the encoded video. When audio is present, it decodes each clip separately, trims audio to the requested clip length, and joins it with the concat filter before re-encoding to AAC. This keeps clip encoder padding from accumulating as audio delay. This review reel is not a gapless audio master or a concatenator for unrelated files. Listen at the joins before publishing.
The libx264 encoder uses CRF 23 as a
quality target, without a bitrate cap. A smaller value generally retains more detail and increases
file size. The fast preset controls encoding effort, not which scenes are selected.
The output-side -ss option decodes and discards footage
before each clip’s start time. This straightforward decoding path does not depend on usable seek
indexes, but repeats work for late highlights. For long, indexed recordings, benchmark input-side
accurate seeking and verify clip boundaries and audio/video offsets before changing that behavior.
Save this caller beside the class as generate_highlights.rb:
require_relative 'highlight_extractor'
abort 'Usage: ruby generate_highlights.rb <input-video> <new-output-directory>' unless ARGV.length == 2
begin
count = HighlightExtractor.new(*ARGV).generate
puts(count.zero? ? 'No scene changes selected' : "Created #{count} candidate highlights")
rescue StandardError
warn 'Highlight generation failed. Check the input, destination, and FFmpeg installation.'
exit 1
end
Run it from your terminal:
env -u RUBYOPT -u RUBYLIB -u BUNDLE_GEMFILE \
ruby generate_highlights.rb my_video.mp4 my_highlights_folder
Keep both scripts together and replace my_video.mp4 with your input path. Quote paths that contain
spaces. A successful run reports the candidate count and writes highlight_001.mp4,
subsequent numbered clips, and highlight_reel.mp4 inside my_highlights_folder. Open the reel in
your video player. Each default clip starts 2 seconds before its selected cut, clamped to the input
start, and lasts up to 5 seconds. Cuts are selected chronologically at least 10 seconds apart, up
to a maximum of 20 clips.
An existing output directory is refused and remains untouched; choose a new name for a retry. A processing error after creating the job directory removes that job’s partial outputs. Fix the input or installation, then repeat the same command. A valid video with no selected scene changes prints No scene changes selected, leaves its timestamp file, and succeeds without clips or a reel. Local FFmpeg diagnostics can include filenames; keep sensitive paths out of shared logs.
Performance optimization
Run a short representative recording first and check the resulting cuts. The twenty-clip cap bounds generated clips, not input duration, the full scene-analysis pass, or output size. Leave enough disk space for both the individual clips and the reel. Decoder and encoder threads are bounded here; running several jobs still multiplies their CPU and storage use.
Optimizing highlight selection
A lower scene threshold selects more changes, including noise; a higher threshold can miss subtle
transitions. The min_gap setting keeps selected timestamps at least that many seconds apart and
must be at least clip_duration. Selection is chronological, not a ranking of the most exciting
moments. Adjust pre-roll and duration after reviewing your content.
Practical examples and use cases
Scene-change candidates can help review edited interviews, slide transitions, and montage footage. Do not assume scene detection recognizes goals, wins, specific speakers, or spoken conclusions. Those tasks require additional signals and separate validation. The generated reel is a starting point for editorial review.
Conclusion and potential enhancements
If the reel misses moments you need, lowering the scene threshold may only add more visual noise. Add audio analysis or a validated event detector when your selection depends on speech or activity within an unchanged shot.
For managed processing, Transloadit's video encoding service provides FFmpeg-based Robots for encoding and media workflows.