Video Significant Frames Extraction Option

[pl-shernandez]

I'm loving the new video processing functionality!

Something really cool to enhance it would be to extract just significant frames with an option. I found, well GPT-4o, that there is functionality in FFMPEG to do this with a threshold argument. Had previously used the python OpenCV library to reduce the number of frames being fired into the LLM.

Running this on the CLI I wasn't getting significant frames until my threshold was around .001 so it's very sensitive. For a 10 minute video with only one significant change then you may only get a handful of images.

/**
* Detects significant scene changes in a video using FFmpeg's `select` filter.
*
* @param {string} inputPath - Path to the input video file.
* @param {string} outputPattern - Output file pattern (e.g. 'frames-%04d.png').
* @param {number} sceneThreshold - Float in [0,1] that determines scene-change sensitivity.
*/
function detectSceneChanges(inputPath, outputPattern, sceneThreshold = 0.3) {
 const command = ffmpeg(inputPath)
   // The `select='gt(scene, threshold)'` logic picks frames that exceed scene threshold.
   // The `showinfo` filter logs frame info (including scene change info) to stderr.
   .outputOptions([
     `-vf`,
     `select='gt(scene,${sceneThreshold})',showinfo`,
     `-vsync`,
     `0`
   ])
   // Save each triggered frame as an image (PNG in this example)
   .output(outputPattern)
   // Optional: If you only want the scene detection info without saving images,
   // you can skip the output() call or output to null sink (on Linux).
   .on('start', commandLine => {
     console.log('Spawned FFmpeg with command:', commandLine);
   })
   .on('codecData', data => {
     console.log(`Input is ${data.audio} audio with ${data.video} video`);
   })
   .on('stderr', stderrLine => {
     // The `showinfo` filter logs to stderr, so you can parse lines here for scene change info
     console.log('FFmpeg stderr output:', stderrLine);
   })
   .on('end', () => {
     console.log('Processing finished!');
   })
   .on('error', (err, stdout, stderr) => {
     console.error('An error occurred:', err.message);
     console.error('ffmpeg stdout:\n', stdout);
     console.error('ffmpeg stderr:\n', stderr);
   });

 command.run();
}

[github-actions]

Here's a brief enhancement suggestion for the feature:

• Consider adding support for different image formats besides PNG, providing users with an option to specify their preferred format (e.g., JPEG, TIFF).
• If performance becomes an issue, particularly with longer videos or higher sensitivity thresholds, explore optimization techniques in the FFmpeg command.
• Adding support for batch processing of multiple videos could make the feature more versatile and user-friendly.

This will help broaden the utility of the significant frames extraction option.

AI-generated content by issue-reviewer may be incorrect

[pelikhan]

Here's a PR that does the ffmpeg magic to render infra-frames/keyframes.

#1019

[pelikhan]

Try 1.93.0 and sceneThreshold option (or also keyframes)