ClassifyVideo does not work and other issues

[zoinkydoink]

Created a dumb console app (.NET 6) with the following
static void Main(string[] args)
{

        Console.WriteLine("Hello, World!");
        var nsfwSpy = new NsfwSpy();
      
        var result = nsfwSpy.ClassifyVideo(@"D:\movies for checking\Edit.mp4");

       
        Console.ReadKey();
    }

It seems that the code execution never comes out of using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
No exceptions or anything, i even left it running overnight thinking it might be taking long (the Edit.mp4 is first 10 minute clip of God Father Part 2), I cut it only to 10 min thinking it might be the size/time of all the movie being at (3h:22m run time and 1.4GB).
Here is what my console looks like as the program is running.

I have a 3070 running, with all the libraries installed properly (I think)

Hello, World!
2023-11-29 09:49:57.528567: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2023-11-29 09:49:57.727815: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-29 09:49:57.739943: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2602ddd4620 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-11-29 09:49:57.740026: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-11-29 09:49:57.741758: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2023-11-29 09:49:57.758938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2023-11-29 09:49:57.759034: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2023-11-29 09:49:58.152471: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2023-11-29 09:49:58.200779: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2023-11-29 09:49:58.228869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2023-11-29 09:49:58.447674: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2023-11-29 09:49:58.677289: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2023-11-29 09:49:58.928462: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
>>>>>2023-11-29 09:49:58.928755:** I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2023-11-29 09:51:52.426382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
>>>>>>2023-11-29 09:51:52.426450:** I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2023-11-29 09:51:52.426524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2023-11-29 09:51:52.426887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6591 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:01:00.0, compute capability: 8.6)
2023-11-29 09:51:52.429524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2606633c450 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-11-29 09:51:52.429562: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 3070, Compute Capability 8.6

I think u should be upgrading to .NET 6 at least (preferably 8 as 2.0 is deprecated), also note the time in bold in the above, it took a while for it to get to var collection = new MagickImageCollection(video, MagickFormat.Mp4)

Seems like the SciSharp, Magick, and Microsoft.ML can also be upgraded to later versions

I also wanted to do a PR where I add a progress event to the video processing so one can know how its going, below is the code if u wanna incorporate.

        //Code Added
        public delegate void UpdateVideoProgressDelegate(int currentFrame, int totalFrames);
        public event UpdateVideoProgressDelegate UpdateVideoProgress;
        
        public NsfwSpyFramesResult ClassifyVideo(byte[] video, VideoOptions videoOptions = null)
        {
            if (videoOptions == null)
                videoOptions = new VideoOptions();

            if (videoOptions.ClassifyEveryNthFrame < 1)
                throw new Exception("VideoOptions.ClassifyEveryNthFrame must not be less than 1.");

            var results = new ConcurrentDictionary<int, NsfwSpyResult>();

            var watch = Stopwatch.StartNew();
            using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
            {
                Console.WriteLine($"Loading video took {watch.Elapsed} to load video into collection of images");
                collection.Coalesce();
                var frameCount = collection.Count;

                Parallel.For(0, frameCount, (i, state) =>
                {
                    if (i % videoOptions.ClassifyEveryNthFrame != 0)
                        return;

                    if (state.ShouldExitCurrentIteration)
                        return;

                    var frame = collection[i];
                    frame.Format = MagickFormat.Jpg;

                    var result = ClassifyImage(frame.ToByteArray());
                    results.GetOrAdd(i, result);

                    //Code Added
                    if (videoOptions.EmitProgress)
                    {
                        UpdateVideoProgress?.Invoke(i, frameCount);
                    }

                    // Stop classifying frames if Nsfw frame is found
                    if (result.IsNsfw && videoOptions.EarlyStopOnNsfw)
                        state.Break();
                });
            }

            var resultDictionary = results.OrderBy(r => r.Key).ToDictionary(r => r.Key, r => r.Value);
            var gifResult = new NsfwSpyFramesResult(resultDictionary);
            return gifResult;
        }

Also how long does it take to process a 10 min vid in general.

[d00ML0rDz]

Hey, thanks for the issue.

In regards to the application hanging, from what I understand, when we get to this point:

using (var collection = new MagickImageCollection(video, MagickFormat.Mp4))
{
    collection.Coalesce();

the Magick library converts the video into individual frames which get saved to a temp folder which only runs against the CPU, as last I checked they didn't have GPU support. This is not a quick process. For a 10 minute video at 24 frames per second, that would be over 14000 frames. I did a quick test on my laptop and a 180 frame video took about a minute to process. For 14000, that could take an hour. From other tests I've done, NsfwSpy's classification is pretty fast, with 10,000 images not taking more than a few minutes, so I'm pretty sure the video frame extraction is the bulk of the processing.

I'm sure this process could be faster, but I don't really have the time to look into other methods of extracting the frames at the moment. Because the extraction process is the time consuming part, the UpdateVideoProgress you added won't really provide an accurate idea of how quickly the frames are being processed as the extraction process takes so long before it.

Regarding the depedency updates, if I find the time, I can try working that in to an update 😊

[d00ML0rDz]

This thread talks about some of the memory usage requirements for extracting the frames from a 521 frame GIF - https://stackoverflow.com/questions/69059486/imagemagick-convert-ultra-slow

If you can find a better way to extract the frames from a longer video, I'd be keen to hear and then we could work that into NsfwSpy 👍

[zoinkydoink]

I poked around a bit and it seems that direct ffmpeg extraction is probably the fastest/best way and it uses GPU
I ran the following command ffmpeg -c:v h264_cuvid -i "D:\movies for checking\The.Godfather.1972.1080p.BrRip.x264.BOKUTOX.YIFY.mp4" -vf fps=2 "c:\temp\gf1\output-%04d.png"

which took few minutes to extrct 21k frames (2 f rames per second) of a almost 3 hour movie.
Given the nature of the detection this library is doing, I dont think more than 2 frame per second is needed, even 1 is enough.

This thread talks about ffmpeg and breaking into images, i found the second command (one above) to be faster
https://stackoverflow.com/questions/64485121/extracting-frames-with-nvidia-gpu

I would think the next step in this library is to remove MagickImage completely and rely on ffmpeg (since it seems to be using it anyways since its a required install according to your docs).

Next Steps

1. Upgrade libraries to latest (SciSharp, ML.Net, etc)
2. Upgrade CUDA libraries to latest instead of the older versions
3. Upgrade to .NET 6+ (preferably 8.0)
4. Incorporate ffmpeg directly with gpu support of extracting frames and remove MagickImage

[somegenericdev]

A different approach could be using just one frame every N seconds. This behavior could be controlled by a parameter, e.g. bool computeEntireVideo.

I've hacked something like this together in my application, I'm sure I could replicate it in the library if you think it's a good idea.