Perform pixel format conversion in the GPU?

[teacherguy2020]

I could be wrong, but it seems that the virtual cam is using the CPU, even though I have asked OBS to prioritize my eGPU. Is it possible to handle the processing on the eGPU?

[johnboiles]

This might be possible! The most CPU intensive part of this plugin is the conversion of pixel formats that needs to happen from NV12 (or whatever pixel format you're using in OBS) to the YUV422 format that macOS wants. Pixel format conversion should be obviously parallelizable. However, you might lose any performance gains by needing to copy frame buffers back and forth from ram to the GPU memory (though this is where my knowledge is lacking).

@jp9000 has perhaps thought about this more.

[johnboiles]

But to be clear, pixel format conversion currently happens in the CPU in OBS as far as I can tell from reading the code.

[jp9000]

If the program had 422 support you could just set the program to 422, but we don't have that right now so it's just cpu conversion via ffmpeg. The conversion should be reasonably fast though. I'm stretched too thin so I can't write extended GPU operations for alternate formats or alternate outputs right now (or probably any time soon). Not really sure if it's worth it.

Not sure why 422 is needed on macOS though, guess it must not support NV12 or I420 or something from the sound of it. On Windows I was able to get away with NV12/I420.

[teacherguy2020]

OK good to know. If I am only using the Virtual Camera (not streaming, not recording), which settings would help, if any?

[johnboiles]

Thank you for your perspective @jp9000! I'm also unsure if it's worth it -- I suspect the extra framebuffer copies might eat up the performance gains of doing the conversion on the GPU. But if someone wanted to give it a shot, it would be an interesting experiment!

I wasn't able to get macOS to recognize any of the native OBS formats. The only format that worked for me on macOS was yuv422 (VIDEO_FORMAT_UYVY in OBS). More details in #102

That's great that you were able to use NV12 directly on Windows!

@teacherguy2020 there aren't any settings you can tweak at this time to increase performance. Reducing the number of pixels (reducing resolution, framerate) will improve performance in exchange for quality, but that's it.

[jp9000]

Haha, I'm terribly sorry, I suppose that wasn't a particularly useful bit of information, the way you responded to that was sort funny, almost like "well, great for you buddy!" (I know you didn't mean it that way, I'm just teasing)

The ideal solution would be to implement native 422 output in OBS, that way there would be no conversion for it. It probably shouldn't be too difficult, although personally I'm just burned out, it's basically another thing on my to-do list at this point. I apologize that we haven't already implemented it.

[johnboiles]

Haha no worries I'm sincerely glad NV12 worked for you :)

So native 422 is something loosely on the OBS roadmap? How difficult do you think that project would be? Maybe we could rally someone to help out with that

[richardbrockie]

Thank you for your perspective @jp9000! I'm also unsure if it's worth it -- I suspect the extra framebuffer copies might eat up the performance gains of doing the conversion on the GPU. But if someone wanted to give it a shot, it would be an interesting experiment!

I wasn't able to get macOS to recognize any of the native OBS formats. The only format that worked for me on macOS was yuv422 (VIDEO_FORMAT_UYVY in OBS). More details in #102

That's great that you were able to use NV12 directly on Windows!

@teacherguy2020 there aren't any settings you can tweak at this time to increase performance. Reducing the number of pixels (reducing resolution, framerate) will improve performance in exchange for quality, but that's it.

Not sure if this is specific to the Mac VirtualCam project:

If there were a way to offload more OBS processing to the GPU I would definitely appreciate it. My typical usage case is having my macBook docked to an eGPU using TB3. With OBS using my CamLink4k as a source there is some offloading to the eGPU, but there is also high CPU usage which ramps up the temperature and therefore CPU fan substantially.

[gxalpha]

I think its getting too far into OBS territory here, maybe ask in the OBS Discord or open an issue at https://github.com/obsproject/obs-studio (but only if you don't get an answer on Discord)

[johnboiles]

This might not be beneficial, even if it's possible since it'd mean you'd need to copy the frame to the GPU. Instead it'd just be better to add native 422 support to OBS and then you wouldn't need a pixel format transformation (#102)