Adding support for vulkan decode extensions

[FelixLamontagne2]

Description

Hi,

I have a Vulkan application that currently "works", but when trying to launch it with RenderDoc I'm missing the decode extensions in the physical device. When launching without RenderDoc the extensions are there and the decoding "works". I say "works" because I'm getting a green image and would love to debug this with RenderDoc 😅

It would be amazing if RenderDoc was able to actually debug the decoding itself, but this is probably a major undertaking so perhaps as a first step just allowing these extensions so that I can inspect the resulting Image would be enough?

I'd love to provide more details, but I'm unsure about the internals of how RenderDoc works so feel free to ask me any additional questions.

Environment

• RenderDoc version: 1.34
• Operating System: Windows
• Graphics API: Vulkan 1.3.290

Extensions:

1. VK_KHR_video_decode_h264
2. VK_KHR_video_decode_h265
3. VK_KHR_video_decode_av1
4. VK_KHR_video_queue
5. VK_KHR_video_decode_queue

[baldurk]

I'll leave this open for now but this may not ever happens so the issue may be closed.

The vulkan video extensions were not designed with support for capture/replay tools in mind, so I don't think this will be possible. Aside from that these extensions are a huge API surface that is for the most part unrelated and orthogonal to actual rendering - RenderDoc will never implement any meaningful debugging for these extensions as it is a graphics debugger not a video decoding debugger, so a video player program using this API would not get any use out of RenderDoc and you would not be able to use it to solve your problem.

[FelixLamontagne2]

Thanks for the quick (and honest) response!

I think it would help here that I clarify my proposition and maybe we can make a list of v1 features that are much more digestible for RenderDoc as a product.

I definitely do not expect RenderDoc to allow debuging the actual decoding processing that's going on in the gpu. As you said, the api wasn't made with that in mind and a lot of what's happening is just the driver doing it's thing (and also the video demuxing on the cpu side which has nothing to do with Vulkan or RenderDoc).

What I do find unfortunate though is that just because I want to decode in my application then I can't inspect every other Vulkan commands and resources that RenderDoc currently has great support for.

Would it be possible as a first step to support a handful of the most important decode calls? For example, we can see in the nsight capture the vkCmdDecodeVideoKHR which allows me to inspect the parameters of the function and, most importantly, which image is being used in a decode call, but there is no "decode debuging".

Even if the decode commands weren't supported I'd still hapilly use RenderDoc to debug my own Vulkan issues with rendering and image processing, but right now I can't because the device creation fails (VK_ERROR_EXTENSION_NOT_PRESENT). Maybe RenderDoc could have some kind of extension bypass that allows it to work, but not capture the calls from certain extensions? I don't know how feasible that is though.

so a video player program using this API would not get any use out of RenderDoc and you would not be able to use it to solve your problem

Turns out that my issue was due to incorect vulkan image barriers which nsight was able to help with (although I would prefer using RenderDoc). The video was being correctly decoded, but I was using an incorect image layout which resulted in a green screen.

Below is screenshot of an nisght capture showing some of the commands going on. This doesn't show every single commands otherwise the screenshot would be too big, but as we can see it's mostly normal commands like barriers, fences and image copies.

[baldurk]

Yes in my previous comment I was talking about basic capture and replay with no introspection - the extensions were not designed with that in mind and I'm not sure it's practical. There is no way to 'skip' and extension or selectively implement certain parts of an extension except where allowed for by optional functionality, if it exists.

Note that you should check for extension availability via the enumeration functions before calling vkCreateDevice, the VK_ERROR_EXTENSION_NOT_PRESENT error in theory should not be encountered for a correct application and is there only as a guardrail.