Make it possible/easier to determine a suitable resolution for equirects, cylinders, (cubemaps?) and quads

[dominiccooney]

The layer types with XR*LayerInit : XRLayerInit require authors to specify viewPixelWidth and viewPixelHeight. Working out effective values for these seems difficult.

I have my source media (environment map/photo/video/etc.) available in multiple resolutions. How do I work out what resolution will be effective for the device that is rendering the layer? Similar to issue #224, I need to pump a frame to find out views and work out a field of view from that; and I guess I need to create a projection layer, and fetch the subimage to work out its width and height, and throw that layer away to then calculate pixels per degree and pick media based on that.

That seems like a lot of work for what I think is a pretty common use case. In 2D media we have srcset, source and media queries for selecting media and, worst case, clientWidth/clientHeight and devicePixelRatio I can work out the size of a video element and pick suitable media. But if I take that same video and put it on a quad layer I need to specify pixel sizes and break that whole pipeline.

I have a few ideas how this might be improved:

• For video specifically, the video element already has <source>, srcset, etc. Why can't that just continue to operate when the video is bound to a layer?
• For everything else, it would be nice if I did not have to render them with WebGL. For example, say I want to put an img or SVG image on a quad. These already have some resolution independence, source selection, etc. so it would be nice if XRMediaBinding accepted CanvasImageSource. Yes, no doubt there's some spec work. For example to fetch the right image from the srcset means the layer content will update at some point. But video can advance frames, so why not?
• For exceptional cases, maybe I'm rendering some exotic media (say PDF) with WebGL or canvas, sure, makes sense to have an explicit view pixel size to target rendering to. In that case it should be easy to work out a "good" target size.

[cabanier]

The layer types with XR*LayerInit : XRLayerInit require authors to specify viewPixelWidth and viewPixelHeight. Working out effective values for these seems difficult.
...

• For video specifically, the video element already has <source>, srcset, etc. Why can't that just continue to operate when the video is bound to a layer?

Video layers don't have an inherent pixel width and height. The spec is currently silent at what resolution they are rendered. Inside the Oculus browser, the layer is allocated at the pixel width and height of the video.
Maybe I should add a note to the spec to clarify this? I would prefer to leave it up to the UA to pick the exact behavior.

Also, we should specify what happens if the video source changes. Likely during the next Raf call, the underlying video layer texture should be reallocated.

• For everything else, it would be nice if I did not have to render them with WebGL. For example, say I want to put an img or SVG image on a quad. These already have some resolution independence, source selection, etc. so it would be nice if XRMediaBinding accepted CanvasImageSource. Yes, no doubt there's some spec work. For example to fetch the right image from the srcset means the layer content will update at some point. But video can advance frames, so why not?

DOM layers would inherit all this from the regular 2D world. The spec would have to define how the size of the layer would translate to CSS pixels.
Or are you proposing that we have an intermediate layer type that supports SVG and images or that we expand media layers?

• For exceptional cases, maybe I'm rendering some exotic media (say PDF) with WebGL or canvas, sure, makes sense to have an explicit view pixel size to target rendering to. In that case it should be easy to work out a "good" target size.

How would you envision that this is specced?
Maybe: "if pixel width/height are not passed, the UA is free to pick a size based on the current width/height"

We would have to add a note that changing the dimensions after creation, will not reallocate the texture.

[dominiccooney]

Video layers don't have an inherent pixel width and height.

It seems to the author that they do. See my first paragraph about XRLayerInit requiring viewPixelWidth and viewPixelHeight.

Also, we should specify what happens if the video source changes.

Agreed. I was wondering what happens if the aspect ratio of the content changes.

DOM layers would inherit all this from the regular 2D world. The spec would have to define how the size of the layer would translate to CSS pixels. Or are you proposing that we have an intermediate layer type that supports SVG and images or that we expand media layers?

Images are special because they have an intrinsic aspect ratio, right? So say I want to put an image on a quad, making me determine the image's aspect ratio and specify it for a quad seems inconvenient. But for DOM content in general, sure, having me specify the aspect ratio of a quad or equirect seems reasonable.

How would you envision that this is specced? Maybe: "if pixel width/height are not passed, the UA is free to pick a size based on the current width/height" ... We would have to add a note that changing the dimensions after creation, will not reallocate the texture.

Seems reasonable! And it would be splendid, like #224, if I did not have to wait for a frame to find out the resolution.

[cabanier]

Video layers don't have an inherent pixel width and height.

It seems to the author that they do. See my first paragraph about XRLayerInit requiring viewPixelWidth and viewPixelHeight.

Video layers use a different structure. See here

Also, we should specify what happens if the video source changes.

Agreed. I was wondering what happens if the aspect ratio of the content changes.

The spec already defines how the aspect ratio is calculated. The new spec text will refer to that.

DOM layers would inherit all this from the regular 2D world. The spec would have to define how the size of the layer would translate to CSS pixels. Or are you proposing that we have an intermediate layer type that supports SVG and images or that we expand media layers?

Images are special because they have an intrinsic aspect ratio, right? So say I want to put an image on a quad, making me determine the image's aspect ratio and specify it for a quad seems inconvenient. But for DOM content in general, sure, having me specify the aspect ratio of a quad or equirect seems reasonable.

That is true. Are there enough arguments to add support?
Since this is additional functionality, we could file is as a future enhancement.

How would you envision that this is specced? Maybe: "if pixel width/height are not passed, the UA is free to pick a size based on the current width/height" ... We would have to add a note that changing the dimensions after creation, will not reallocate the texture.

Seems reasonable! And it would be splendid, like #224, if I did not have to wait for a frame to find out the resolution.

Yes, that should be possible. We need to hear from other UAs if they can return pixel information right away.

[cabanier]

@dominiccooney I marked it as "future enhancement". Please let me know if you feel strongly that this needs to be in the initial version of the spec.