Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cropping metadata #346

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions 06.bitstream.syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,8 @@ at least one byte of the payload data (including the trailing bit) shall not be
| metadata_scalability( )
| else if ( metadata_type == METADATA_TYPE_TIMECODE )
| metadata_timecode( )
| else if ( metadata_type == METADATA_TYPE_CROP )
| metadata_crop( )
| }
{:.syntax }

Expand Down Expand Up @@ -572,6 +574,23 @@ at least one byte of the payload data (including the trailing bit) shall not be
| }
{:.syntax }

#### Metadata cropping syntax

| --------------------------------------------------------- | ---------------- |
| metadata_crop( ) { | **Type**
| @@crop_width_minus_1 | f(16)
| @@crop_height_minus_1 | f(16)
| @@crop_offset_present | f(1)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we make it byte-aligned in the first place? having something like:

f(7) reserved
f(1) crop_offset_present;

would be better IMO than relying on the trailing_bits at the end of the OBU.

| if ( crop_offset_present == 1 ) {
| @@crop_x_offset | f(16)
| @@crop_y_offset | f(16)
| } else {
| crop_x_offset = 0
| crop_y_offset = 0
| }
| }
{:.syntax }

### Frame header OBU syntax

#### General frame header OBU syntax
Expand Down
38 changes: 37 additions & 1 deletion 07.bitstream.semantics.md
Original file line number Diff line number Diff line change
Expand Up @@ -575,7 +575,8 @@ Metadata OBUs may or may not have an OBU extension header. If there is no extens
| 4 | METADATA_TYPE_ITUT_T35 | payload-specific
| 5 | METADATA_TYPE_TIMECODE | N
| 6-31 | Unregistered user private | -
| 32 and greater | Reserved for AOM use | -
| 32 | METADATA_TYPE_CROP | Y
| 33 and greater | Reserved for AOM use | -
{:.table .table-sm .table-bordered }

The semantics of the column “Layer-specific” and its values are defined in Section 6.2.2.
Expand Down Expand Up @@ -1314,6 +1315,41 @@ coded video sequence.
number of bits used to represent time_offset_value is equal to time_offset_length. When
time_offset_value is not present, its value is inferred to be equal to 0.

#### Metadata crop semantics

When present the metadata_crop OBU applies starting at the next frame in the sequence
with matching temporal_id and spatial_id, and shall apply to all matching frames until the next Key Frame
Copy link

@podborski podborski Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When present the metadata_crop OBU applies starting at the next frame in the sequence
with matching temporal_id and spatial_id, ...

Alexis and myself are wondering here what the behavior should be if no extension header is present. Perhaps in that case it should apply to all layers? Then the language on the persistence should also be clarified.

Also instead of:

and shall apply to all matching frames

it would be better to write "and shall apply to all frames with a matching temporal_id and spatial_id"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I think it should apply to all layers if no header is present. I will add wording clarifying that.

or metadata_crop OBU. The output picture should be cropped to the region as specified in the
cropping metadata OBU. When applied, the crop shall be after all normal decode operations as a

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When applied, the crop shall be after all normal decode operations as a post-processing step (after film grain synthesis in the output process). This metadata has no effect on the decoding process.

Maybe it would be better to re-write like this:

The crop shall be applied after all normal decode operations as a post-processing step. This metadata information has no effect on the decoding process.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used your new wording.

post-processing step (after film grain synthesis in the output process). This metadata has no
effect on the decoding process.

**crop_x_offset** specifies the minimum pixel column containing picture data which should be rendered.
mattrwoz marked this conversation as resolved.
Show resolved Hide resolved

**crop_y_offset** specifies the minimum pixel row containing picture data which should be rendered.

**crop_width_minus_1** specifies the number of pixel columns minus 1 which which should be rendered after applying crop_x_offset.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which which => which

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


**crop_height_minus_1** specifies the number of pixel rows minus 1 which which should be rendered after applying crop_y_offset.

If RenderWidth or RenderHeight are present in the frame header, then render width and height
shall be applied after cropping.

The total cropped area shall not exceed the frame size:

* crop_width_minus_1 + crop_x_offset shall be less than or equal to frame_width_minus_1.
* crop_height_minus_1 + crop_y_offset shall be less than or equal to frame_height_minus_1.

The crop area shall align with the chroma subsampling grid:

* crop_x_offset and crop_width_minus_1 + 1 must both be a multiple of subsampling_x + 1.
* crop_y_offset and crop_height_minus_1 + 1 must both be a multiple of subsampling_y + 1.

When muxed into a container that supports signaling cropping information, this metadata should
be removed from the bitstream and included in the container’s signaling mechanism.
If both the container and bitstream signal cropping information, then the container’s cropping
information takes precedence.
Copy link

@podborski podborski Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of:

When muxed into a container that supports signaling cropping information, this metadata should
be removed from the bitstream and included in the container’s signaling mechanism.
If both the container and bitstream signal cropping information, then the container’s cropping
information takes precedence.

We could write:

In cases where cropping information is present in both the bitstream and the delivery or container format, the latter should be preferred.

NOTE: Container or delivery formats that package the AV1 bitstream are recommended to address this redundancy, potentially by excluding bitstream cropping information when it is already available in the container or delivery format.

Then we will have to update the AV1 ISOBMFF spec and say there that we use the clap box and remove this metadata OBU. Thoughts @cconcolato ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to new wording.


### Frame header OBU semantics

#### General frame header OBU semantics
Expand Down