VMAF relevancy

[MillingtonAllen]

I've been experimenting with VMAF to try to see what CRF calculates to be about equal VMAF score across various speed presets.

I generally use animation for testing because it's my primary use case. However, x265 superfast CRF 21 gives marginally above the 1080p VMAF score of slower CRF 28 1080p. Despite being nearly identical in VMAF score, the supeprfast one has a significantly more x265 artifacting.

I can't blame VMAF too much for a few reasons:

1. I'm using a 4K monitor, so I should really be using the 4K preset. That being said, I'm using 1080p media, so I'm not fully sure the recommendation.
2. I'm probably using VMAF outside it's intended use
3. VMAF is generally low (50's?) even for settings like CRF 10. I'm not sure why it's rating the source material so low.

Another odd observation. VMAF at 4K seems to generally be scored higher than 1080p vmaf, though I'm not sure why.

I need to re-run the experiment using vmaf_4k, but I suspect the results will be similar.

This might not be as much of a problem for live media.

[InB4DevOps]

To my knowledge VMAF was not developed for content that is watched on computer screens. You're sitting much nearer to your computer screen than to a TV in your living room. VMAF is for TVs - execept for the phone model of course..
Do you still the the artifacts when you're farther away from your screen?

I don't know how but maybe one could train a model for computer screen consumption? Maybe ask in https://github.com/Netflix/vmaf ?

[MillingtonAllen]

The documentation in VMQ for VMAF says that it's viewing distance is 3X the height of the screen for 1080p content. For vmaf 4K, the distance is 1.5x the height of the screen (a distance where 4K is actually appreciable), which is pretty much my PC use case.

From ~3X height of the screen, overall the artifacting was less notable, but it still had a segment where artifacting was notable. Looking into it before, I had found that slower / veryslow tends to handle edges a lot better.

The VMAF4 4k had the same CRF indicating equal score as vmaf (1080p).

I still find it odd that vmaf_4k gives better scores than vmaf (1080p). The 4K situation is going to be far less forgiving than the 1080p scenario, though maybe that's where the peripheral vision part comes in, idk. Both are a lot lower than I'd expect though (1080p: 53 and 4K: 63), which might be because larger color patches in animated content. Even if using CRF of 0, min vmaf is zero and I've experienced this across shows of animated content (My guess is that it's dislikes frames of pure black / white and it's handling dividing by zero, but I'm not sure).

VMAF score is a good measure within an x265 preset, but, at least for animation, exact vmaf score across presets it starts to lose meaning.

I had found that most media I encode, slower CRF 30-34 produces acceptable results, so I had settled on CRF 28 slower just to be safe. I was looking to see if there was an equivalent CRF of faster encode that would produce similar results by comparing average vmaf and sd of a slower encode to faster encodes of varying CRF. For now, I'll just stick to visual comparisons.

[InB4DevOps]

Wait a sec..
so you're using the 4k VMAF for 1080p transcodes?
Is your source video 1080 or 4k?

[MillingtonAllen]

I've tested it using both VMAF 1080p and VMAF 4K. Source is 1080p, but monitor and viewing conditions resemble 4K profile (aside from media source).

The results didn't really change with either VMAF used. The VMAF score Slow CRF 28 ~= fast, faster, veryfast 23, regardless whether 4K or 1080p profile is used.

[CrypticSignal]

I'm using a 4K monitor, so I should really be using the 4K preset. That being said, I'm using 1080p media, so I'm not fully sure the recommendation.

The VMAF documentation states "As of v1.3.7 (June 2018), we have added a new 4K VMAF model at model/vmaf_4k_v0.6.1.json, which predicts the subjective quality of video displayed on a 4KTV and viewed from the distance of 1.5 times the height of the display device (1.5H)."

As there is no mention of source resolution, just "displayed on a 4KTV", my guess is that it's the resolution of the display that matters rather than the resolution of the source. I suppose this makes sense, as no matter what the source is, the video is upscaled or downscaled to fit the display nicely. For example, when you're viewing 1080p content on a 4K TV, upscaling takes place and you're technically viewing a 4K image even though the source is 1080p.

Even if using CRF of 0, min vmaf is zero

Is this when using Video Quality Metrics or when using FFmpeg yourself? You should not get a score of 0 for any frame. If you are, I'm guessing that this is when using FFmpeg yourself and you didn't specify the framerate of the input files with -r and/or you didn't use setpts=PTS-STARTPTS. See this issue for more details.

[MillingtonAllen]

Yeah, it still seems weird to me that the VMAF would be higher when running a 4K preset (sitting closer with a sharper display).

As for 0 vmaf, it's when using Video Quality Metrics. I'm using the feature-set [2] in the documentation to compare multiple crf. I'm not using the -tvp flag.

[CrypticSignal]

Yeah, it still seems weird to me that the VMAF would be higher when running a 4K preset (sitting closer with a sharper display).

Hmm. 4K has four times as many pixels as 1080p, that's four times the pixel density, so I suppose that more than makes up for viewing distance being halved. I'm certainly not an expert on VMAF but doesn't it make sense that you get higher values with the 4K model due to the higher pixel density of a 4K display?

Personally, I don't really care about the absolute VMAF values. For me, it's about how different presets/CRF values compare to each other in terms of VMAF. And, no matter which model you choose, the same model will used for all presets/CRF values and therefore the comparison is valid no matter what the values themselves are.

By the way, in case you don't know, the models have been trained subjectively, where participants have been asked to rate quality of different videos. So surely VMAF isn't an objective quality metric anyway. You can read about how the 4K model was trained here (26 clips were used, half of them were 1080p and the other half were 4K. Viewed on a 43" 4K TV).

As for 0 vmaf, it's when using Video Quality Metrics. I'm using the feature-set [2]

I don't think there should ever be a score of 0. For example, for a plain black frame in both the original and the transcoded video, I assume you would get a score of 100 for that frame. Are you using Overview Mode too?