• Philip Langdale's avatar
    lavu/pixdesc: favour formats where depth and subsampling exactly match · ed83a3a5
    Philip Langdale authored
    Since introducing the various packed formats used by VAAPI (and p012),
    we've noticed that there's actually a gap in how
    av_find_best_pix_fmt_of_2 works. It doesn't actually assign any value
    to having the same bit depth as the source format, when comparing
    against formats with a higher bit depth. This usually doesn't matter,
    because av_get_padded_bits_per_pixel() will account for it.
    
    However, as many of these formats use padding internally, we find that
    av_get_padded_bits_per_pixel() actually returns the same value for the
    10 bit, 12 bit, 16 bit flavours, etc. In these tied situations, we end
    up just picking the first of the two provided formats, even if the
    second one should be preferred because it matches the actual bit depth.
    
    This bug already existed if you tried to compare yuv420p10 against p016
    and p010, for example, but it simply hadn't come up before so we never
    noticed.
    
    But now, we actually got a situation in the VAAPI VP9 decoder where it
    offers both p010 and p012 because Profile 3 could be either depth and
    ends up picking p012 for 10 bit content due to the ordering of the
    testing.
    
    In addition, in the process of testing the fix, I realised we have the
    same gap when it comes to chroma subsampling - we do not favour a
    format that has exactly the same subsampling vs one with less
    subsampling when all else is equal.
    
    To fix this, I'm introducing a small score penalty if the bit depth or
    subsampling doesn't exactly match the source format. This will break
    the tie in favour of the format with the exact match, but not offset
    any of the other scoring penalties we already have.
    
    I have added a set of tests around these formats which will fail
    without this fix.
    ed83a3a5
pixdesc.h 14.9 KB