• Philip Langdale's avatar
    lavu/pixdesc: handle xv30be in av_[read|write]_image_line · 9651f873
    Philip Langdale authored
    xv30be is an obnoxious format that I shouldn't have included in the
    first place. xv30 packs 3 10bit channels into 32bits and while our
    byte-oriented logic can handle Little Endian correctly, it cannot
    handle Big Endian. To avoid that, I marked xv30be as a bitstream
    format, but while that didn't produce FATE errors, it turns out that
    the existing read/write code silently produces incorrect results, which
    can be revealed via ubsan.
    
    In all likelyhood, the correct fix here is to remove the format. As
    this format is only used by Intel vaapi, it's only going to show up
    in LE form, so we could just drop the BE version. But I don't want to
    deal with creating a hole in the pixfmt list and all the weirdness that
    comes from that. Instead, I decided to write the correct read/write
    code for it.
    
    And that code isn't too bad, as long as it's specialised for this
    format, as the channels are all bit-aligned inside a 32bit word.
    9651f873
pixdesc.c 102 KB