• Andreas Rheinhardt's avatar
    avcodec/vlc, bitstream: Fix multi VLC with uint8_t syms on BE · 4ab82d2f
    Andreas Rheinhardt authored
    VLC_MULTI_ELEM contains an uint8_t array that is supposed
    to be treated as an array of uint16_t when the used symbols
    have a size of two; otherwise it should be treated as just
    an array of uint8_t, but it was not always treated that way:
    
    vlc_multi_gen() initialized the first entry of the array
    by writing the symbol via AV_WN16; on big endian systems,
    the intended value was instead written into the second entry
    of the array (where it would likely be overwritten lateron
    during initialization).
    
    read_vlc_multi() also treated this case incorrectly: In case
    the code is so long that it needs a classical multi-stage lookup,
    the symbol has been written to the destination as if via AV_WN16.
    On little endian systems, this sets the correct first symbol and
    clobbers (zeroes) the next one, but the next one will be overwritten
    lateron anyway, so it won't be recognized. But on big-endian systems,
    the first symbol will be set to zero and the actually read symbol
    will be put into the slot for the next one (where it will be overwritten
    lateron).
    
    This commit fixes this; this fixes the magicyuv and utvideo FATE-tests
    on big endian arches.
    Signed-off-by: 's avatarAndreas Rheinhardt <andreas.rheinhardt@outlook.com>
    4ab82d2f
magicyuv.c 23.1 KB