• Andreas Rheinhardt's avatar
    avutil/aes: Don't use misaligned pointers · 73930e4f
    Andreas Rheinhardt authored
    The AES code uses av_aes_block, a union consisting of
    uint64_t[2], uint32_t[4], uint8_t[4][4] and uint8_t[16].
    subshift() performs byte-wise manipulations of two av_aes_blocks,
    but when encrypting, it does so with a shift of two bytes;
    more precisely, it uses
    "av_aes_block *s1 = (av_aes_block *) (s0[0].u8 - s)"
    and lateron uses the uint8_t[16] member to access s0.
    Yet av_aes_block requires to be suitably aligned for
    the uint64_t[2] member, which s0[0].u8 - 2 is certainly
    not. This is in violation of 6.3.2.3 (7) of C11. UBSan
    reports this in the aes_ctr, mov-3elist-encrypted,
    mov-frag-encrypted, mov-tenc-only-encrypted and srtp
    tests.
    Furthermore, there is another issue here: The pointer points
    outside of s0; this works, because all the accesses lateron
    use an index >= 3. (Clang-)UBSan reports this as
    "runtime error: index -2 out of bounds for type 'uint8_t[16]'".
    
    This commit fixes both of these issues: The latter issue
    is fixed by applying an offset of "+ 3" during the cast
    and subtracting this from the indices used lateron.
    The former issue is solved by not casting to av_aes_block*
    at all; instead simply cast to unsigned char*.
    Signed-off-by: 's avatarAndreas Rheinhardt <andreas.rheinhardt@outlook.com>
    73930e4f
aes.c 8.32 KB