[vlc-devel] [patch] i420_rgb: buffer overflow

[Francois Cartegnie]

Le 22/01/2019 à 18:44, jnqnfe at gmail.com a écrit :
>      xmm5 = _mm_or_si128(xmm5, xmm7);                \
> -    _mm_storeu_si128((__m128i*)(p_buffer+16), xmm5);
> +    _mm_storeu_si128((__m128i*)(p_buffer+8), xmm5);

To me matches the Assembly one Line 204

# convert rgb24 plane to rgb15 pack for pixel 8-15
    \n\
punpckhbw %%xmm4, %%xmm7        # ________ ________ g7g6g5g4 g3______
    \n\
punpckhbw %%xmm1, %%xmm5        # r7r6r5r4 r3______ ______b7 b6b5b4b3
    \n\
psllw     $2,%%xmm7             # ________ ____g7g6 g5g4g3__ ________
    \n\
por       %%xmm7, %%xmm5        # r7r6r5r4 r3__g7g6 g5g4g3b7 b6b5b4b3
    \n\
movdqu    %%xmm5, 16(%3)        # store pixel 4-7
    \n\
"

and the caller also increases buffer by 16

-- 
Francois Cartegnie
VideoLAN - VLC Developer