[vlc-devel] [patch] i420_rgb: buffer overflow

[jnqnfe at gmail.com]

On Tue, 2019-01-22 at 21:05 +0100, Francois Cartegnie wrote:
> Le 22/01/2019 à 18:44, jnqnfe at gmail.com a écrit :
> >      xmm5 = _mm_or_si128(xmm5, xmm7);                \
> > -    _mm_storeu_si128((__m128i*)(p_buffer+16), xmm5);
> > +    _mm_storeu_si128((__m128i*)(p_buffer+8), xmm5);
> 
> To me matches the Assembly one Line 204
> 
> # convert rgb24 plane to rgb15 pack for pixel 8-15
>     \n\
> punpckhbw %%xmm4, %%xmm7        # ________ ________ g7g6g5g4 g3______
>     \n\
> punpckhbw %%xmm1, %%xmm5        # r7r6r5r4 r3______ ______b7 b6b5b4b3
>     \n\
> psllw     $2,%%xmm7             # ________ ____g7g6 g5g4g3__ ________
>     \n\
> por       %%xmm7, %%xmm5        # r7r6r5r4 r3__g7g6 g5g4g3b7 b6b5b4b3
>     \n\
> movdqu    %%xmm5, 16(%3)        # store pixel 4-7
>     \n\
> "
> 
> and the caller also increases buffer by 16
> 

the assembly is doing +16 bytes, the function based implementation for
RGB15/16 is working with a word-pointer, thus needs to do +8 words.
doing +16 in this one instance results in +16 words thus writing twice
as far out as it should.

note that for the 32-bit formats the function based ones are working on
a dword-pointer, so the assembly uses +16, while the functions use +4