[vlc-devel] [patch] i420_rgb: buffer overflow

[jnqnfe at gmail.com]

On Tue, 2019-01-22 at 20:52 +0100, Francois Cartegnie wrote:
> Le 22/01/2019 à 18:44, jnqnfe at gmail.com a écrit :
> > patch attached
> > 
> > Incorrect pointer offset calculation in SSE2 (non-assembly version)
> > RGB15 unpacking.
> > 
> > Could, I believe, allow almost 128 bytes to be written past the end
> > of
> > the end of the buffer on last loop iteration.
> 
> Each picture is stride aligned.
> That's not unusual to overwrite past visible_width by less than 16 in
> assembly for efficiency.
> 
> visible_width (w) < width (stride)
> 

In this case it isn't an efficiency thing, it's calculating the wrong
place to write the data. Compare it with the aligned case and ASM impl
and with the impls for the other formats.

I am not certain that it is writing into extra stride space, I'm not
familiar enough with some of the properties like x/y offsets yet to
comprehend properly what is being done, I just noticed an inconsistency
where it was writing to the wrong place, which may or may not be
outside of the buffer. The size of the buffer seems to be
offset+visible based, as are the loops... Ah, if it's horizontal
scaling then it allocates such a buffer, otherwise using an existing
one, of i-don't-know-what size...

I'll leave it to the experts to determine whether or not it is actually
writing outside of a buffer...