[vlc-devel] Picture size differs from encoder init fmt

[Rémi Denis-Courmont]

On Wed, 15 Jan 2014 03:21:02 +0100, Francois Cartegnie <fcvlcdev at free.fr>
wrote:
> I've found some heap overflow case with at least one encoder.
> This is due to requested format different the resulting picture format
> passed to encoder, and then overflows the allocated buffer when copying.
> 
> fmt_in is 640x336 which becomes 640x352 in pictures.

Pictures are always padded in both dimensions for the benefit of hardware
and SIMD optimizations.

I think encoders have to use the plane sizes stored in the picture
structure; the input format is only indicative of the user/useful size of
the pictures. If you really need to know the physical picture size you can
use picture_Setup() or something like that (e.g. rawvideo codec). It is not
very pretty, but that's how Laurent has been dealing with it so far.

-- 
Rémi Denis-Courmont
Sent from my collocated server