[vlc-devel] Picture size differs from encoder init fmt

[Rafaël Carré]

On 01/15/14 20:39, Francois Cartegnie wrote:
> Le 15/01/2014 19:58, Rafaël Carré a écrit :
>> Can you give all the struct members to avoid any ambiguity?

> (gdb) print p_enc->fmt_in
> video = {i_chroma = 0, i_width = 640, i_height = 336,
> i_visible_width = 640, i_visible_height = 336,

> (gdb) print *p_pic
> $1 = {format = {i_chroma = 808596553, i_width = 640, i_height = 352,
>  i_visible_width = 640, i_visible_height = 336

This looks correct, the 2 formats are identical although the decoded
picture has extra padding in height. The encoder is using a smaller
height so it should not read past the 336th line.

>  p = {{p_pixels = 0x7fffdc20ca80
> i_lines = 356, i_pitch = 640, i_pixel_pitch = 1,
>       i_visible_lines = 352, i_visible_pitch = 640},

> p_pixels = 0x7fffdc244480
> i_lines = 178, i_pitch = 320, i_pixel_pitch = 1,
> i_visible_lines = 176, i_visible_pitch = 320},

> p_pixels = 0x7fffdc252300
> i_lines = 178, i_pitch = 320, i_pixel_pitch = 1,
> i_visible_lines = 176, i_visible_pitch = 320},

visible lines look incorrect though here, not sure if that matters anyway.

Where/when does the heap overflow happen ?

Do you have valgrind trace?