[vlc-devel] Crash transcoding, scaling up, with threads >= 1

[David Menestrina]

Hi,

For the past few days I've been debugging a crash I've been getting
while transcoding.  The crash is due to the fact that I was scaling a
video up, and somehow pictures of the decoder's size were being sent
to the encoder (x264 in this case).  When this happens, the encoder
was reading past the end of the small picture, since it assumed it any
picture provided be at the proper width and height.

Dark_Shikari gave me a patch to x264 to return an error rather than
crash, but VLC still does not work properly in this case.  It is
incorrect for VLC to be handing these small pictures to the encoder
anyways, so I started working to determine the cause of the problem.
I've determined that the problem is due to lines 746-751 of
modules/stream_out/transcode/video.c.  These lines attempt to create a
copy (p_pic2) of the current picture (p_pic), but p_pic2 is created
with the format of the decoder output, not the encoder input.  So
p_pic2 gets passed to the encoder, which expects a larger picture,
which causes the failure.

A potential fix would be to create this copy based on the encoder
input format, but I am not familiar with this code, and so I'm not
sure if this fix could introduce problems in other cases.  Further,
lines 684-689 make a similar copy in handling subpictures that could
also cause the same problem.

Could someone familiar with this code take a look at this and let me
know if creating the picture copy in the format of the encoder input
format would be the right way to fix this issue, for both p_pic2 and
the subpicture handling code?

Thank you,
david