[vlc-devel] segfault when decoding rgb-packed huffyuv when --ffmpeg-pp is enabled

[Matthew Hodgson]

Hi,

When I try to decode a huffyuv-encoded AVI (packed as RGB), I get a segfault 
when using any of the --ffmpeg-pp options (e.g. --ffmpeg-pp-q=6).  This is 
on VLC & ffmpeg HEAD of a few days ago.

The backtrace looks like this (ffmpeg & vlc compiled -g -O0):

$ ./vlc --reset-config -vvv -I dummy --ffmpeg-pp-q=6 --vout dummy 
/home/matthew/lipsync.avi

yields:

(gdb) bt
#0  0x08330122 in blockCopy_MMX (
     dst=0x8882a78 
"ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200ÿÿÿ\200\001", 
dstStride=8, src=0x8744b58 "\001", srcStride=-1209792064, levelFix=0, 
packedOffsetAndScale=0xbedff344) at postprocess_template.c:3285
#1  0x0832ae88 in postProcess_MMX (src=0x8744b58 "\001", 
srcStride=-1209792064, dst=0x7f8 <Address 0x7f8 out of bounds>, dstStride=8, 
width=88, height=72,
     QPs=0x874cf58 '\001' <repeats 11 times>, "\bi", QPStride=0, isColor=1, 
c2=0x8846320) at postprocess_template.c:3511
#2  0x083301dd in postProcess (src=0x8744b58 "\001", srcStride=-1209792064, 
dst=0x7f8 <Address 0x7f8 out of bounds>, dstStride=8, width=88, height=72, 
QPs=0x874cf58 '\001' <repeats 11 times>, "\bi",
     QPStride=0, isColor=1, vm=0x8745e98, vc=0x8846320) at 
libpostproc/postprocess.c:707
#3  0x0832dc42 in pp_postprocess (src=0xbedff94c, srcStride=0xbedff92c, 
dst=0xbedff93c, dstStride=0xbedff91c, width=88, height=72, 
QP_store=0x874cf58 '\001' <repeats 11 times>, "\bi", QPStride=0,
     vm=0x8745e98, vc=0x8846320, pict_type=0) at libpostproc/postprocess.c:1137
#4  0x080c6e1b in PostprocPict__ffmpeg (p_dec=0xc0585518, p_data=0x87449a8, 
p_pic=0xbedff94c, p_ff_pic=0xbedff92c) at postprocess.c:193
#5  0x080c2609 in ffmpeg_CopyPicture (p_dec=0x8741e20, p_pic=0x877dccc, 
p_ff_pic=0x87448a0) at video.c:784
#6  0x080c2075 in DecodeVideo__ffmpeg (p_dec=0x8741e20, pp_block=0x8741e20) 
at video.c:572
#7  0x080a07d4 in DecoderDecode (p_dec=0x8741e20, p_block=0x87687b8) at 
src/input/decoder.c:663
#8  0x080a1d14 in DecoderThread (p_dec=0x8741e20) at src/input/decoder.c:488
#9  0xb7f87cc4 in pthread_start_thread () from /lib/libpthread.so.0
#10 0xb7f87d3f in pthread_start_thread_event () from /lib/libpthread.so.0
#11 0xb7defbe7 in clone () from /lib/libc.so.6

which as I understand it, is due to 
libpostprocess/postprocess.c:pp_postprocess() being called with a PPMode 
with chromMode flag set - but being fed a RGBA packed picture.   Because 
chromMode is set, postprocess_template.c:postProcess() gets called on the 
nonexistent & uninitialized U and V channels as well as the 'Y' channel - 
resulting in the segfault.

I don't understand how pp_postprocess() could ever convert from RGBA32 to 
YUV, as it all seems to be intended for manipulating planar YUV of various 
shapes.  modules/codec/ffmpeg/video.c:ffmpeg_CopyPicture() checks to see if 
the AVContext's pix_fmt is one of a known list (which includes 
PIX_FMT_RGBA32) using ffmpeg_PixFmtToChroma - and if the pixel format is in 
the list, tries to convert to YUV (i assume) using the postprocess module if 
pp'ing is enabled in vlc.

This seems wrong, as the postprocess code simply doesn't seem to understand 
packed data, let alone packed RGBA.

Any explanation would be hugely appreciated, as the bug is proving a real 
pain... a small video which demonstrates the bug can be found at 
http://opensource.mxtelecom.com/vlc/huffyuv_test.avi

cheers,

Matthew.

-- 
______________________________________________________________
Matthew Hodgson   matthew at mxtelecom.com   Tel: +44 845 6667778
                 Systems Analyst, MX Telecom Ltd.

-- 
This is the vlc-devel mailing-list, see http://www.videolan.org/vlc/
To unsubscribe, please read http://developers.videolan.org/lists.html