[vlc-devel] [vlc-commits] avcodec: fix direct rendering with some codecs

[Laurent Aimar]

On Sun, Dec 11, 2011 at 03:50:33PM -0500, Rafaël Carré wrote:
> So chroma is I2AL
> 
> The picture is here:
> 	width/height = visible width/height = 720x486
> For the 3 planes though:
> 	width/height = 736x498
> 	visible width/height = 720x486
That's normal.
For the width:
 U/V -> 720/2 -> 360 -> 368 (mod 16)
 Y -> 2 * U/V (I422° -> 2*368 -> 736 (and already mod 16)

For the height:
 Y/U/V -> 486 -> 496 (mod 16)
and there is 2 extra hardcoded lines to ensure that the pictures will
be compatible with avcodec_align_dimensions() for H264 (otherwise
dr will be disabled by ffmpeg_GetFrameBuf()).

> 
> [0x6a18628] avcodec decoder debug: using direct rendering
> ==2248== Invalid write of size 8
> ==2248==    at 0x145B00A8: ??? (proresdsp.asm:428)
> ==2248==    by 0x14453BB0: decode_slice_plane (proresdec.c:492)
> ==2248==    by 0x1445403F: decode_slice (proresdec.c:587)
> ==2248==    by 0x144D690E: avcodec_default_execute (utils.c:568)
> ==2248==    by 0x1445315C: decode_frame (proresdec.c:624)
> ==2248==    by 0x144D7277: avcodec_decode_video2 (utils.c:866)
> ==2248==    by 0x13EAEF70: DecodeVideo (video.c:590)
> ==2248==    by 0x5085966: DecoderDecodeVideo (decoder.c:1512)
> ==2248==    by 0x508567F: DecoderProcess (decoder.c:1878)
> ==2248==    by 0x508577A: DecoderThread (decoder.c:938)
> ==2248==    by 0x532AEFB: start_thread (pthread_create.c:304)
> ==2248==    by 0x582585C: clone (clone.S:112)
> ==2248==  Address 0x96c5650 is 0 bytes after a block of size 1,466,112 alloc'd
> ==2248==    at 0x4C278E8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==2248==    by 0x4C27997: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==2248==    by 0x50C05AF: picture_NewFromResource (vlc_common.h:935)
> ==2248==    by 0x50C1267: picture_pool_NewFromFormat (picture_pool.c:138)
> ==2248==    by 0x50AF3DE: vout_InitWrapper (vout_wrapper.c:153)
> ==2248==    by 0x50A78FB: ThreadStart (video_output.c:1360)
> ==2248==    by 0x50A8085: Thread (video_output.c:1499)
> ==2248==    by 0x532AEFB: start_thread (pthread_create.c:304)
> ==2248==    by 0x582585C: clone (clone.S:112)
> ==2248== 
> ==2248== 
> 
> 
> ff_prores_idct_put_10_sse4 () at libavcodec/x86/proresdsp.asm:428
> 428	idct_put_fn sse4, 16
> (gdb) bt
> #0  ff_prores_idct_put_10_sse4 () at libavcodec/x86/proresdsp.asm:428
> #1  0x0000000014453bb1 in decode_slice_plane (ctx=<optimized out>, td=0x2e00, buf=<optimized out>, data_size=<optimized out>, 
>     out_ptr=0x96c2290, linesize=1472, mbs_per_slice=8, blocks_per_mb=2, plane_size_factor=4, qmat=0x66e1270)
>     at libavcodec/proresdec.c:492
> #2  0x0000000014454040 in decode_slice (avctx=<optimized out>, td=0xc328f20) at libavcodec/proresdec.c:587
> #3  0x00000000144d690f in avcodec_default_execute (c=0x66df560, func=0x14453d10 <decode_slice>, arg=<optimized out>, 
>     ret=<optimized out>, count=112, size=4128) at libavcodec/utils.c:568
> #4  0x000000001445315d in decode_picture (avctx=0x66df560, pic_num=1, ctx=0x66e0f20) at libavcodec/proresdec.c:624
> #5  decode_frame (avctx=0x66df560, data=0x66dff60, data_size=0x8383cec, avpkt=0x8383c90) at libavcodec/proresdec.c:669
> #6  0x00000000144d7278 in avcodec_decode_video2 (avctx=0x66df560, picture=0x66dff60, got_picture_ptr=0x8383cec, 
>     avpkt=0x8383c90) at libavcodec/utils.c:866
> #7  0x0000000013eaef71 in DecodeVideo (p_dec=0x6a18628, pp_block=<optimized out>)
>     at ../../../../vlc/modules/codec/avcodec/video.c:590
> #8  0x0000000005085967 in DecoderDecodeVideo (p_dec=0x6a18628, p_block=0x7291190) at ../../vlc/src/input/decoder.c:1512
> #9  0x0000000005085680 in DecoderProcessVideo (b_flush=false, p_block=0x7291190, p_dec=0x6a18628)
>     at ../../vlc/src/input/decoder.c:1878
> #10 DecoderProcess (p_dec=0x6a18628, p_block=<optimized out>) at ../../vlc/src/input/decoder.c:2052
> #11 0x000000000508577b in DecoderThread (p_data=0x6a18628) at ../../vlc/src/input/decoder.c:938
> #12 0x000000000532aefc in start_thread (arg=0x8384700) at pthread_create.c:304
> #13 0x000000000582585d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #14 0x0000000000000000 in ?? ()
> (gdb) 

 Did yoy try the decoder from ffmpeg iof libav? Just in case. Also, did you
try disabling sse4 and see if the invalid write still happen?

-- 
fenrir