[vlc-devel] h264: support frame parameter changes during frame-mt

[Rafaël Carré]

Hello,

Le 18/12/2012 20:01, Janne Grunau a écrit :
> Module: libav
> Branch: master
> Commit: 9e696d2e5ffaa763c564682ec18c3b51b3e5fccc
> 
> Author:    Janne Grunau <janne-libav at jannau.net>
> Committer: Janne Grunau <janne-libav at jannau.net>
> Date:      Thu Sep  6 13:30:08 2012 +0200
> 
> h264: support frame parameter changes during frame-mt
> 
> Fixes CVE-2012-2782.

This libav commit broke vaapi support in VLC :

libva: VA-API version 0.32.0
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva: va_openDriver() returns 0
[0x7ff658000ce8] main generic debug: using hw decoder module "vaapi"
[0x7ff64d059ec8] avcodec decoder debug: Available decoder output format
61 (dxva2_vld)
[0x7ff64d059ec8] avcodec decoder debug: Available decoder output format
53 (vaapi_vld)
[0x7ff64d059ec8] avcodec decoder: Using VA API version 0.32 for hardware
decoding.
[0x7ff658e620f8] main generic debug: looking for hw decoder module
matching "vaapi": 1 candidates
libva: VA-API version 0.32.0
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva: va_openDriver() returns 0
[0x7ff658e620f8] main generic debug: using hw decoder module "vaapi"
[0x7ff64d059ec8] avcodec decoder debug: Available decoder output format
61 (dxva2_vld)
[0x7ff64d059ec8] avcodec decoder debug: Available decoder output format
53 (vaapi_vld)
[0x7ff64d059ec8] avcodec decoder: Using VA API version 0.32 for hardware
decoding.
zsh: segmentation fault (core dumped)  cvlc --ignore-config -vv
--avcodec-hw vaapi bbb-1080-stereo.mp4


As you see the vaapi module is probed multiple times; running VLC
multiple times will show that module is probed a random number of times
between 2 and 7 before segfaulting.

I did not debug anything yet, just experienced it and wanted to share
git bisect result, so suggestions (or fix) welcome :)

backtraces often come from i965_drv_video.so but not always..

vlc: i965_drv_video.c:1105: i965_UnmapBuffer: Assertion `obj_buffer &&
obj_buffer->buffer_store' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe95ef700 (LWP 18475)]
0x00007ffff7414425 in __GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff7414425 in __GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff7417b8b in __GI_abort () at abort.c:91
#2  0x00007ffff740d0ee in __assert_fail_base (fmt=<optimized out>,
    assertion=0x7fffd55b3510 "obj_buffer && obj_buffer->buffer_store",
    file=0x7fffd55b314e "i965_drv_video.c", line=<optimized out>,
function=<optimized out>)
    at assert.c:94
#3  0x00007ffff740d192 in __GI___assert_fail (
    assertion=0x7fffd55b3510 "obj_buffer && obj_buffer->buffer_store",
    file=0x7fffd55b314e "i965_drv_video.c", line=1105,
function=0x7fffd55b3f70 "i965_UnmapBuffer")
    at assert.c:103
#4  0x00007fffd559b264 in ?? () from
/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
#5  0x00007fffd74f283c in render_picture (surface=67108880,
vactx=0x7fffc800a3d8)
    at libavcodec/vaapi.c:48
#6  ff_vaapi_common_end_frame (s=0x7fffcd06d960) at libavcodec/vaapi.c:188
#7  0x00007fffd7314fdd in field_end (in_setup=0, h=0x7fffcd06d960) at
libavcodec/h264.c:2310
#8  decode_frame (avctx=0x7fffcd06c020, data=0x7fffcd06c460,
got_frame=0x7fffe95eeccc,
    avpkt=<optimized out>) at libavcodec/h264.c:4201
#9  0x00007fffd74ecc6b in avcodec_decode_video2
(avctx=avctx at entry=0x7fffcd06c020,