[vlc-devel] Crash in ThreadDisplayRenderPicture

[Juha Jeronen]

Hi all,

Posting this onto the list as per Rémi's recommendation...

I found a crash in ThreadDisplayRenderPicture() 
(src/video_output/video_output.c).

Consider the following snippet:

---8<---8<---8<---
         if (vout->p->is_decoder_pool_slow) {
             direct = picture_pool_Get(vout->p->display_pool);
             if (direct)
                 picture_Copy(direct, render);
             picture_Release(render);

         } else {
             direct = render;
         }
         VideoFormatCopyCropAr(&direct->format, &filtered->format);
---8<---8<---8<---

The variable "direct" is a picture_t* initialized as NULL. If 
vout->p->is_decoder_pool_slow is true, but for some reason there is no 
picture in the display pool, direct will remain NULL and VLC will crash.

It seems it's just a simple mistake - just after the above snippet, the 
renderer aborts if direct is NULL.

A reliable way to reproduce the crash is to set any framerate doubling 
deinterlacer as the default (try bob or yadif2x), restart VLC and then 
open a DVD. It will also happen randomly during DVD playback, if a 
framerate doubling deinterlacer is enabled. Also very often when pausing.

I fixed this in my copy by adding a check:

---8<---8<---8<---
|        if(direct)
             VideoFormatCopyCropAr(&direct->format, &filtered->format);
         else
             msg_Warn( vout, "ThreadDisplayRenderPicture(): direct is 
NULL, aborting" );
|---8<---8<---8<---

I don't know if the warning is a good idea in the long run, but I think 
the check is required.

It would be interesting to find out why the display pool occasionally 
becomes empty when a framerate doubling deinterlacer is enabled, but the 
core is a bit too complicated for me to figure it out any time soon ;)

My VLC version is 1.2-git from mid-December and I'm running on Debian 
Squeeze.


I originally posted about this on the forum (the first post contains a 
gdb backtrace, and the second one the same solution I've reported in 
this message):

http://forum.videolan.org/viewtopic.php?f=32&t=85930


  -J