[vlc-devel] Bug in filter_NewPicture()?

[Laurent Aimar]

Hi,

On Sun, Jan 09, 2011 at 06:46:43PM +0200, Juha Jeronen wrote:
> I think I've found a bug. The function filter_NewPicture() (which just  
> calls p_filter->pf_video_buffer_new), when called from the deinterlacer,  
> seems to sometimes return pictures which have their p_next member filled  
> with random data. The deinterlacer doesn't guard against this.
>
> When such pictures are released (deleted), FilterDeletePictures() (in  
> src/misc/filter_chain.c) passes the nonsense pointer to Release() in  
> src/misc/picture_pool.c when it's freeing the linked list. If my reading  
> of the source is correct, this situation may either cause a segfault,  
> trigger the assert failure I mentioned or corrupt memory silently.
 Should be fixed (by [e2ec60cea8b4b9a0afd8b3b3f5517aa5e7657fbb] and
[3f50c72acbf4fe8ca4f9631a3bff596cba2d3a05]).
>
> For some unknown reason, this problem seems to occur only if  
> private_picture in src/video_output/vout_wrapper.c is larger than 3.
>
> If I protect against nonsense p_next in the deinterlacer, setting p_next  
> to NULL manually for pictures that shouldn't have one, the assert  
> failure stops happening.
>
> If I instead assert - after calling filter_NewPicture() - that p_next is  
> NULL before I write anything there, I get the occasional assert failure  
> from my own assert under exactly the circumstances that used to cause  
> the picture_pool.c assert failure. I think this is sufficient to confirm  
> that the problem indeed is what I suspect it to be.
>
> I think there are two main options to fix the problem:
>
> 1) Change the video buffer allocator (wherever it's actually  
> implemented) to initialize the memory it returns.
>
> 2) Define memory initialization as the caller's problem (like malloc()  
> does), and fix everything that calls filter_NewPicture().
>
> Which do you guys think is preferable?
>
> Thanks for bearing with me ;)
>
>  -J
>
> P.S. In other news, I fail at error handling. I'll post an updated soft  
> field repeat patch soon.
>
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> http://mailman.videolan.org/listinfo/vlc-devel

-- 

-- 
    ()      Laurent Aimar <fenrir at videolan.org>
 __(  )__   Developer of VideoLAN - www.videolan.org
| ( `´ ) |  The video streaming solution for Linux, Windows, MacOSX...
|  `--´  |       " Life is a cone "
 ````````