[vlc-devel] Bug in filter_NewPicture()?

[Juha Jeronen]

Hi again everyone,

I think I've found a bug. The function filter_NewPicture() (which just 
calls p_filter->pf_video_buffer_new), when called from the deinterlacer, 
seems to sometimes return pictures which have their p_next member filled 
with random data. The deinterlacer doesn't guard against this.

When such pictures are released (deleted), FilterDeletePictures() (in 
src/misc/filter_chain.c) passes the nonsense pointer to Release() in 
src/misc/picture_pool.c when it's freeing the linked list. If my reading 
of the source is correct, this situation may either cause a segfault, 
trigger the assert failure I mentioned or corrupt memory silently.

For some unknown reason, this problem seems to occur only if 
private_picture in src/video_output/vout_wrapper.c is larger than 3.

If I protect against nonsense p_next in the deinterlacer, setting p_next 
to NULL manually for pictures that shouldn't have one, the assert 
failure stops happening.

If I instead assert - after calling filter_NewPicture() - that p_next is 
NULL before I write anything there, I get the occasional assert failure 
from my own assert under exactly the circumstances that used to cause 
the picture_pool.c assert failure. I think this is sufficient to confirm 
that the problem indeed is what I suspect it to be.

I think there are two main options to fix the problem:

1) Change the video buffer allocator (wherever it's actually 
implemented) to initialize the memory it returns.

2) Define memory initialization as the caller's problem (like malloc() 
does), and fix everything that calls filter_NewPicture().

Which do you guys think is preferable?

Thanks for bearing with me ;)

  -J

P.S. In other news, I fail at error handling. I'll post an updated soft 
field repeat patch soon.