[vlc-devel] Bug in filter_NewPicture()?
juha.jeronen at jyu.fi
Sun Jan 9 17:46:43 CET 2011
Hi again everyone,
I think I've found a bug. The function filter_NewPicture() (which just
calls p_filter->pf_video_buffer_new), when called from the deinterlacer,
seems to sometimes return pictures which have their p_next member filled
with random data. The deinterlacer doesn't guard against this.
When such pictures are released (deleted), FilterDeletePictures() (in
src/misc/filter_chain.c) passes the nonsense pointer to Release() in
src/misc/picture_pool.c when it's freeing the linked list. If my reading
of the source is correct, this situation may either cause a segfault,
trigger the assert failure I mentioned or corrupt memory silently.
For some unknown reason, this problem seems to occur only if
private_picture in src/video_output/vout_wrapper.c is larger than 3.
If I protect against nonsense p_next in the deinterlacer, setting p_next
to NULL manually for pictures that shouldn't have one, the assert
failure stops happening.
If I instead assert - after calling filter_NewPicture() - that p_next is
NULL before I write anything there, I get the occasional assert failure
from my own assert under exactly the circumstances that used to cause
the picture_pool.c assert failure. I think this is sufficient to confirm
that the problem indeed is what I suspect it to be.
I think there are two main options to fix the problem:
1) Change the video buffer allocator (wherever it's actually
implemented) to initialize the memory it returns.
2) Define memory initialization as the caller's problem (like malloc()
does), and fix everything that calls filter_NewPicture().
Which do you guys think is preferable?
Thanks for bearing with me ;)
P.S. In other news, I fail at error handling. I'll post an updated soft
field repeat patch soon.
More information about the vlc-devel