[x264-devel] [PATCH] improve malloc checks to allow graceful exit on malloc failures

[Steven Walters]

As it stands now, x264 can seg fault crash/exit on a consistent basis
from memory allocation failures, it takes a couple sets of fixes to
allow x264 to 'gracefully exit' rather than throw those nasty seg
faults.

1. fix x264_malloc and x264_realloc
(does anything actually use x264_realloc?)
to return NULL before seg faulting out from memory allocation
failures for non -DHAVE_MALLOC_H/-DSYS_MACOSX systems
(Yes, I'm looking at you mingw)

2. increase the frequency of CHECKED_MALLOC and similar malloc'd
memory checks to allow x264 to gracefully exit with the messages:
x264 [error]: malloc failed
x264 [error]: x264_encoder_open failed
rather than crash-exiting from corresponding segmentation faults on
attempting to manipulate NULL pointers.

the 2nd point is more involved and did end up changing some of the
ABI slightly to have some void functions which use x264_malloc
to instead return ints to indicate malloc errors occurring.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x264_check_alloc_lvl_up.diff
Type: application/octet-stream
Size: 11908 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20090623/d4b37278/attachment-0001.obj>