[x264-devel] [PATCH] improve malloc checks to allow graceful exit on malloc failures

[Jason Garrett-Glaser]

On Mon, Jun 22, 2009 at 9:27 PM, Steven Walters<kemuri9 at gmail.com> wrote:
> As it stands now, x264 can seg fault crash/exit on a consistent basis
> from memory allocation failures, it takes a couple sets of fixes to
> allow x264 to 'gracefully exit' rather than throw those nasty seg
> faults.
>
> 1. fix x264_malloc and x264_realloc
> (does anything actually use x264_realloc?)
> to return NULL before seg faulting out from memory allocation
> failures for non -DHAVE_MALLOC_H/-DSYS_MACOSX systems
> (Yes, I'm looking at you mingw)
>
> 2. increase the frequency of CHECKED_MALLOC and similar malloc'd
> memory checks to allow x264 to gracefully exit with the messages:
> x264 [error]: malloc failed
> x264 [error]: x264_encoder_open failed
> rather than crash-exiting from corresponding segmentation faults on
> attempting to manipulate NULL pointers.
>
> the 2nd point is more involved and did end up changing some of the
> ABI slightly to have some void functions which use x264_malloc
> to instead return ints to indicate malloc errors occurring.
>
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel
>
>

This patch is extremely incomplete and still leaves dozens if not
hundreds of places that malloc failure will still result in a crash.

Additionally, it breaks compilation on most non-Windows systems.

I will fix the rest of the crashes and commit a fixed patch later this week.

Dark Shikari