[x264-devel] segfault in predict_8x8_ddl_sse2 in shared lib

[Dave Brueck]

On Sep 12, 2008, at 9:43 AM, Jason Garrett-Glaser wrote:

> On Fri, Sep 12, 2008 at 8:23 AM, Dave Brueck <x264list at protoven.com>  
> wrote:
>> Hello,
>>
>> I get a crash in predict_8x8_ddl_sse2 when I load x264 as a shared
>> library, but encoding the same file using the x264 command line  
>> binary
>> does not crash. It may just be that I am doing something incorrect
>> when loading libx264.so (i.e. I'm not 100% sure this is x264's
>> problem), so I'm mostly looking for any tips you all could give me on
>> where to look next.
>
> This is probably due to a lack of stack alignment, a rather annoying
> problem in the 32-bit x86 ABI.
>
> One possible cause is that x264_slicetype_decide( h ) is not under a
> function that forces stack alignment (I'm a bit curious why this
> is--feel like enlightening me, Loren?).  I know that when calling x264
> as a shared library from certain Windows apps, some people add the
> gcc4.2-only force_aligned_arg_pointer or whatever it is to force stack
> alignment when calling as a shared library.


Thanks very much for the quick reply. I tried adding that to  
slicetype_decide but the problem remained. However, when I modified  
x264_slicetype_mb_cost the segfault went away (yay!).

I changed encoder/slicetype.c from:

static int x264_slicetype_mb_cost( x264_t *h, x264_mb_analysis_t *a, ...

to:

static int __attribute__((force_align_arg_pointer))  
x264_slicetype_mb_cost( x264_t *h, x264_mb_analysis_t *a, ...

I hesitate to try and formulate a patch since I'm unsure if this is  
the preferred fix generally, I don't know under what conditions it  
should be used (shared lib only? 32 bit only? gcc x.y.z only?), and  
what other functions might need the same thing, but if someone more  
expert in the x264 code creates a patch at some point, I'd be more  
than happy to help test it.

Thanks again for your help!
-Dave