[x264-devel] segfault in predict_8x8_ddl_sse2 in shared lib

[Jason Garrett-Glaser]

On Fri, Sep 12, 2008 at 8:23 AM, Dave Brueck <x264list at protoven.com> wrote:
> Hello,
>
> I get a crash in predict_8x8_ddl_sse2 when I load x264 as a shared
> library, but encoding the same file using the x264 command line binary
> does not crash. It may just be that I am doing something incorrect
> when loading libx264.so (i.e. I'm not 100% sure this is x264's
> problem), so I'm mostly looking for any tips you all could give me on
> where to look next.

This is probably due to a lack of stack alignment, a rather annoying
problem in the 32-bit x86 ABI.

One possible cause is that x264_slicetype_decide( h ) is not under a
function that forces stack alignment (I'm a bit curious why this
is--feel like enlightening me, Loren?).  I know that when calling x264
as a shared library from certain Windows apps, some people add the
gcc4.2-only force_aligned_arg_pointer or whatever it is to force stack
alignment when calling as a shared library.

Dark Shikari