[x264-devel] x264 BUG - movdqa SSSE3 instruction with unaligned memory address

[Loren Merritt]

On Thu, 28 Apr 2011, Eugene Leviant wrote:

> I've found a bug in x264 which will very likely cause a crash on x86
> machines, supporting SSSE3 instruction set
> This bug is in mc_chroma_ssse3_cache64 function and is caused by movdqa
> instruction used to transfer xmm6 register
> content to unaligned address on stack (r0m register). Just changing movdqa
> to movdqu does not solve the problem,
> as below is pmaddsubsw instruction which has the same address on stack as a
> parameter

x264 makes certain assumptions beyond the universal x86_32 abi (see 
doc/standards.txt). One of those assumptions is that the stack is 16byte 
aligned upon calling any asm function. This assumption is is true if you 
compile with gcc or icc.
We also support the non-aligned case (X264_CPU_STACK_MOD4), but we don't 
optimize for it: the fallback is to not use mc_chroma_ssse3_cache64.

If you found a case where cpudetect fails to set X264_CPU_STACK_MOD4, then 
we should fix that. But mc_chroma is not broken.

--Loren Merritt