[x264-devel] x264 BUG - movdqa SSSE3 instruction with unaligned memory address

[Eugene Leviant]

Hi, All!

I've found a bug in x264 which will very likely cause a crash on x86
machines, supporting SSSE3 instruction set
This bug is in mc_chroma_ssse3_cache64 function and is caused by movdqa
instruction used to transfer xmm6 register
content to unaligned address on stack (r0m register). Just changing movdqa
to movdqu does not solve the problem,
as below is pmaddsubsw instruction which has the same address on stack as a
parameter

As all xmm registers are being used in this function one cannot just uase
some xmmN register as temporary storage.
My solution is not very effective but still working - I'm using temporary
r4m register to save xmm6 register, than load value
from r0m to xmm6 and than use xmm6 as an argument for pmaddubsw. Just like
this

;    This is what was originally in mc-a.asm (mult1 is an alias for r0m)
;    pmaddubsw  m2, mult1
;    pmaddubsw  m3, mult1

; Below is my changes (applied for x86 arch only)
    movu r4m, m6
    movu m6, r0m
    pmaddubsw  m2, m6
    pmaddubsw  m3, m6
    movu m6, r4m


This patch above makes x264 work on my Atom N270 netbook.
Some comments from x264 developers will be appreciated. Can submit a patch
but do not know how to do this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20110428/5415addb/attachment.html>