[x264-devel] x86 + yasm + pic causes x264 to segfault in x264_pixel_ssim_4x4x2_core_sse2

[Alexis Ballier]

Hi, 

I'm hitting a weird bug, I thought it was worth reporting it : 
On x86 (linux), with yasm installed, I ran : 
./configure --enable-pic

everything compiles fine, but then : 

 ./x264 -o ../toto.264 ../example.y4m -q 10
yuv4mpeg: 384x288 at 25/1fps, 0:0
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow! 
Segmentation fault


(gdb) run -o ../toto.264 ../example.y4m -q 10
Starting program: /root/x264-svn/x264 -o ../toto.264 ../example.y4m -q 10
(no debugging symbols found)
warning: Lowest section in system-supplied DSO at 0xffffe000 is .hash at ffffe0b4
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread -135865664 (LWP 1226)]
yuv4mpeg: 384x288 at 25/1fps, 0:0
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow! 

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -135865664 (LWP 1226)]
0x080e2b45 in x264_pixel_ssim_4x4x2_core_sse2 ()




(gdb) i r
eax            0x8151ef0        135601904
ecx            0xf7c902c2       -137821502
edx            0x1c0    448
ebx            0x80ef02b        135196715
esp            0xffbfebc8       0xffbfebc8
ebp            0xf7e49bca       0xf7e49bca
esi            0x8151f10        135601936
edi            0xf7c8fbca       -137823286
eip            0x80e2b45        0x80e2b45 <x264_pixel_ssim_4x4x2_core_sse2+277>
eflags         0x10216  [ PF AF IF RF ] 
cs             0x23     35
ss             0x2b     43
ds             0x2b     43
es             0x2b     43
fs             0x0      0
gs             0x63     99

(gdb) x/8i $pc
0x80e2b45 <x264_pixel_ssim_4x4x2_core_sse2+277>:        movdqa 0xffffefa4(%ebx),%xmm7
0x80e2b4d <x264_pixel_ssim_4x4x2_core_sse2+285>:        pshufd $0xb1,%xmm3,%xmm5
0x80e2b52 <x264_pixel_ssim_4x4x2_core_sse2+290>:        pmaddwd %xmm7,%xmm1
0x80e2b56 <x264_pixel_ssim_4x4x2_core_sse2+294>:        pmaddwd %xmm7,%xmm2
0x80e2b5a <x264_pixel_ssim_4x4x2_core_sse2+298>:        pshufd $0xb1,%xmm4,%xmm6
0x80e2b5f <x264_pixel_ssim_4x4x2_core_sse2+303>:        packssdw %xmm2,%xmm1
0x80e2b63 <x264_pixel_ssim_4x4x2_core_sse2+307>:        paddd  %xmm5,%xmm3
0x80e2b67 <x264_pixel_ssim_4x4x2_core_sse2+311>:        pmaddwd %xmm7,%xmm1



I get the same result with yasm 0.5.0 and 0.6.0.
The weird thing is that if I remove yasm and install nasm, everything is fine :/
(except that nasm ignores the noexec stack instruction in i386inc.asm,
 because of the quotes but that doesn't seem related)
If I disable pic, it's fine also.

Is anybody else able to reproduce that ?
Should I blame yasm or is it because of subtle differences between yasm and nasm ? 

It also doesn't segfault if I give --no-ssim option to x264.


Regards, 

Alexis.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.videolan.org/pipermail/x264-devel/attachments/20070603/49a2d121/attachment.pgp