[x264-devel] Access violation in SSE2 assembly when compiling with Visual Studio

[Rico Huijbers]

Hi,

I'm having a problem getting x264 to work after compiling it with Visual 
Studio Express C++ 2008 and NASM 2.03.01.

I already encountered this problem with the source from 06/11 so I 
figured I should update but it still happens with the daily tarball from 
06/19. I'm running the command-line application to encode a test avi, by 
the way.

What happens is that I receive an Access Violation inside the 
_x264_deblock_v_luma_sse2 (assembler) routine. While the code calling 
that routine has changed between the two versions, the error occurs 
twice in exactly the same location [1]. In the 06/11 release I could 
also tell the original call was to _x264_deblock_h_luma_sse2, which in 
turn called the _v_ version. With the 06/19 version I'm not so sure 
anymore but I imagine it's the same.

When I initialize the application and disable CPU enhancements (h->cpu = 
0), everything runs fine, so the problem is definitely within the SSE2 
asm routine.

And there's another strange thing. The error message says:

     Unhandled exception at 0x0044de82 in x264.exe: 0xC0000005:
     Access violation reading location 0xffffffff.

But when I traced the disassembly to the instruction causing the error,
[esi+ecx] didn't work out to 0xffffffff at all, but some entirely 
different number. I'm not sure what's causing that.

Anway, that's my situation right now, and I'm not sure what's wrong. Is 
it a misconfiguration of Visual Studio? Is it a version mismatch of 
NASM? Is there an actual bug in the code? Or was the code never intended 
to be compiled with Visual Studio?

I'd be very grateful for any insight and help on how to proceed.

Best regards,
Rico Huijbers



=========================================
[1] My disassembly looks like this:

_x264_deblock_v_luma_sse2:
0044DE60  push        ebx
0044DE61  push        esi
0044DE62  mov         eax,dword ptr [esp+0Ch]
0044DE66  mov         ecx,dword ptr [esp+10h]
0044DE6A  mov         edx,dword ptr [esp+14h]
0044DE6E  mov         ebx,dword ptr [esp+18h]
0044DE72  mov         esi,dword ptr [esp+1Ch]
0044DE76  lea         esi,[ecx+ecx*2]
0044DE79  dec         edx
0044DE7A  neg         esi
0044DE7C  dec         ebx
0044DE7D  add         esi,eax
0044DE7F  sub         esp,24h
0044DE82  movaps      xmm0,xmmword ptr [esi+ecx] <== * ERROR HERE *
0044DE86  movaps      xmm1,xmmword ptr [esi+ecx*2]
0044DE8A  movaps      xmm2,xmmword ptr [eax]
0044DE8D  movaps      xmm3,xmmword ptr [eax+ecx]
0044DE91  movd        xmm4,edx
0044DE95  movd        xmm5,ebx
0044DE99  pshuflw     xmm4,xmm4,0
0044DE9E  movlhps     xmm4,xmm4
0044DEA1  pshuflw     xmm5,xmm5,0
0044DEA6  movlhps     xmm5,xmm5
0044DEA9  packuswb    xmm4,xmm4