[x264-devel] x264 version 104 crash

[191919]

x264.exe works fine. I dynamically linked to a VC project, and gdb
cannot recognize pdb file.

0x401536 calls x264_encoder_encode().

/xx$ gdb ./1.exe
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-mingw32"...
(no debugging symbols found)
(gdb) r
Starting program: c:\MinGW\xx/./1.exe
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)

Program received signal SIGSEGV, Segmentation fault.
0x6fb6bc83 in x264_load_deinterleave_8x8x2_fenc_ssse3 ()
   from c:\MinGW\xx\libx264-104.dll
(gdb) where
#0  0x6fb6bc83 in x264_load_deinterleave_8x8x2_fenc_ssse3 ()
   from c:\MinGW\xx\libx264-104.dll
#1  0x6fb3ecd3 in x264_ac_energy_mb (h=<incomplete type>, mb_x=0, mb_y=0,
    frame=0x6e13a0) at encoder/ratecontrol.c:227
#2  0x6fb3f1a0 in x264_adaptive_quant_frame (h=<incomplete type>,
    frame=0x6e13a0, quant_offsets=0x0) at encoder/ratecontrol.c:330
#3  0x6fb5b1f1 in x264_encoder_encode (h=<incomplete type>, pp_nal=0x12fe90,
    pi_nal=0x12fe88, pic_in=0x361b18, pic_out=0x12fe98)
    at encoder/encoder.c:2335
#4  0x00401536 in ?? ()
#5  0x001d27b0 in ?? ()
#6  0x0012fe90 in ?? ()
#7  0x0012fe88 in ?? ()
#8  0x00361b18 in ?? ()
#9  0x0012fe98 in ?? ()
#10 0x00000000 in ?? ()
(gdb)

2010/7/16 Jason Garrett-Glaser <darkshikari at gmail.com>:
> On Thu, Jul 15, 2010 at 9:28 AM, 191919 <191919 at gmail.com> wrote:
>> The problem is caused by a newly-added SSSE3 optimization:
>>
>> x264_load_deinterleave_8x8x2_fenc_ssse3: push        ebx
>> .6FB8C291: 8B442408                       mov         eax,[esp][8]
>> .6FB8C295: 8B4C240C                       mov         ecx,[esp][00C]
>> .6FB8C299: 8B542410                       mov         edx,[esp][010]
>> .6FB8C29D: 660F6F2500A4BA6F               movdqa      xmm4,[06FBAA400]
>> .6FB8C2A5: BB04000000                     mov         ebx,4
>> .6FB8C2AA: 660F6F01                      movdqa      xmm0,[ecx]
>> .6FB8C2AE: 660F3800C4                     pshufb      xmm0,xmm4
>> .6FB8C2B3: 660F7F00                       movdqa      [eax],xmm0
>> .6FB8C2B7: 660F6F0411                     movdqa      xmm0,[ecx][edx]
>> .6FB8C2BC: 660F3800C4                     pshufb      xmm0,xmm4
>> .6FB8C2C1: 660F7F4010                     movdqa      [eax][010],xmm0
>> .6FB8C2C6: 83C020                         add         eax,020
>> .6FB8C2C9: 8D0C51                         lea         ecx,[ecx][edx]*2
>> .6FB8C2CC: 4B                             dec         ebx
>> .6FB8C2CD: 7FDB                           jg         .06FB8C2AA
>> .6FB8C2CF: 5B                             pop         ebx
>> .6FB8C2D0: C3                             retn
>>
>> In ".6FB8C2B3: 660F7F00 movdqa      [eax],xmm0", eax is not aligned to
>> 16 bytes. YET ANOTHER GCC BUG. :(
>>
>> I am using gcc-4.4.3 and gcc-4.5.1 MinGW, both generated bad stack.
>
> Do you have a backtrace?
>
> Dark Shikari
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel
>