[x264-devel] x264 version 104 crash

[Jason Garrett-Glaser]

On Thu, Jul 15, 2010 at 6:19 PM, 191919 <191919 at gmail.com> wrote:
> x264.exe works fine. I dynamically linked to a VC project, and gdb
> cannot recognize pdb file.
>
> 0x401536 calls x264_encoder_encode().
>
> /xx$ gdb ./1.exe
> Copyright (C) 2007 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-mingw32"...
> (no debugging symbols found)
> (gdb) r
> Starting program: c:\MinGW\xx/./1.exe
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x6fb6bc83 in x264_load_deinterleave_8x8x2_fenc_ssse3 ()
>   from c:\MinGW\xx\libx264-104.dll
> (gdb) where
> #0  0x6fb6bc83 in x264_load_deinterleave_8x8x2_fenc_ssse3 ()
>   from c:\MinGW\xx\libx264-104.dll
> #1  0x6fb3ecd3 in x264_ac_energy_mb (h=<incomplete type>, mb_x=0, mb_y=0,
>    frame=0x6e13a0) at encoder/ratecontrol.c:227
> #2  0x6fb3f1a0 in x264_adaptive_quant_frame (h=<incomplete type>,
>    frame=0x6e13a0, quant_offsets=0x0) at encoder/ratecontrol.c:330
> #3  0x6fb5b1f1 in x264_encoder_encode (h=<incomplete type>, pp_nal=0x12fe90,
>    pi_nal=0x12fe88, pic_in=0x361b18, pic_out=0x12fe98)
>    at encoder/encoder.c:2335
> #4  0x00401536 in ?? ()
> #5  0x001d27b0 in ?? ()
> #6  0x0012fe90 in ?? ()
> #7  0x0012fe88 in ?? ()
> #8  0x00361b18 in ?? ()
> #9  0x0012fe98 in ?? ()
> #10 0x00000000 in ?? ()
> (gdb)
>
> 2010/7/16 Jason Garrett-Glaser <darkshikari at gmail.com>:
>> On Thu, Jul 15, 2010 at 9:28 AM, 191919 <191919 at gmail.com> wrote:
>>> The problem is caused by a newly-added SSSE3 optimization:
>>>
>>> x264_load_deinterleave_8x8x2_fenc_ssse3: push        ebx
>>> .6FB8C291: 8B442408                       mov         eax,[esp][8]
>>> .6FB8C295: 8B4C240C                       mov         ecx,[esp][00C]
>>> .6FB8C299: 8B542410                       mov         edx,[esp][010]
>>> .6FB8C29D: 660F6F2500A4BA6F               movdqa      xmm4,[06FBAA400]
>>> .6FB8C2A5: BB04000000                     mov         ebx,4
>>> .6FB8C2AA: 660F6F01                      movdqa      xmm0,[ecx]
>>> .6FB8C2AE: 660F3800C4                     pshufb      xmm0,xmm4
>>> .6FB8C2B3: 660F7F00                       movdqa      [eax],xmm0
>>> .6FB8C2B7: 660F6F0411                     movdqa      xmm0,[ecx][edx]
>>> .6FB8C2BC: 660F3800C4                     pshufb      xmm0,xmm4
>>> .6FB8C2C1: 660F7F4010                     movdqa      [eax][010],xmm0
>>> .6FB8C2C6: 83C020                         add         eax,020
>>> .6FB8C2C9: 8D0C51                         lea         ecx,[ecx][edx]*2
>>> .6FB8C2CC: 4B                             dec         ebx
>>> .6FB8C2CD: 7FDB                           jg         .06FB8C2AA
>>> .6FB8C2CF: 5B                             pop         ebx
>>> .6FB8C2D0: C3                             retn
>>>
>>> In ".6FB8C2B3: 660F7F00 movdqa      [eax],xmm0", eax is not aligned to
>>> 16 bytes. YET ANOTHER GCC BUG. :(
>>>
>>> I am using gcc-4.4.3 and gcc-4.5.1 MinGW, both generated bad stack.
>>
>> Do you have a backtrace?
>>
>> Dark Shikari
>> _______________________________________________
>> x264-devel mailing list
>> x264-devel at videolan.org
>> http://mailman.videolan.org/listinfo/x264-devel
>>
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel
>

Fixed.  Will be in the next release.  Patch attached.

Dark Shikari
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.diff
Type: application/octet-stream
Size: 1450 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20100715/d089d5e2/attachment.obj>