[x264-devel] x264 version 104 crash

[191919]

The problem is caused by a newly-added SSSE3 optimization:

x264_load_deinterleave_8x8x2_fenc_ssse3: push        ebx
.6FB8C291: 8B442408                       mov         eax,[esp][8]
.6FB8C295: 8B4C240C                       mov         ecx,[esp][00C]
.6FB8C299: 8B542410                       mov         edx,[esp][010]
.6FB8C29D: 660F6F2500A4BA6F               movdqa      xmm4,[06FBAA400]
.6FB8C2A5: BB04000000                     mov         ebx,4
.6FB8C2AA: 660F6F01                      movdqa      xmm0,[ecx]
.6FB8C2AE: 660F3800C4                     pshufb      xmm0,xmm4
.6FB8C2B3: 660F7F00                       movdqa      [eax],xmm0
.6FB8C2B7: 660F6F0411                     movdqa      xmm0,[ecx][edx]
.6FB8C2BC: 660F3800C4                     pshufb      xmm0,xmm4
.6FB8C2C1: 660F7F4010                     movdqa      [eax][010],xmm0
.6FB8C2C6: 83C020                         add         eax,020
.6FB8C2C9: 8D0C51                         lea         ecx,[ecx][edx]*2
.6FB8C2CC: 4B                             dec         ebx
.6FB8C2CD: 7FDB                           jg         .06FB8C2AA
.6FB8C2CF: 5B                             pop         ebx
.6FB8C2D0: C3                             retn

In ".6FB8C2B3: 660F7F00 movdqa      [eax],xmm0", eax is not aligned to
16 bytes. YET ANOTHER GCC BUG. :(

I am using gcc-4.4.3 and gcc-4.5.1 MinGW, both generated bad stack.

2010/7/15 191919 <191919 at gmail.com>:
> I just copied code of x264_picture_alloc and replaced the x264_malloc
> line with mine. So all initialization codes are kept.
>
> 2010/7/15 Jason Garrett-Glaser <darkshikari at gmail.com>:
>> On Thu, Jul 15, 2010 at 7:08 AM, 191919 <191919 at gmail.com> wrote:
>>> Thanks for your reply. I took a look at commons/common.c in which
>>> x264_picture_alloc is implemented. Yes you are right, I did
>>> double-allocate the yuv buffer, but I don't think this is the cause of
>>> the library crash.
>>>
>>> I have removed the redundant memory allocations (thank you!), the
>>> result is same: my program crashed in x264_encoder_encode().
>>>
>>
>> Did you forget to the set the colorspace and i_plane of your
>> x264_picture_t?  picture_init doesn't do it for you.
>>
>> Dark Shikari
>> _______________________________________________
>> x264-devel mailing list
>> x264-devel at videolan.org
>> http://mailman.videolan.org/listinfo/x264-devel
>>
>