[x264-devel] Segfault in x264_pixel_avg2_w16_sse2

[Noboru Asai]

Hi,

This patch fixes the following problem.

commit 207a83429c86800470b9524a731f9a4de6903915
Author: Noboru Asai <noboru.asai at gmail.com>
Date:   Fri Mar 21 20:32:19 2008 +0900

    [PATCH] fix r764

diff --git a/encoder/analyse.c b/encoder/analyse.c
index 0264621..5f73f31 100644
--- a/encoder/analyse.c
+++ b/encoder/analyse.c
@@ -1443,7 +1443,8 @@ static void x264_mb_analyse_inter_direct( x264_t
*h, x264_mb_analysis_t *a )

 static void x264_mb_analyse_inter_b16x16( x264_t *h, x264_mb_analysis_t *a )
 {
-    uint8_t pix1[16*16], pix2[16*16];
+    DECLARE_ALIGNED( uint8_t, pix1[16*16], 16 );
+    DECLARE_ALIGNED( uint8_t, pix2[16*16], 16 );
     uint8_t *src2;
     int stride2 = 16;
     int weight;


2008/3/22, Ian Rickards <ian.rickards at btinternet.com>:
>
>
> Hi all,
>
> There is a problem with the new SSE2 optimizations - specifically when
> bframes option is used.  Linux with C2D.
>
> Starting program: /mnt/net/progs/x264/x264 --bframes=3 --pass=1
> --bitrate=1200 topgear_720x576.yuv -o ttt.264
> x264 [info]: file name gives 720x576
> x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x080a27dc in x264_pixel_avg2_w16_sse2 ()
> (gdb) where
> #0  0x080a27dc in x264_pixel_avg2_w16_sse2 ()
> #1  0x080a0774 in mc_luma_sse2 (dst=0xbffcc7c8 "ÿÿÿÿ", i_dst_stride=16,
>     src=0xb7d172e3, i_src_stride=832, mvx=-1211546909, mvy=16,
>     i_width=-1211010333, i_height=135023920) at common/x86/mc-c.c:139
> #2  0x080a0774 in mc_luma_sse2 (dst=Cannot access memory at address 0x18
> ) at common/x86/mc-c.c:139
> Previous frame inner to this frame (corrupt stack?)
>
> Thanks
>
> Ian
>
> _______________________________________________
>  x264-devel mailing list
>  x264-devel at videolan.org
>  http://mailman.videolan.org/listinfo/x264-devel
>
>