[x264-devel] x264 segfault in x264_pixel_satd_8x8_ssse3

[Jerome Venturi]

On x86 (Gentoo Linux) the x264 encoder compil fine (pthread enable) but when
using it , i get an segfault

[*] mencoder :

GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
(no debugging symbols found)
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run MOVIE.EVO -vc ffvc1 -fps 24000/1001 -of rawvideo -ofps 24000/1001
-ovc x264 -x264encopts
bitrate=3878:pass=1:subq=4:me=dia:frameref=5:bframes=2:b_pyramid:weight_b:direct_pred=auto:partitions=all:qcomp=
0.75:ip_factor=1.10:pb_factor=1.40:threads=2 -vf crop=1920:784:0:148
-nosound -o /dev/null
Starting program: /usr/bin/mencoder MOVIE.EVO -vc ffvc1 -fps 24000/1001 -of
rawvideo -ofps 24000/1001 -ovc x264 -x264encopts
bitrate=3878:pass=1:subq=4:me=dia:frameref=5:bframes=2:b_pyramid:weight_b:direct_pred=auto:partitions=all:qcomp=
0.75:ip_factor=1.10:pb_factor=1.40:threads=2 -vf crop=1920:784:0:148
-nosound -o /dev/null
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
MEncoder dev-SVN-r24537 (C) 2000-2007 MPlayer Team
CPU: Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz (Family: 6, Model: 15,
Stepping: 6)
CPUflags: Type: 6 MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compilé avec détection du CPU à l'exécution.115 codecs audio & 236 codecs
vidéo
succès : format : 0  data : 0x0 - 0x5bca1000
Fichier de type MPEG-PS détecté.
Searching for VC1 sequence header... found
VIDEO:  VC-1  1920x1080, 29.970 fps, header len: 33
[V] filefmt:2  fourcc:0x31435657  size:1920x1080  fps:29.97  ftime:=0.0334
Le fps d'entrée sera plutôt interprété comme 23.98
Ouverture du filtre vidéo : [expand osd=1]
Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1
Ouverture du filtre vidéo : [crop w=1920 h=784 x=0 y=148]
Crop: 1920 x 784, 0 ; 148
==========================================================================
Codec vidéo forcé : ffvc1
Ouverture du décodeur vidéo : [ffmpeg] FFmpeg's libavcodec codec family
Codec vidéo choisi : [ffvc1] vfm : ffmpeg (FFmpeg M$ WVC1)
==========================================================================
VDec : requête de config de vo - 1920 x 1080 (espace colorimétrique préferé
: Planar YV12)
VDec: using Planar I420 as output csp (no 1)
L'aspect du film est 1.78:1 - pré-redimensionnement à l'aspect correct.
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSSE3
New_Face failed. Maybe the font path is wrong.
Please supply the text font file (~/.mplayer/subfont.ttf).
subtitle font: load_sub_face failed.
Pos:   0.0s      3f ( 0%)  0.00fps Trem:   0min   0mb  A-V:0.000 [0:0]
Program received signal SIGSEGV, Segmentation fault.
0xb7ef8efe in x264_pixel_satd_8x8_ssse3 () from /usr/lib/libx264.so.56
#0  0xb7ed8efe in x264_pixel_satd_8x8_ssse3 () from /usr/lib/libx264.so.56
#1  0xb7e8bc04 in x264_slicetype_mb_cost () from /usr/lib/libx264.so.56
#2  0x00000008 in ?? ()
#3  0x00000000 in ?? ()



[*] ffmpeg :

run -i MOVIE.EVO -an -pass 1 -vcodec libx264 -b 6000 OUTPUT.mp4
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/bin/ffmpeg -i MOVIE.EVO -an -pass 1 -vcodec libx264
-b 6000 OUTPUT.mp4
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
FFmpeg version SVN-r10503, Copyright (c) 2000-2007 Fabrice Bellard, et al.
  configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib
--mandir=/usr/share/man --enable-static --enable-shared
--cc=i686-pc-linux-gnu-gcc --disable-altivec --disable-debug
--disable-network --disable-ipv6 --disable-ffplay --enable-liba52
--enable-pthreads --enable-libogg --enable-libvorbis --enable-libx264
--enable-libfaad --enable-libfaac --enable-gpl --enable-pp --disable-strip
  libavutil version: 49.5.0
  libavcodec version: 51.44.0
  libavformat version: 51.13.3
  built on Sep 15 2007 18:38:04, gcc: 4.2.0 (Gentoo 4.2.0 p1.4)
Input #0, mpeg, from 'MOVIE.EVO':
  Duration: 02:16:17.7, start: 0.036222, bitrate: 22514 kb/s
  Stream #0.0[0xfd55]: Video: vc1, yuv420p, 1920x1080, 29.97 fps(r)
  Stream #0.1[0xfd56]: Video: vc1, yuv420p, 720x480, 29.97 fps(r)
  Stream #0.2[0xc8]: Audio: liba52, 48000 Hz, stereo, 192 kb/s
  Stream #0.3[0xc7]: Audio: liba52, 48000 Hz, stereo, 192 kb/s
  Stream #0.4[0xc6]: Audio: liba52, 48000 Hz, stereo, 192 kb/s
  Stream #0.5[0xc5]: Audio: liba52, 48000 Hz, stereo, 192 kb/s
  Stream #0.6[0xc4]: Audio: liba52, 48000 Hz, stereo, 192 kb/s
  Stream #0.7[0xc3]: Audio: liba52, 48000 Hz, stereo, 192 kb/s
  Stream #0.8[0xb1]: Audio: 0x0000
  Stream #0.9[0xc2]: Audio: liba52, 48000 Hz, 5:1, 640 kb/s
  Stream #0.10[0xc0]: Audio: liba52, 48000 Hz, 5:1, 640 kb/s
File 'OUTPUT.mp4' already exists. Overwrite ? [y/N] y
Output #0, mp4, to 'OUTPUT.mp4':
  Stream #0.0: Video: libx264, yuv420p, 720x480, q=2-31, pass 1, 6 kb/s,
29.97 fps(c)
Stream mapping:
  Stream #0.0 -> #0.0
[libx264 @ 0x43fdc9d0]using SAR=10/11
[libx264 @ 0x43fdc9d0]using cpu capabilities: MMX MMXEXT SSE SSE2 SSSE3
Press [q] to stop encoding

Program received signal SIGSEGV, Segmentation fault.
0xb7fbaefe in x264_pixel_satd_8x8_ssse3 () from /usr/lib/libx264.so.56
(gdb) bt
#0  0xb7fbaefe in x264_pixel_satd_8x8_ssse3 () from /usr/lib/libx264.so.56
#1  0xb7f6dc04 in x264_slicetype_mb_cost () from /usr/lib/libx264.so.56
#2  0x00000008 in ?? ()
#3  0x00000000 in ?? ()



[*] x264cli :

gdb x264
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
(no debugging symbols found)
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) run -o outfile.264 test.yuv 720x480 -p 1 -B 1000
Starting program: /usr/bin/x264 -o outfile.264 test.yuv 720x480 -p 1 -B 1000
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSSE3

Program received signal SIGSEGV, Segmentation fault.
0x080be91e in x264_pixel_satd_8x8_ssse3 ()
(gdb) bt
#0  0x080be91e in x264_pixel_satd_8x8_ssse3 ()
#1  0x08078ab4 in ?? ()
#2  0xbfb11e78 in ?? ()
#3  0x00000020 in ?? ()
#4  0x080d7800 in ?? ()
#5  0x00000010 in ?? ()
#6  0x00000008 in ?? ()
#7  0x00000000 in ?? ()





So i'm sure this is cause by ssse3, sure because I have an another pc which
run exactly the same revision (x264, mencoder, ffmpeg) and same system as
well but it don't have ssse3
Anyway, when disable asm (noasm), it's working (but it's really slow even
with two threads).

Note : gcc version 4.2.0 (Gentoo 4.2.0 p1.4)

Jérome Venturi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.videolan.org/pipermail/x264-devel/attachments/20070916/3693caef/attachment.htm