[x264-devel] Segmentation Fault in EvalPow

[Thoerin]

My apologies in advance if this is the wrong place for bug reports.

The error occurs regardless of input file and options so far as I can
tell. Other codecs work normally in ffmpeg.

GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/
gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...

(gdb) r -i Fracture-003.vob -threads 2 -vcodec libx264 -b 2000k -acodec
copy Fracture-0032.avi
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /home/alex/Setups/ffmpeg/ffmpeg_g -i Fracture-003.vob
-threads 2 -vcodec libx264 -b 2000k -acodec copy Fracture-0032.avi
[Thread debugging using libthread_db enabled]
FFmpeg version SVN-r14946, Copyright (c) 2000-2008 Fabrice Bellard, et
al.
  configuration: --enable-pthreads --enable-libfaac --enable-libmp3lame
--enable-libxvid --enable-gpl --enable-libx264 --enable-liba52
  libavutil 49.10. 0 / 49.10. 0
  libavcodec 51.69. 0 / 51.69. 0
  libavformat 52.21. 0 / 52.21. 0
  libavdevice 52. 1. 0 / 52. 1. 0
  built on Aug 24 2008 12:43:45, gcc: 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

Seems stream 0 codec frame rate differs from container frame rate:
29.97 (30000/1001) -> 59.94 (60000/1001)
Input #0, mpeg, from 'Fracture-003.vob':
  Duration: 00:24:43.36, start: 2733.816633, bitrate: 5790 kb/s
  Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x480 [PAR 32:27
DAR 16:9], 9801 kb/s, 59.94 tb(r)
  Stream #0.1[0x80]: Audio: liba52, 48000 Hz, 5:1, s16, 448 kb/s
  Stream #0.2[0x81]: Audio: liba52, 48000 Hz, 5:1, s16, 448 kb/s
  Stream #0.3[0x21]: Subtitle: dvdsub
  Stream #0.4[0x20]: Subtitle: dvdsub
[New Thread 0x7f569ed586e0 (LWP 20981)]
[New Thread 0x40bd1950 (LWP 20982)]
[New Thread 0x413d2950 (LWP 20983)]
[New Thread 0x41bd3950 (LWP 20984)]
[New Thread 0x423d4950 (LWP 20985)]
File 'Fracture-0032.avi' already exists. Overwrite ? [y/N] y
Output #0, avi, to 'Fracture-0032.avi':
  Stream #0.0: Video: libx264, yuv420p, 720x480 [PAR 32:27 DAR 16:9],
q=2-31, 2000 kb/s, 59.94 tb(c)
  Stream #0.1: Audio: ac3, 48000 Hz, 5:1, s16, 448 kb/s
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1
[libx264 @ 0xf44170]using SAR=32/27
[libx264 @ 0xf44170]using cpu capabilities: MMX2 SSE2Slow
Press [q] to stop encoding

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f569ed586e0 (LWP 20981)]
0x0000000000858cc2 in evalPow ()
(gdb) bt
#0 0x0000000000858cc2 in evalPow ()
#1 0x000000000085885a in evalFactor ()
#2 0x00000000008589a8 in evalTerm ()
#3 0x0000000000858ad8 in evalExpression ()
#4 0x0000000000858c6a in x264_eval ()
#5 0x0000000000844a6f in get_qscale ()
#6 0x00000000008462d4 in rate_estimate_qscale ()
#7 0x000000000084715b in x264_ratecontrol_start ()
#8 0x00000000007e929a in x264_encoder_encode ()
#9 0x000000000073a132 in X264_frame (ctx=<value optimized out>,
  buf=0x7f569ec06010 "", bufsize=1382400, data=<value optimized out>)
  at libavcodec/libx264.c:91
#10 0x00000000004a53bb in avcodec_encode_video (avctx=0xf44170,
  buf=0x7fffa6d77610 "", buf_size=12577056, pict=0x0)
  at libavcodec/utils.c:891
#11 0x000000000042a4c8 in output_packet (ist=0xf45500, ist_index=0,
  ost_table=0xf45690, nb_ostreams=2, pkt=0x7fffa6d78290) at
ffmpeg.c:957
#12 0x000000000042de0a in main (argc=<value optimized out>,
  argv=<value optimized out>) at ffmpeg.c:2132
(gdb) disass $pc-32 $pc+32
Dump of assembler code from 0x858ca2 to 0x858ce2:
0x0000000000858ca2 <x264_eval+162>: push %rdi
0x0000000000858ca3 <x264_eval+163>: rorb $0xc4,-0x7f(%rax)
0x0000000000858ca7 <x264_eval+167>: js 0x858cac
0x0000000000858ca9 <x264_eval+169>: add %al,(%rax)
0x0000000000858cab <x264_eval+171>: retq
0x0000000000858cac: nopl 0x0(%rax)
0x0000000000858cb0 <evalPow+0>: push %r12
0x0000000000858cb2 <evalPow+2>: push %rbp
0x0000000000858cb3 <evalPow+3>: push %rbx
0x0000000000858cb4 <evalPow+4>: mov %rdi,%rbx
0x0000000000858cb7 <evalPow+7>: sub $0x30,%rsp
0x0000000000858cbb <evalPow+11>: mov 0x328(%rdi),%rax
0x0000000000858cc2 <evalPow+18>: movzbl (%rax),%edx
0x0000000000858cc5 <evalPow+21>: cmp $0x2b,%dl
0x0000000000858cc8 <evalPow+24>: je 0x858df0 <evalPow+320>
0x0000000000858cce <evalPow+30>: xor %r12d,%r12d
0x0000000000858cd1 <evalPow+33>: cmp $0x2d,%dl
0x0000000000858cd4 <evalPow+36>: je 0x858dd7 <evalPow+295>
0x0000000000858cda <evalPow+42>: cmp $0x28,%dl
0x0000000000858cdd <evalPow+45>: je 0x858d90 <evalPow+224>
End of assembler dump.
(gdb) info all-registers
rax 0x0 0
rbx 0x7fffa6d77260 140735992525408
rcx 0xbfe910 12577040
rdx 0xbfe920 12577056
rsi 0x7fffa6d77610 140735992526352
rdi 0x7fffa6d77260 140735992525408
rbp 0x11b4400 0x11b4400
rsp 0x7fffa6d77150 0x7fffa6d77150
r8 0xbfe900 12577024
r9 0x0 0
r10 0x0 0
r11 0x0 0
r12 0x7fffa6d77260 140735992525408
r13 0x0 0
r14 0x0 0
r15 0x0 0
rip 0x858cc2 0x858cc2 <evalPow+18>
eflags 0x10206 [ PF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
---Type <return> to continue, or q <return> to quit---
gs 0x0 0
st0 -nan(0x1010101010000000) (raw 0xffff1010101010000000)
st1 -nan(0x1010101010101010) (raw 0xffff1010101010101010)
st2 -inf (raw 0xffff0000000000000000)
st3 -nan(0x1010101010000000) (raw 0xffff1010101010000000)
st4 -nan(0x1010101010101010) (raw 0xffff1010101010101010)
st5 -nan(0x1010101010101010) (raw 0xffff1010101010101010)
st6 -nan(0x10001000100010) (raw 0xffff0010001000100010)
st7 -nan(0x1010101010101010) (raw 0xffff1010101010101010)
fctrl 0x37f 895
fstat 0x0 0
ftag 0xffff 65535
fiseg 0x0 0
fioff 0x0 0
foseg 0x0 0
fooff 0x0 0
fop 0x0 0
xmm0 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0,
0x0},
  v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0,
0x0,
  0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0,
0x0},
  uint128 = 0x00000000000000000000000000000000}
xmm1 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0,
0x0},
  v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0,
0x0,
---Type <return> to continue, or q <return> to quit---
  0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0,
0x0},
  uint128 = 0x00000000000000000000000000000000}
xmm2 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {
  0x8000000000000000, 0x0}, v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xf8,
  0xff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0,
0x0, 0x0,
  0xfff8, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0xfff80000, 0x0,
0x0},
  v2_int64 = {0xfff8000000000000, 0x0},
  uint128 = 0x0000000000000000fff8000000000000}
xmm3 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {
  0x8000000000000000, 0x0}, v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xf8,
  0xff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0,
0x0, 0x0,
  0xfff8, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0xfff80000, 0x0,
0x0},
  v2_int64 = {0xfff8000000000000, 0x0},
  uint128 = 0x0000000000000000fff8000000000000}
xmm4 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {
  0x8000000000000000, 0x0}, v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xf8,
  0xff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0,
0x0, 0x0,
  0xfff8, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0xfff80000, 0x0,
0x0},
  v2_int64 = {0xfff8000000000000, 0x0},
  uint128 = 0x0000000000000000fff8000000000000}
xmm5 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {
  0x8000000000000000, 0x0}, v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xf8,
  0xff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0,
0x0, 0x0,
---Type <return> to continue, or q <return> to quit---
  0xfff8, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0xfff80000, 0x0,
0x0},
  v2_int64 = {0xfff8000000000000, 0x0},
  uint128 = 0x0000000000000000fff8000000000000}
xmm6 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {
  0x8000000000000000, 0x0}, v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xf8,
  0xff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0,
0x0, 0x0,
  0xfff8, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0xfff80000, 0x0,
0x0},
  v2_int64 = {0xfff8000000000000, 0x0},
  uint128 = 0x0000000000000000fff8000000000000}
xmm7 {v4_float = {0x0, 0x9, 0x0, 0x0}, v2_double = {0x6424c,
0x0},
  v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x30, 0x9, 0x19, 0x41, 0x1c, 0x97,
0x2f,
  0x0, 0x26, 0xcb, 0x10, 0x0}, v8_int16 = {0x0, 0x0, 0x930, 0x4119,
0x971c,
  0x2f, 0xcb26, 0x10}, v4_int32 = {0x0, 0x41190930, 0x2f971c,
0x10cb26},
  v2_int64 = {0x4119093000000000, 0x10cb26002f971c},
  uint128 = 0x0010cb26002f971c4119093000000000}
xmm8 {v4_float = {0x0, 0x1, 0x0, 0x0}, v2_double = {0x1,
0x0},
  v16_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf0, 0x3f, 0x0, 0x0, 0x0,
0x0,
  0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0, 0x0, 0x0, 0x3ff0, 0x0, 0x0,
0x0,
  0x0}, v4_int32 = {0x0, 0x3ff00000, 0x0, 0x0}, v2_int64 = {
  0x3ff0000000000000, 0x0}, uint128 =
0x00000000000000003ff0000000000000}
xmm9 {v4_float = {0x1, 0x0, 0x0, 0x0}, v2_double = {0x0,
0x0},
  v16_int8 = {0x0, 0x0, 0x80, 0x3f, 0x0 <repeats 12 times>}, v8_int16 =
{0x0,
  0x3f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x3f800000, 0x0,
0x0,
---Type <return> to continue, or q <return> to quit---
  0x0}, v2_int64 = {0x3f800000, 0x0},
  uint128 = 0x0000000000000000000000003f800000}
xmm10 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0,
0x0},
  v16_int8 = {0x29, 0x51, 0xce, 0xa0, 0xc8, 0x45, 0x48, 0x3f, 0x0, 0x0,
0x0,
  0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x5129, 0xa0ce, 0x45c8,
0x3f48, 0x0,
  0x0, 0x0, 0x0}, v4_int32 = {0xa0ce5129, 0x3f4845c8, 0x0, 0x0},
v2_int64 = {
  0x3f4845c8a0ce5129, 0x0}, uint128 =
0x00000000000000003f4845c8a0ce5129}
xmm11 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0,
0x0},
  v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0,
0x0,
  0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0,
0x0},
  uint128 = 0x00000000000000000000000000000000}
xmm12 {v4_float = {0x0, 0x4b, 0x0, 0x0}, v2_double =
{0x5ffffffffff,
  0x0}, v16_int8 = {0x2c, 0xfd, 0xff, 0xff, 0xff, 0xff, 0x97, 0x42,
0x0,
  0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0xfd2c, 0xffff,
0xffff,
  0x4297, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0xfffffd2c, 0x4297ffff,
0x0,
  0x0}, v2_int64 = {0x4297fffffffffd2c, 0x0},
  uint128 = 0x00000000000000004297fffffffffd2c}
xmm13 {v4_float = {0x0, 0x2, 0x0, 0x0}, v2_double = {0x3,
0x0},
  v16_int8 = {0x0, 0x0, 0x0, 0x58, 0x55, 0x55, 0x9, 0x40, 0x0, 0x0,
0x0, 0x0,
  0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0, 0x5800, 0x5555, 0x4009, 0x0,
0x0,
  0x0, 0x0}, v4_int32 = {0x58000000, 0x40095555, 0x0, 0x0}, v2_int64
= {
  0x4009555558000000, 0x0}, uint128 =
0x00000000000000004009555558000000}
xmm14 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0,
0x0},
---Type <return> to continue, or q <return> to quit---
  v16_int8 = {0x0, 0x0, 0x0, 0x58, 0x55, 0x55, 0x55, 0xbe, 0x0, 0x0,
0x0, 0x0,
  0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0, 0x5800, 0x5555, 0xbe55, 0x0,
0x0,
  0x0, 0x0}, v4_int32 = {0x58000000, 0xbe555555, 0x0, 0x0}, v2_int64
= {
  0xbe55555558000000, 0x0}, uint128 =
0x0000000000000000be55555558000000}
xmm15 {v4_float = {0x0, 0x14, 0x0, 0x0}, v2_double =
{0x8000001,
  0x0}, v16_int8 = {0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0xa0, 0x41, 0x0,
0x0, 0x0,
  0x0, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x0, 0x200, 0x0, 0x41a0, 0x0,
0x0,
  0x0, 0x0}, v4_int32 = {0x2000000, 0x41a00000, 0x0, 0x0}, v2_int64 =
{
  0x41a0000002000000, 0x0}, uint128 =
0x000000000000000041a0000002000000}
mxcsr 0x1fa1 [ IE PE IM DM ZM OM UM PM ]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.videolan.org/pipermail/x264-devel/attachments/20080824/96a6d1b0/attachment.htm