[x264-devel] [Bug Report] Segmentation Fault in 2nd Pass

[Baptiste Coudurier]

Hi, I also experience the bug.

Rodolfo Quesada wrote:
>> [...]
>>
>> Jason Garrett-Glaser wrote:
>>>> 0xb7fd57b0 in x264_rd_cost_i8x8_chroma () from /usr/lib/libx264.so.59
>>>> (gdb) bt
>>>> #0  0xb7fd57b0 in x264_rd_cost_i8x8_chroma () from /usr/lib/libx264.so.59
>>>> #1  0x00000000 in ?? ()
>>> Can you give a disass, e.g. "disass $pc-50 $pc+50 or something like
>>> that?  The function is relatively small so the asm might be useful.
>>>


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7be96b0 (LWP 11507)]
x264_rd_cost_i8x8_chroma (h=0x96802a0, i_lambda2=580, i_mode=683213150,
b_dct=0) at encoder/cavlc.c:786
786         h->out.bs.i_bits_encoded = bs_size_ue(
x264_mb_pred_mode8x8c_fix[ h->mb.i_chroma_pred_mode ] );
(gdb) bt
#0  x264_rd_cost_i8x8_chroma (h=0x96802a0, i_lambda2=580,
i_mode=683213150, b_dct=0) at encoder/cavlc.c:786
#1  0x08490c98 in x264_intra_rd_refine (h=0x96802a0, a=0xbfc58e24) at
encoder/analyse.c:931
#2  0x0849435c in x264_macroblock_analyse (h=0x96802a0) at
encoder/analyse.c:2584
#3  0x0846a6cf in x264_slice_write (h=0x96802a0) at encoder/encoder.c:1053
#4  0x0846ad54 in x264_stack_align ()
#5  0x084664ed in x264_slices_write (h=0x96802a0) at encoder/encoder.c:1197
#6  0x0846874e in x264_encoder_encode (h=0x96802a0, pp_nal=0xbfc5b178,
pi_nal=0xbfc5b174, pic_in=0x9572fac,
    pic_out=0xbfc5b138) at encoder/encoder.c:1458
#7  0x083c6692 in X264_frame (ctx=0x9574230, buf=0x95d11c0 "",
bufsize=307200, data=0xbfc5b2cc)
    at libavcodec/libx264.c:91
#8  0x080fe590 in avcodec_encode_video (avctx=0x9574230, buf=0x95d11c0
"", buf_size=307200, pict=0xbfc5b2cc)
    at libavcodec/utils.c:868
#9  0x08063785 in output_packet (ist=0x9568230, ist_index=0,
ost_table=0x9568300, nb_ostreams=2, pkt=0xbfc5c3f8)
    at ffmpeg.c:905
#10 0x08067d52 in av_encode (output_files=0x858dbe0, nb_output_files=1,
input_files=0x858dae0, nb_input_files=1,
    stream_maps=0x858dc40, nb_stream_maps=0) at ffmpeg.c:2059
#11 0x08068286 in main (argc=Cannot access memory at address 0x10
) at ffmpeg.c:3850
(gdb) disass $pc-50 $pc+50
Dump of assembler code from 0x848644b to 0x84864af:
0x0848644b <x264_rd_cost_i8x8_chroma+323>:      clc
0x0848644c <x264_rd_cost_i8x8_chroma+324>:      std
0x0848644d <x264_rd_cost_i8x8_chroma+325>:      (bad)
0x0848644e <x264_rd_cost_i8x8_chroma+326>:      decl   -0x46221708(%ecx)
0x08486454 <x264_rd_cost_i8x8_chroma+332>:      (bad)
0x08486455 <x264_rd_cost_i8x8_chroma+333>:      incl   -0x397cfe3d(%ebx)
0x0848645b <x264_rd_cost_i8x8_chroma+339>:      and    %al,-0x2b8ae705(%ebx)
0x08486461 <x264_rd_cost_i8x8_chroma+345>:      mov    0xc(%ebp),%eax
0x08486464 <x264_rd_cost_i8x8_chroma+348>:      imull  -0x1e8(%ebp)
0x0848646a <x264_rd_cost_i8x8_chroma+354>:      add    $0x8000,%eax
0x0848646f <x264_rd_cost_i8x8_chroma+359>:      adc    $0x0,%edx
0x08486472 <x264_rd_cost_i8x8_chroma+362>:      shrd   $0x10,%edx,%eax
0x08486476 <x264_rd_cost_i8x8_chroma+366>:      mov    %eax,%edx
0x08486478 <x264_rd_cost_i8x8_chroma+368>:      jmp    0x8486595
<x264_rd_cost_i8x8_chroma+653>
0x0848647d <x264_rd_cost_i8x8_chroma+373>:      movzbl 0x857e88d(%ebx),%eax
0x08486484 <x264_rd_cost_i8x8_chroma+380>:      cmp    $0xfe,%eax
0x08486489 <x264_rd_cost_i8x8_chroma+385>:      ja     0x8486494
<x264_rd_cost_i8x8_chroma+396>
0x0848648b <x264_rd_cost_i8x8_chroma+387>:      movzbl 0x857e5e0(%eax),%eax
0x08486492 <x264_rd_cost_i8x8_chroma+394>:      jmp    0x84864b5
<x264_rd_cost_i8x8_chroma+429>
0x08486494 <x264_rd_cost_i8x8_chroma+396>:      add    $0x1,%eax
0x08486497 <x264_rd_cost_i8x8_chroma+399>:      mov    $0x0,%edx
0x0848649c <x264_rd_cost_i8x8_chroma+404>:      cmp    $0xff,%eax
0x084864a1 <x264_rd_cost_i8x8_chroma+409>:      jbe    0x84864ac
<x264_rd_cost_i8x8_chroma+420>
0x084864a3 <x264_rd_cost_i8x8_chroma+411>:      add    $0x10,%edx
0x084864a6 <x264_rd_cost_i8x8_chroma+414>:      shr    $0x8,%eax
0x084864a9 <x264_rd_cost_i8x8_chroma+417>:      sub    $0x1,%eax
0x084864ac <x264_rd_cost_i8x8_chroma+420>:      movzbl 0x857e5e0(%eax),%eax
End of assembler dump.

./configure --enable-debug.

I hope this helps. It seems that it goes away when I remove b frames.

-- 
Baptiste COUDURIER                              GnuPG Key Id: 0x5C1ABAAA
Smartjog USA Inc.                                http://www.smartjog.com
Key fingerprint                 8D77134D20CC9220201FC5DB0AC9325C5C1ABAAA