[x264-devel] [Bug Report] Segmentation Fault in 2nd Pass

Rodolfo J. Quesada Zumbado rquesada at roqz.net
Sun Jun 8 18:34:12 CEST 2008


Hi,

I've playing with the x264 code to see what I could find about the
aforementioned bug, using the git head that now produces a
libx264.so.60.

I traced the data flow from x264_rd_cost_i8x8_chroma() in rdo.c and I
found that the error further propagates to cabac.c in
x264_i8x8_chroma_size_cabac(), from there to
x264_cabac_mb_intra_chroma_pred_mode() and there the error occurs when
trying to access a value from the array x264_mb_pred_mode8x8c_fix[7]
with the mysterious index "1151172282", always the same value, and
clearly out of bounds for that array and that's why the segfault occurs.

So I traced back to see where that value might come from, in rdo.c in
function x264_rd_cost_i8x8_chroma() the value comes from the i_mode
variable, and this one comes from analyse.c in x264_intra_rd_refine()
where the structure member a->i_predict8x8chroma holds that weird value.

A bit before in the flow, x264_intra_rd_refine is called from
x264_macroblock_analyse where that "1151172282" value has its origins.

At some point when the variable "x264_mb_analysis_t analysis" is defined
(at the beginning of x264_macroblock_analyse) the structure member
analysis.i_predict8x8chroma seems to get any value in the memory
allocated to it (or something like that), in one of the many function
calls, that value seems to be 1151172282 and from there it propagates
all the way down to the array x264_mb_pred_mode8x8c_fix[].

What is weird is that if I place a pair of fprintfs printing the value
of analysis.i_predict8x8chroma, one after the variable declaration, and
another after the x264_mb_analyse_init() function call... The bug
disappears! :)

Obviously a wrong bug fix! Hehe!

I tried initializing analysis.i_predict8x8chroma = 0 (The values are one
of 0, 1, 2 or 5 at run time) after the call to x264_mb_analyse_init()
and the problem fades, but I don't know what impact it might have, as
the 0 value might travel all the path to the array untouched and maybe
that wasn't the expected or optimal value. Also, x264_mb_analyse_init()
doesn't set any initial value to analysis.i_predict8x8chroma, however, I
don't know if that should be the case.

I encoded a 60 second segment with that quick fix and it played
perfectly.

Hope this helps!

On Fri, 06 Jun 2008 10:33:42 -0600, "Rodolfo J. Quesada Zumbado"
<rquesada at roqz.net> said:
> Sure, here it is:
> 
> 0xb7ee27b0 in x264_rd_cost_i8x8_chroma () from /usr/lib/libx264.so.59
> (gdb) disass $pc-100 $pc+100
> Dump of assembler code from 0xb7ee274c to 0xb7ee2814:
> 0xb7ee274c <x264_rd_cost_i8x8_chroma+540>:	xor    %eax,(%eax)
> 0xb7ee274e <x264_rd_cost_i8x8_chroma+542>:	add    %cl,-0x76ebdbac(%ebx)
> 0xb7ee2754 <x264_rd_cost_i8x8_chroma+548>:	stc
> 0xb7ee2755 <x264_rd_cost_i8x8_chroma+549>:	mov    %esi,%eax
> 0xb7ee2757 <x264_rd_cost_i8x8_chroma+551>:	add    $0x1,%edi
> 0xb7ee275a <x264_rd_cost_i8x8_chroma+554>:	mov    %ebx,(%esp)
> 0xb7ee275d <x264_rd_cost_i8x8_chroma+557>:	add    $0x20,%ebx
> 0xb7ee2760 <x264_rd_cost_i8x8_chroma+560>:	movl   $0xf,0x4(%esp)
> 0xb7ee2768 <x264_rd_cost_i8x8_chroma+568>:	call   0xb7ee2040 
> <block_residual_write_cavlc>
> 0xb7ee276d <x264_rd_cost_i8x8_chroma+573>:	cmp    $0x18,%edi
> 0xb7ee2770 <x264_rd_cost_i8x8_chroma+576>:	jne    0xb7ee2690 
> <x264_rd_cost_i8x8_chroma+352>
> 0xb7ee2776 <x264_rd_cost_i8x8_chroma+582>:	jmp    0xb7ee25ec 
> <x264_rd_cost_i8x8_chroma+188>
> 0xb7ee277b <x264_rd_cost_i8x8_chroma+587>:	nop
> 0xb7ee277c <x264_rd_cost_i8x8_chroma+588>:	lea    0x0(%esi),%esi
> 0xb7ee2780 <x264_rd_cost_i8x8_chroma+592>:	lea    0x1050(%esi),%eax
> 0xb7ee2786 <x264_rd_cost_i8x8_chroma+598>:	mov    %eax,0x4(%esp)
> 0xb7ee278a <x264_rd_cost_i8x8_chroma+602>:	lea    0x40(%esp),%eax
> 0xb7ee278e <x264_rd_cost_i8x8_chroma+606>:	movl   $0x1d0,0x8(%esp)
> 0xb7ee2796 <x264_rd_cost_i8x8_chroma+614>:	mov    %eax,(%esp)
> ---Type <return> to continue, or q <return> to quit---
> 0xb7ee2799 <x264_rd_cost_i8x8_chroma+617>:	call   *0x42f4(%esi)
> 0xb7ee279f <x264_rd_cost_i8x8_chroma+623>:	mov    0x1dfc(%esi),%ebx
> 0xb7ee27a5 <x264_rd_cost_i8x8_chroma+629>:	xor    %ecx,%ecx
> 0xb7ee27a7 <x264_rd_cost_i8x8_chroma+631>:	mov    0x1fe4(%esi),%eax
> 0xb7ee27ad <x264_rd_cost_i8x8_chroma+637>:	test   $0x1,%bl
> 0xb7ee27b0 <x264_rd_cost_i8x8_chroma+640>:	movzbl -0x480c1173(%eax),%edi
> 0xb7ee27b7 <x264_rd_cost_i8x8_chroma+647>:	jne    0xb7ee2a58 
> <x264_rd_cost_i8x8_chroma+1320>
> 0xb7ee27bd <x264_rd_cost_i8x8_chroma+653>:	and    $0x2,%ebx
> 0xb7ee27c0 <x264_rd_cost_i8x8_chroma+656>:	jne    0xb7ee2a40 
> <x264_rd_cost_i8x8_chroma+1296>
> 0xb7ee27c6 <x264_rd_cost_i8x8_chroma+662>:	xor    %edx,%edx
> 0xb7ee27c8 <x264_rd_cost_i8x8_chroma+664>:	test   %edi,%edi
> 0xb7ee27ca <x264_rd_cost_i8x8_chroma+666>:	setg   %dl
> 0xb7ee27cd <x264_rd_cost_i8x8_chroma+669>:	add    $0x40,%ecx
> 0xb7ee27d0 <x264_rd_cost_i8x8_chroma+672>:	movzbl 0x44(%esp,%ecx,1),%eax
> 0xb7ee27d5 <x264_rd_cost_i8x8_chroma+677>:	lea    (%edx,%eax,2),%eax
> 0xb7ee27d8 <x264_rd_cost_i8x8_chroma+680>:	movzbl -0x480c36c0(%eax),%edx
> 0xb7ee27df <x264_rd_cost_i8x8_chroma+687>:	mov    %dl,0x44(%esp,%ecx,1)
> 0xb7ee27e3 <x264_rd_cost_i8x8_chroma+691>:	movzwl 
> -0x480c3580(%eax,%eax,1),%ecx
> 0xb7ee27eb <x264_rd_cost_i8x8_chroma+699>:	add    0x40(%esp),%ecx
> 0xb7ee27ef <x264_rd_cost_i8x8_chroma+703>:	test   %edi,%edi
> ---Type <return> to continue, or q <return> to quit---
> 0xb7ee27f1 <x264_rd_cost_i8x8_chroma+705>:	mov    %ecx,0x40(%esp)
> 0xb7ee27f5 <x264_rd_cost_i8x8_chroma+709>:	je     0xb7ee2856 
> <x264_rd_cost_i8x8_chroma+806>
> 0xb7ee27f7 <x264_rd_cost_i8x8_chroma+711>:	movzbl 0x87(%esp),%edx
> 0xb7ee27ff <x264_rd_cost_i8x8_chroma+719>:	xor    %eax,%eax
> 0xb7ee2801 <x264_rd_cost_i8x8_chroma+721>:	cmp    $0x1,%edi
> 0xb7ee2804 <x264_rd_cost_i8x8_chroma+724>:	setg   %al
> 0xb7ee2807 <x264_rd_cost_i8x8_chroma+727>:	lea    (%eax,%edx,2),%edx
> 0xb7ee280a <x264_rd_cost_i8x8_chroma+730>:	movzwl 
> -0x480c3580(%edx,%edx,1),%eax
> 0xb7ee2812 <x264_rd_cost_i8x8_chroma+738>:	movzbl -0x480c36c0(%edx),%ebx
> End of assembler dump.
> (gdb)
> 
> Here is 'info all-registers' too:
> 
> (gdb) info all-registers
> eax            0xbd048ad3	-1123775789
> ecx            0x0	0
> edx            0x0	0
> ebx            0x1	1
> esp            0xbff886e0	0xbff886e0
> ebp            0x2db	0x2db
> esi            0x8bc2240	146547264
> edi            0x0	0
> eip            0xb7ee27b0	0xb7ee27b0 <x264_rd_cost_i8x8_chroma+640>
> eflags         0x10202	[ IF RF ]
> cs             0x73	115
> ss             0x7b	123
> ds             0x7b	123
> es             0x7b	123
> fs             0x0	0
> gs             0x33	51
> st0            -nan(0xc4e380c00000000)	(raw 0xffff0c4e380c00000000)
> st1            -nan(0x403c342c48584e38)	(raw 0xffff403c342c48584e38)
> st2            -nan(0x313c3f43310a1c21)	(raw 0xffff313c3f43310a1c21)
> st3            -nan(0x672731534943413f)	(raw 0xffff672731534943413f)
> st4            -inf	(raw 0xffff0000000000000000)
> st5            -nan(0x8585858585858585)	(raw 0xffff8585858585858585)
> st6            -nan(0x8585858585858585)	(raw 0xffff8585858585858585)
> ---Type <return> to continue, or q <return> to quit---
> st7            -inf	(raw 0xffff0000000000000000)
> fctrl          0x37f	895
> fstat          0x20	32
> ftag           0xaaaa	43690
> fiseg          0x0	0
> fioff          0x0	0
> foseg          0x0	0
> fooff          0x0	0
> fop            0x0	0
> xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0xb0, 0x2, 0x0, 0x0, 0x58, 0x1, 0x0, 0x0, 0x58, 0x1, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0, 0x0}, v8_int16 = {0x2b0, 0x0, 0x158, 0x0, 0x158, 
> 0x0, 0x0,
>      0x0}, v4_int32 = {0x2b0, 0x158, 0x158, 0x0}, v2_int64 = 
> {0x158000002b0,
>      0x158}, uint128 = 0x000000000000015800000158000002b0}
> xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> xmm2           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> xmm3           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
> ---Type <return> to continue, or q <return> to quit---
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> xmm5           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> xmm6           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
>    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 
> 0x0,
>      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 
> 0x0},
>    uint128 = 0x00000000000000000000000000000000}
> mxcsr          0x1fa0	[ PE IM DM ZM OM UM PM ]
> mm0            {uint64 = 0xc4e380c00000000, v2_int32 = {0x0, 0xc4e380c},
>    v4_int16 = {0x0, 0x0, 0x380c, 0xc4e}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 
> 0xc,
>      0x38, 0x4e, 0xc}}
> ---Type <return> to continue, or q <return> to quit---
> mm1            {uint64 = 0x403c342c48584e38, v2_int32 = {0x48584e38,
>      0x403c342c}, v4_int16 = {0x4e38, 0x4858, 0x342c, 0x403c}, v8_int8 = 
> {0x38,
>      0x4e, 0x58, 0x48, 0x2c, 0x34, 0x3c, 0x40}}
> mm2            {uint64 = 0x313c3f43310a1c21, v2_int32 = {0x310a1c21,
>      0x313c3f43}, v4_int16 = {0x1c21, 0x310a, 0x3f43, 0x313c}, v8_int8 = 
> {0x21,
>      0x1c, 0xa, 0x31, 0x43, 0x3f, 0x3c, 0x31}}
> mm3            {uint64 = 0x672731534943413f, v2_int32 = {0x4943413f,
>      0x67273153}, v4_int16 = {0x413f, 0x4943, 0x3153, 0x6727}, v8_int8 = 
> {0x3f,
>      0x41, 0x43, 0x49, 0x53, 0x31, 0x27, 0x67}}
> mm4            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0,
> 0x0,
>      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
> mm5            {uint64 = 0x8585858585858585, v2_int32 = {0x85858585,
>      0x85858585}, v4_int16 = {0x8585, 0x8585, 0x8585, 0x8585}, v8_int8 = 
> {0x85,
>      0x85, 0x85, 0x85, 0x85, 0x85, 0x85, 0x85}}
> mm6            {uint64 = 0x8585858585858585, v2_int32 = {0x85858585,
>      0x85858585}, v4_int16 = {0x8585, 0x8585, 0x8585, 0x8585}, v8_int8 = 
> {0x85,
>      0x85, 0x85, 0x85, 0x85, 0x85, 0x85, 0x85}}
> mm7            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0,
> 0x0,
>      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
> (gdb)
> 
> 
> 
> Jason Garrett-Glaser wrote:
> >> 0xb7fd57b0 in x264_rd_cost_i8x8_chroma () from /usr/lib/libx264.so.59
> >> (gdb) bt
> >> #0  0xb7fd57b0 in x264_rd_cost_i8x8_chroma () from /usr/lib/libx264.so.59
> >> #1  0x00000000 in ?? ()
> > 
> > Can you give a disass, e.g. "disass $pc-50 $pc+50 or something like
> > that?  The function is relatively small so the asm might be useful.
> > 
> > Dark Shikari
> > _______________________________________________
> > x264-devel mailing list
> > x264-devel at videolan.org
> > http://mailman.videolan.org/listinfo/x264-devel
> 
> -- 
> Rodolfo J. Quesada Zumbado
> rquesada at roqz.net
> 


More information about the x264-devel mailing list