[x264-devel] core dump on solaris 10/space with x264-snapshot-20100112-2245

[Håkan Hjort]

On Sat, Jan 16, 2010 at 06:14, Chris Brookes <cbrookes at gmail.com> wrote:
> 2010/1/15 Håkan Hjort <hakan.hjort at gmail.com>:
>
>> I think interesting info would be the the register values and the
>> surrounding machine code.
>> Not sure how to do it in dbx, using gdb it would be something like;
>> info regs
>> disassemble $pc-32 $pc+16
>> p dequant_mf
>
>
>
>
> t at 1 (l at 1) program terminated by signal SEGV (no mapping at the fault address)
> Current function is x264_mb_encode_8x8_chroma
>   59       int dmf = dequant_mf[i_qp%6][0];
> (dbx) where
> current thread: t at 1
> =>[1] x264_mb_encode_8x8_chroma(h = 0xab340, b_inter = 0, i_qp = 19),
> line 59 in "macroblock.c"
>  [2] x264_macroblock_encode(h = 0xab340), line 892 in "macroblock.c"
>  [3] x264_rd_cost_mb(h = 0xab340, i_lambda2 = 2925), line 145 in "rdo.c"
>  [4] x264_macroblock_analyse(h = 0xab340), line 992 in "analyse.c"
>  [5] x264_slice_write(h = 0xab340), line 1672 in "encoder.c"
>  [6] x264_encoder_encode(h = 0xab340, pp_nal = 0xffbff18c, pi_nal =
> 0xffbff188, pic_in = 0x3f, pic_out = 0xffbff1e0), line 1904 in
> "encoder.c"
>  [7] main(argc = 0, argv = 0xa9a10), line 1236 in "x264.c"
>
>
>
> (dbx) regs
> current thread: t at 1
> current frame:  [1]
> g0-g3    0x00000000 0x00000000 0x00000000 0x00000000
> g4-g7    0x00000000 0x000aef60 0x00000000 0xff362a00
> o0-o3    0xefed4800 0x00000000 0xffbfb644 0x00002c00
> o4-o7    0xfff70000 0x00750000 0xffbfb650 0x0000fff7
> l0-l3    0xffbfb51b 0x000adf40 0x000ab340 0x00000002
> l4-l7    0x0000007e 0xfffffffe 0x00000002 0x00020000
> i0-i3    0x000ab340 0x00000000 0x00000075 0x000b1d40
> i4-i7    0xfffffff7 0x00000000 0xffbfb7f0 0x0007ba00
> y        0x00000000
> ccr      0xfe001000
> pc       0x0007a638:x264_mb_encode_8x8_chroma+0x5a0    ld       [%i3 + %o0], %l1
> npc      0x0007a63c:x264_mb_encode_8x8_chroma+0x5a4    cmp      %l0, 0
>

%i3 = 0x000b1d40
%o0 = 0xefed4800
ld       [%i3 + %o0], %l1

If this is supposed to be
     int dmf = dequant_mf[i_qp%6][0];
and surrounding code indicated that it is.. then the modulo 6 (i_qp%6)
get seriously
screwed up because 0xefed4800 is hardly the right answer..

It looks like it's GCC's divmod that causes the mis-compilation, used
because both
i_qp%6 and i_qp/6 are used close together.

I compiled it my self using GCC 3.4.4 though.. this is the code that
sets fp-244 (I checked
it's what is used in the same ld instruction) Unrelated instructions
marked with #

    mov 6, %o1              ! load 6
    call    .rem, 0            ! the % part
    ld  [%fp+76], %o0   ! load i_qp (this is the delay slot of the call)
#  ld  [%fp+72], %l4
#  sll %l4, 2, %o7
#  st  %o7, [%fp-260]
#  ld  [%fp-260], %o4
#  add %o4, %i0, %i2
#  st  %i2, [%fp-288]
    ld  [%fp+76], %o0 ! another i_qp load (overwriting the .rem result!!)
    call    .div, 0            ! the / part
    mov 6, %o1            ! load 6 (this is the delay slot of the call)
    sll %l6, 6, %l7       ! this is what sets %l7 but i's %l6 >> 6 and
%l6 is uninitalized
#  st  %l6, [%fp-232]
#  ld  [%fp+76], %l5
#  sll %l5, 5, %l6
    st  %l7, [%fp-244] ! spill %l7 to the stack (should have contained i_qp%6)


Anyway something must have gone wrong for you in configure.  The
script should have
added -mcpu=ultrasparc to your CFLAGS. If this had worked you would
not be getting
and calls to (.umul|.rem|.div) and so avoiding this GCC bug. Beside
things being faster :)

  sparc)
    if test "$(uname -m)" = "sun4u"; then
      ARCH="UltraSparc"
      CFLAGS="$CFLAGS -mcpu=ultrasparc"

Regards,
Håkan