[x264-devel] [BUG] segfault when second pass has more frames than the first

[Corey Hickey]

Hello,

I ran into this segfault while I was trying to figure out another bug,
and thought I'd see if I could fix it. I know that second passes should
have the same number of frames as the first, but x264 appears to have
code to handle this already, and that code just isn't quite working.

To reproduce:

$ mencoder -demuxer rawvideo -rawvideo w=16:h=16 /dev/zero \
  -o /dev/null -ovc x264 -x264encopts pass=1:bitrate=200 -frames 1

$ mencoder -demuxer rawvideo -rawvideo w=16:h=16 /dev/zero \
  -o /dev/null -ovc x264 -x264encopts pass=2:bitrate=200 -frames 2

The segfault occurs in an fwrite() call at encoder/ratecontrol.c:1387.
The file stream is NULL because the file was never opened. Here's why:

1. At encoder/ratecontrol.c:785, the mbtree stat file is opened if:
   h->param.rc.b_mb_tree && !h->param.rc.b_stat_read

2. At encoder/ratecontrol.c:1293, when reading a frame that wasn't
   recorded in the 2-pass stat file, h->param.rc.b_stat_read is set to
   0.

3. At encoder/ratecontrol.c:1380, x264 chooses to write to the mbtree
   stat file if:
   h->param.rc.b_mb_tree && h->fenc->b_kept_as_ref &&
   !h->param.rc.b_stat_read
   Since h->param.rc.b_stat_read is now 0, the above test is true.


I've attached a patch to fix this segfault, but it still doesn't make
the encode work properly. The next problem is at
encoder/slicetype.c:1024, in x264_slicetype_analyse():
assert( h->frames.b_have_lowres );

Normally, x264_slicetype_analyze() isn't called in the second pass--but
it is when h->param.rc.b_stat_read is 0 (see #2 above). I don't know a
clean way to fix this; see git commit
acee2d5168a39f301b7cda1d4effe943e321e1f8.

Anyway, I hope this report is a little useful.

Thanks,
Corey
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fix-fwrite-segfault.diff
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20100215/9b0e6173/attachment.asc>