[x264-devel] [BUG] segfault in x264_weight_cost_init_luma

[Corey Hickey]

On 2010-10-09 22:35, Jason Garrett-Glaser wrote:
> On Sat, Oct 9, 2010 at 9:30 PM, Corey Hickey <bugfood-ml at fatooh.org> wrote:
>> Hi,
>>
>> I am encountering a segfault under the following conditions:
>> 1. using mencoder
>> 2. with x264 set to use one thread
>> 3. when the second pass has 2 more frames than the first
>>
>> x264 prints the following errors:
>>
>> x264 [error]: Incomplete MB-tree stats file.
>> x264_encoder_encode failed
>> x264 [error]: 2nd pass has more frames than 1st pass (1)
>> x264 [error]: continuing anyway, at constant QP=24
>> x264 [error]: disabling adaptive B-frames
>>
>>
>> (full logs attached)
>>
>>
>> Originally, it would appear to be mencoder's bug that it is feeding x264
>> more frames, but I am having trouble debugging that bug because of this
>> segfault. Rather than send in the large file with which I originally
>> encountered the issue, I can make the segfault happen by manually
>> specifying more frames on the second pass.
>>
>> I distilled the steps to reproduce down to a small script, which I have
>> attached, along with some gdb info from the second pass.
>>
>> I did my best to reproduce this with the x264 program itself, but I
>> haven't been able to. It seems that x264 detects that there are too many
>> frames in the second pass and quits first.
> 
> Is mencoder passing frames to x264 after x264_encoder_encode fails?
> You're really not supposed to do that.

Looks like. I was fooled by the "continuing anyway" message into
thinking x264 intended to handle the extra frames gracefully.

I don't know how to deal with it in mencoder. I don't see anything in
the code for dealing with fatal errors from the video
filter-->encoder-->muxer chain. mencoder just treats the failure as
"this frame got dropped".

But I'll take it to an mplayer list later on.

Thanks,
Corey