[x264-devel] x264 crashs with 32x32 samples + b-frames

[BugMaster]

I make a patch for fixing this bug.

> x264 crashs when encoding video that have height or width equal 32
> (or height equal 16 + interlaced) and use b-frames. Crash occure in
> function 'x264_slicetype_frame_cost' at:

>     ...
>     if( b_intra_penalty )
>     {
>         // arbitrary penalty for I-blocks after B-frames
>         int nmb = (h->sps->i_mb_width - 2) * (h->sps->i_mb_height - 2);
>         i_score += i_score * frames[b]->i_intra_mbs[b-p0] / (nmb * 8);  // <- division by zero
>     }
>     ...

> Also if height or width will be equal 16 then this calculation will
> be incorrect ('nmb' will be negative). So all places in code that
> use 'sps->i_mb_width' or 'sps->i_mb_height' must be checked for
> errors when this variables are small.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 32x32samples_crash.diff
Type: application/octet-stream
Size: 2638 bytes
Desc: not available
Url : http://mailman.videolan.org/pipermail/x264-devel/attachments/20071016/1256d8fe/attachment.obj