[x264-devel] Integer overflow in x264_weights_analyse

[Xuezhi Yan]

Yes, confirm that this fix works. Thanks for making this fix.

On Mon, Apr 22, 2019 at 3:21 PM BugMaster <BugMaster at narod.ru> wrote:

> On Thu, 18 Apr 2019 16:43:05 -0700, Xuezhi Yan wrote:
> > Hi, while running x264 under Clang's UndefinedBehaviorSanitizer,
> > I'm hitting integer overflow in the following line:
>
>
>
> >
> https://code.videolan.org/videolan/x264/blob/master/encoder/slicetype.c#L408
>
>
>
> > where "ref_mean[plane]" is 0.000003 because the reference frame's
> > "i_pixel_sum[plane]" is 0 ("ref_mean[plane]" is not zero because
> > zero_bias is added). With the following clamping using float I am able
> to fix the problem locally:
>
>
> > 408,409c408,409 encoder/slicetype.c
> > <    cur_scale = (1 << mindenom) * (fenc_mean[plane] - cur_offset) /
> ref_mean[plane] + 0.5f;
> > <    cur_scale = x264_clip3( cur_scale, 0, 127 );
> > ---
> >>    float temp = (1 << mindenom) * (fenc_mean[plane] - cur_offset) /
> ref_mean[plane] + 0.5f;
> >>    cur_scale = temp > 127.0f ? 127 : (temp < 0.0f ? 0 : x264_clip3(
> temp, 0, 127 ));
>
>
> > If this looks good, can someone help me merge it? Otherwise is there a
> more appropriate fix?
>
> Hi.
>
> Thanks for bug report. I have attached patch with more correct/clean fix.
> Check that it works for you._______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> https://mailman.videolan.org/listinfo/x264-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190422/b3a36313/attachment.html>