[x264-devel] Integer overflow in x264_weights_analyse

[Xuezhi Yan]

Thanks in advance for any help!

On Thu, Apr 18, 2019 at 4:43 PM Xuezhi Yan <xuezhiy at google.com> wrote:

> Hi, while running x264 under Clang's UndefinedBehaviorSanitizer, I'm
> hitting integer overflow in the following line:
>
>
> https://code.videolan.org/videolan/x264/blob/master/encoder/slicetype.c#L408
>
> where "ref_mean[plane]" is 0.000003 because the reference frame's
> "i_pixel_sum[plane]" is 0 ("ref_mean[plane]" is not zero because zero_bias
> is added). With the following clamping using float I am able to fix the
> problem locally:
>
> 408,409c408,409 encoder/slicetype.c
> <    cur_scale = (1 << mindenom) * (fenc_mean[plane] - cur_offset) /
> ref_mean[plane] + 0.5f;
> <    cur_scale = x264_clip3( cur_scale, 0, 127 );
> ---
> >    float temp = (1 << mindenom) * (fenc_mean[plane] - cur_offset) /
> ref_mean[plane] + 0.5f;
> >    cur_scale = temp > 127.0f ? 127 : (temp < 0.0f ? 0 : x264_clip3(
> temp, 0, 127 ));
>
> If this looks good, can someone help me merge it? Otherwise is there a
> more appropriate fix?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190418/6ace48e9/attachment.html>