[x264-devel] Integer overflow in x264_weights_analyse

[Xuezhi Yan]

Hi, while running x264 under Clang's UndefinedBehaviorSanitizer, I'm
hitting integer overflow in the following line:

https://code.videolan.org/videolan/x264/blob/master/encoder/slicetype.c#L408

where "ref_mean[plane]" is 0.000003 because the reference frame's
"i_pixel_sum[plane]" is 0 ("ref_mean[plane]" is not zero because zero_bias
is added). With the following clamping using float I am able to fix the
problem locally:

408,409c408,409 encoder/slicetype.c
<    cur_scale = (1 << mindenom) * (fenc_mean[plane] - cur_offset) /
ref_mean[plane] + 0.5f;
<    cur_scale = x264_clip3( cur_scale, 0, 127 );
---
>    float temp = (1 << mindenom) * (fenc_mean[plane] - cur_offset) /
ref_mean[plane] + 0.5f;
>    cur_scale = temp > 127.0f ? 127 : (temp < 0.0f ? 0 : x264_clip3( temp,
0, 127 ));

If this looks good, can someone help me merge it? Otherwise is there a more
appropriate fix?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190418/486192ed/attachment.html>