[x264-devel] Integer overflow in x264_weights_analyse

[BugMaster]

On Thu, 18 Apr 2019 16:43:05 -0700, Xuezhi Yan wrote:
> Hi, while running x264 under Clang's UndefinedBehaviorSanitizer,
> I'm hitting integer overflow in the following line:



> https://code.videolan.org/videolan/x264/blob/master/encoder/slicetype.c#L408



> where "ref_mean[plane]" is 0.000003 because the reference frame's
> "i_pixel_sum[plane]" is 0 ("ref_mean[plane]" is not zero because
> zero_bias is added). With the following clamping using float I am able to fix the problem locally:


> 408,409c408,409 encoder/slicetype.c
> <    cur_scale = (1 << mindenom) * (fenc_mean[plane] - cur_offset) / ref_mean[plane] + 0.5f;
> <    cur_scale = x264_clip3( cur_scale, 0, 127 );
> ---
>>    float temp = (1 << mindenom) * (fenc_mean[plane] - cur_offset) / ref_mean[plane] + 0.5f;
>>    cur_scale = temp > 127.0f ? 127 : (temp < 0.0f ? 0 : x264_clip3( temp, 0, 127 ));


> If this looks good, can someone help me merge it? Otherwise is there a more appropriate fix?

Hi.

Thanks for bug report. I have attached patch with more correct/clean fix.
Check that it works for you.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-integer-overflow-detected-by-UBSan-in-weightp-an.patch
Type: application/octet-stream
Size: 1303 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190423/9c02bed6/attachment.obj>