[x265] unsigned promotion prevents encoding frames with negative strides

[chen]

Hi,


Thank you report this bug.
I think the root cause is not sizeof(), the negative stride is invalid in encoder/decoder core.
To avoid these invalid input parameters, the x264 insert a middle-layer that convert color space and images, but x265 doesn't it.


Of course, crash is worst way to report invalid input parameters, we will fix it soon.


Thanks,
Min

At 2018-01-18 00:24:19, "Vittorio Giovara" <vittorio.giovara at gmail.com> wrote:

Hi,

I'm developing an app which flips a frame in this way



    frame->stride[0] *= -1;
    frame->stride[1] *= -1;
    frame->stride[2] *= -1;
    frame->data[0]    = frame->data[1] + frame->stride[0];
    frame->data[1]    = frame->data[2] + frame->stride[1];
    frame->data[2]    = frame->data[2] + (((frame->height >> desc->log2_chroma_h) - 1) * -frame->stride[2]);



and then proceeds to encode it with either x264 or x265.


When feeding this frame to x265_encode(), I get a crash in x265_upShift_16_avx2 at picyuv.cpp:322. While debugging I noticed that there is something wrong in the division for the input stride: for a 640x480 yuv 10 bit frame (with negative strides) the computed value is 9223372036854775168 instead of -640.


I believe the problem lies in the operation pic.stride[0] / sizeof(*yShort) where the result of every division is promoted to unsigned since sizeof() returns size_t (aka unsigned long). Unfortunately this issue is spread across the entire codebase, affecting every arithmetic operation whenever a stride is computed or updated. I think this was introduced four years ago in https://bitbucket.org/multicoreware/x265/commits/eadec14402d6.



The solution would be to properly cast every sizeof() operation to ssize_t or int, or modify the internal functions to operate on bytes instead of pixels. The same frame fed to x264 is correctly encoded just fine.


Can anybody verify and apply a proper fix? Thank you

--

Vittorio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20180118/52f49ac3/attachment.html>