[x265] unsigned promotion prevents encoding frames with negative strides

[Vittorio Giovara]

Hi,
I'm developing an app which flips a frame in this way

    frame->stride[0] *= -1;
    frame->stride[1] *= -1;
    frame->stride[2] *= -1;
    frame->data[0]    = frame->data[1] + frame->stride[0];
    frame->data[1]    = frame->data[2] + frame->stride[1];
    frame->data[2]    = frame->data[2] + (((frame->height >>
desc->log2_chroma_h) - 1) * -frame->stride[2]);

and then proceeds to encode it with either x264 or x265.

When feeding this frame to x265_encode(), I get a crash in
x265_upShift_16_avx2 at picyuv.cpp:322. While debugging I noticed that
there is something wrong in the division for the input stride: for a
640x480 yuv 10 bit frame (with negative strides) the computed value is
9223372036854775168 instead of -640.

I believe the problem lies in the operation pic.stride[0] /
sizeof(*yShort) where
the result of every division is promoted to unsigned since sizeof() returns
size_t (aka unsigned long). Unfortunately this issue is spread across the
entire codebase, affecting every arithmetic operation whenever a stride is
computed or updated. I think this was introduced four years ago in
https://bitbucket.org/multicoreware/x265/commits/eadec14402d6.

The solution would be to properly cast every sizeof() operation to ssize_t
or int, or modify the internal functions to operate on bytes instead of
pixels. The same frame fed to x264 is correctly encoded just fine.

Can anybody verify and apply a proper fix? Thank you
-- 
Vittorio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20180117/1cd5f11e/attachment.html>