[x265] Patch for issue #422 (TestBench failure on FreeBSD with AVX enabled)

[Dimitry Andric]

Hi,

As described in https://bitbucket.org/multicoreware/x265/issues/422,
and originally in https://bugs.freebsd.org/229788, on FreeBSD 11 (or 12)
with AVX enabled, TestBench fails with:

 ===>  Testing for x265-2.6_1
 /symbion/ports/multimedia/x265/work/x265_v2.6/source/test/TestBench
 Using random seed 5B4B6D10 12bit
 Testing primitives: SSE2
 Testing primitives: SSE3
 Testing primitives: SSSE3
 cuTreeFix8Pack failed

 x265: asm primitive has failed. Go and fix that Right Now!

This is due to undefined behavior in cuTreeFix8Pack(), where a double
value is cast directly to uint16_t.  If the double value is negative,
the resulting value from the cast is undefined.

The patch committed in the FreeBSD multimedia/x265 port is as follows:

--- a/common/pixel.cpp
+++ b/common/pixel.cpp
@@ -922,7 +922,7 @@ static void estimateCUPropagateCost(int* dst, const ui

static void cuTreeFix8Pack(uint16_t *dst, double *src, int count)
{
    for (int i = 0; i < count; i++)

-        dst[i] = (uint16_t)(src[i] * 256.0);
+        dst[i] = (uint16_t)(int16_t)(src[i] * 256.0);

}

static void cuTreeFix8Unpack(double *dst, uint16_t *src, int count)

Please consider applying this patch to the default and stable branches.

-Dimitry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 223 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20180815/b5f4b104/attachment.sig>