[x265] [PATCH] distortion: change data type of distortion for 10 bit to avoid overflow

[chen]

At 2015-10-12 18:29:54,"Divya Manivannan" <divya at multicorewareinc.com> wrote:
># HG changeset patch
># User Divya Manivannan <divya at multicorewareinc.com>
># Date 1443607299 -19800
>#      Wed Sep 30 15:31:39 2015 +0530
># Node ID 534d80f9272b43485cca8bb0c0a522ec8abeaa13
># Parent  b6156a08b1def3584647f26096866c1a0c11e54a
>distortion: change data type of distortion for 10 bit to avoid overflow
>            while adding for Y, U, V planes
>
>diff -r b6156a08b1de -r 534d80f9272b source/common/common.h
>--- a/source/common/common.h	Fri Oct 09 20:45:59 2015 +0530
>+++ b/source/common/common.h	Wed Sep 30 15:31:39 2015 +0530
>@@ -141,6 +141,12 @@
> typedef uint64_t sse_ret_t;
> #endif
> 
>+#if X265_DEPTH < 10
>+typedef uint32_t dist_ret_t;
>+#else
>+typedef uint64_t dist_ret_t;
>+#endif
>+
> #ifndef NULL
> #define NULL 0
> #endif
>diff -r b6156a08b1de -r 534d80f9272b source/encoder/rdcost.h
>--- a/source/encoder/rdcost.h	Fri Oct 09 20:45:59 2015 +0530
>+++ b/source/encoder/rdcost.h	Wed Sep 30 15:31:39 2015 +0530
>@@ -89,9 +89,9 @@
>         m_lambda = (uint64_t)floor(256.0 * lambda);
>     }
> 
>-    inline uint64_t calcRdCost(sse_ret_t distortion, uint32_t bits) const
>+    inline uint64_t calcRdCost(dist_ret_t distortion, uint32_t bits) const
>     {
>-#if X265_DEPTH <= 10
>+#if X265_DEPTH < 10
10bpp overflow?

>         X265_CHECK(bits <= (UINT64_MAX - 128) / m_lambda2,
>                    "calcRdCost wrap detected dist: %u, bits %u, lambda: " X265_LL "\n",
>                    distortion, bits, m_lambda2);
>@@ -116,9 +116,9 @@
>     }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20151012/d953e3fa/attachment-0001.html>