[x265] [PATCH] distortion: change data type of distortion for 10 bit to avoid overflow

[Divya Manivannan]

When distortion is added for all three planes, there will be a overflow.

On Mon, Oct 12, 2015 at 7:56 PM, chen <chenm003 at 163.com> wrote:

>
>
> At 2015-10-12 18:29:54,"Divya Manivannan" <divya at multicorewareinc.com> wrote:
> ># HG changeset patch
> ># User Divya Manivannan <divya at multicorewareinc.com>
> ># Date 1443607299 -19800
> >#      Wed Sep 30 15:31:39 2015 +0530
> ># Node ID 534d80f9272b43485cca8bb0c0a522ec8abeaa13
> ># Parent  b6156a08b1def3584647f26096866c1a0c11e54a
> >distortion: change data type of distortion for 10 bit to avoid overflow
> >            while adding for Y, U, V planes
> >
> >diff -r b6156a08b1de -r 534d80f9272b source/common/common.h
> >--- a/source/common/common.h	Fri Oct 09 20:45:59 2015 +0530
> >+++ b/source/common/common.h	Wed Sep 30 15:31:39 2015 +0530
> >@@ -141,6 +141,12 @@
> > typedef uint64_t sse_ret_t;
> > #endif
> >
> >+#if X265_DEPTH < 10
> >+typedef uint32_t dist_ret_t;
> >+#else
> >+typedef uint64_t dist_ret_t;
> >+#endif
> >+
> > #ifndef NULL
> > #define NULL 0
> > #endif
> >diff -r b6156a08b1de -r 534d80f9272b source/encoder/rdcost.h
> >--- a/source/encoder/rdcost.h	Fri Oct 09 20:45:59 2015 +0530
> >+++ b/source/encoder/rdcost.h	Wed Sep 30 15:31:39 2015 +0530
> >@@ -89,9 +89,9 @@
> >         m_lambda = (uint64_t)floor(256.0 * lambda);
> >     }
> >
> >-    inline uint64_t calcRdCost(sse_ret_t distortion, uint32_t bits) const
> >+    inline uint64_t calcRdCost(dist_ret_t distortion, uint32_t bits) const
> >     {
> >-#if X265_DEPTH <= 10
> >+#if X265_DEPTH < 10
> 10bpp overflow?
>
> >         X265_CHECK(bits <= (UINT64_MAX - 128) / m_lambda2,
> >                    "calcRdCost wrap detected dist: %u, bits %u, lambda: " X265_LL "\n",
> >                    distortion, bits, m_lambda2);
> >@@ -116,9 +116,9 @@
> >     }
>
>
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20151013/e7c9be99/attachment-0001.html>