[x265] [PATCH] distortion: change data type of distortion for 10 bit to avoid overflow
Divya Manivannan
divya at multicorewareinc.com
Tue Oct 13 06:17:10 CEST 2015
When distortion is added for all three planes, there will be a overflow.
On Mon, Oct 12, 2015 at 7:56 PM, chen <chenm003 at 163.com> wrote:
>
>
> At 2015-10-12 18:29:54,"Divya Manivannan" <divya at multicorewareinc.com> wrote:
> ># HG changeset patch
> ># User Divya Manivannan <divya at multicorewareinc.com>
> ># Date 1443607299 -19800
> ># Wed Sep 30 15:31:39 2015 +0530
> ># Node ID 534d80f9272b43485cca8bb0c0a522ec8abeaa13
> ># Parent b6156a08b1def3584647f26096866c1a0c11e54a
> >distortion: change data type of distortion for 10 bit to avoid overflow
> > while adding for Y, U, V planes
> >
> >diff -r b6156a08b1de -r 534d80f9272b source/common/common.h
> >--- a/source/common/common.h Fri Oct 09 20:45:59 2015 +0530
> >+++ b/source/common/common.h Wed Sep 30 15:31:39 2015 +0530
> >@@ -141,6 +141,12 @@
> > typedef uint64_t sse_ret_t;
> > #endif
> >
> >+#if X265_DEPTH < 10
> >+typedef uint32_t dist_ret_t;
> >+#else
> >+typedef uint64_t dist_ret_t;
> >+#endif
> >+
> > #ifndef NULL
> > #define NULL 0
> > #endif
> >diff -r b6156a08b1de -r 534d80f9272b source/encoder/rdcost.h
> >--- a/source/encoder/rdcost.h Fri Oct 09 20:45:59 2015 +0530
> >+++ b/source/encoder/rdcost.h Wed Sep 30 15:31:39 2015 +0530
> >@@ -89,9 +89,9 @@
> > m_lambda = (uint64_t)floor(256.0 * lambda);
> > }
> >
> >- inline uint64_t calcRdCost(sse_ret_t distortion, uint32_t bits) const
> >+ inline uint64_t calcRdCost(dist_ret_t distortion, uint32_t bits) const
> > {
> >-#if X265_DEPTH <= 10
> >+#if X265_DEPTH < 10
> 10bpp overflow?
>
> > X265_CHECK(bits <= (UINT64_MAX - 128) / m_lambda2,
> > "calcRdCost wrap detected dist: %u, bits %u, lambda: " X265_LL "\n",
> > distortion, bits, m_lambda2);
> >@@ -116,9 +116,9 @@
> > }
>
>
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20151013/e7c9be99/attachment-0001.html>
More information about the x265-devel
mailing list