[x265] [PATCH] limitTU: fix energy calculation used in limiting TU recursion

[Pradeep Ramachandran]

On Fri, Oct 14, 2016 at 7:20 PM, <bhavna at multicorewareinc.com> wrote:

> # HG changeset patch
> # User Bhavna Hariharan <bhavna at multicorewareinc.com>
> # Date 1476275329 -19800
> #      Wed Oct 12 17:58:49 2016 +0530
> # Node ID 854149baceefa075c3b1af12433680ffda2e3b64
> # Parent  c97805dad9148ad3cddba10a67ed5596508e8f86
> limitTU: fix energy calculation used in limiting TU recursion
>
> This commit changes the output of limit TU
>
> diff -r c97805dad914 -r 854149baceef source/encoder/search.cpp
> --- a/source/encoder/search.cpp Thu Oct 13 17:53:48 2016 +0800
> +++ b/source/encoder/search.cpp Wed Oct 12 17:58:49 2016 +0530
> @@ -3420,14 +3420,15 @@
>          if (m_param->limitTU && bCheckSplit)
>          {
>              // Stop recursion if the TU's energy level is minimal
> +            uint32_t numCoeff = trSize * trSize;
>              if (cbfFlag[TEXT_LUMA][0] == 0)
>                  bCheckSplit = false;
> -            else if (numSig[TEXT_LUMA][0] < (cuGeom.numPartitions / 16))
> +            else if (numSig[TEXT_LUMA][0] < (numCoeff / 64))
>              {
>                  uint32_t energy = 0;
> -                for (uint32_t i = 0; i < cuGeom.numPartitions; i++)
> +                for (uint32_t i = 0; i < numCoeff; i++)
>                      energy += abs(coeffCurY[i]);
> -                if (energy < numSig[TEXT_LUMA][0])
> +                if (energy == numSig[TEXT_LUMA][0])
>                      bCheckSplit = false;
>

Can you give an example where CheckSplit is disabled here? I am finding it
hard to reason conditions under which this condition is satisfied.


>              }
>          }
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20161017/2cded68f/attachment.html>