[x265] [PATCH] quant: remove extra upshift by 4 in s_invQuantScales

[Steve Borho]

On 08/13, deepthi at multicorewareinc.com wrote:
> # HG changeset patch
> # User Deepthi Nandakumar <deepthi at multicorewareinc.com>
> # Date 1407919768 -19800
> #      Wed Aug 13 14:19:28 2014 +0530
> # Node ID 41df534b9605806520c8ff23348e15aff793f8dc
> # Parent  d43e9a6a7cced5b60284c25bd987c55c522c1212
> quant: remove extra upshift by 4 in s_invQuantScales
> 
> Adding 4 to unQuantPer was a bug, since unQuantScale was already upshifted by 4.
> 
> diff -r d43e9a6a7cce -r 41df534b9605 source/common/quant.cpp
> --- a/source/common/quant.cpp	Wed Aug 13 00:19:31 2014 -0500
> +++ b/source/common/quant.cpp	Wed Aug 13 14:19:28 2014 +0530
> @@ -516,18 +516,11 @@
>      int32_t *unquantScale = m_scalingList->m_dequantCoef[log2TrSize - 2][scalingListType][rem];
>      int unquantShift = QUANT_IQUANT_SHIFT - QUANT_SHIFT - transformShift;
>      int unquantRound, unquantPer;
> -    unquantShift += 4;
> +    unquantPer = per;
>      if (unquantShift > per)
> -    {
>          unquantRound = 1 << (unquantShift - per - 1);
> -        unquantPer = per;
> -    }
>      else
> -    {
> -        unquantPer = per + 4;
> -        unquantShift += 4;
>          unquantRound = 0;
> -    }
>  
>  #define SIGCOST(bits)   ((lambda2 * (bits)) >> 8)
>  #define RDCOST(d, bits) ((((int64_t)d * d) << scaleBits) + ((lambda2 * (bits)) >> 8))
> diff -r d43e9a6a7cce -r 41df534b9605 source/common/scalinglist.cpp
> --- a/source/common/scalinglist.cpp	Wed Aug 13 00:19:31 2014 -0500
> +++ b/source/common/scalinglist.cpp	Wed Aug 13 14:19:28 2014 +0530
> @@ -349,7 +349,7 @@
>                      for (int i = 0; i < count; i++)
>                      {
>                          quantCoeff[i] = s_quantScales[rem];
> -                        dequantCoeff[i] = s_invQuantScales[rem] << 4;
> +                        dequantCoeff[i] = s_invQuantScales[rem];
>                      }
>                  }

When the scaling list is enabled, we are upshifting s_quantScales[rem]
by 4 when generating the quant coeff list, and so this shift needs to be
accounted for in UNQUANT() just like how dequant() and dequant_scaling()
behave differently (the latter adds 4 to the right shift value but the
former does not).

This bug only survived this long because we always use the scaling list
array for quant, but use the scalar s_invQuantScales[rem] for dequant
when the scaling list is disabled.

a scalar (non-scaling list) version of quant might improve performance a
tad.

-- 
Steve Borho