[x265] [PATCH] quant: remove extra upshift by 4 in s_invQuantScales
Steve Borho
steve at borho.org
Thu Aug 14 00:36:59 CEST 2014
On 08/13, deepthi at multicorewareinc.com wrote:
> # HG changeset patch
> # User Deepthi Nandakumar <deepthi at multicorewareinc.com>
> # Date 1407919768 -19800
> # Wed Aug 13 14:19:28 2014 +0530
> # Node ID 41df534b9605806520c8ff23348e15aff793f8dc
> # Parent d43e9a6a7cced5b60284c25bd987c55c522c1212
> quant: remove extra upshift by 4 in s_invQuantScales
>
> Adding 4 to unQuantPer was a bug, since unQuantScale was already upshifted by 4.
>
> diff -r d43e9a6a7cce -r 41df534b9605 source/common/quant.cpp
> --- a/source/common/quant.cpp Wed Aug 13 00:19:31 2014 -0500
> +++ b/source/common/quant.cpp Wed Aug 13 14:19:28 2014 +0530
> @@ -516,18 +516,11 @@
> int32_t *unquantScale = m_scalingList->m_dequantCoef[log2TrSize - 2][scalingListType][rem];
> int unquantShift = QUANT_IQUANT_SHIFT - QUANT_SHIFT - transformShift;
> int unquantRound, unquantPer;
> - unquantShift += 4;
> + unquantPer = per;
> if (unquantShift > per)
> - {
> unquantRound = 1 << (unquantShift - per - 1);
> - unquantPer = per;
> - }
> else
> - {
> - unquantPer = per + 4;
> - unquantShift += 4;
> unquantRound = 0;
> - }
>
> #define SIGCOST(bits) ((lambda2 * (bits)) >> 8)
> #define RDCOST(d, bits) ((((int64_t)d * d) << scaleBits) + ((lambda2 * (bits)) >> 8))
> diff -r d43e9a6a7cce -r 41df534b9605 source/common/scalinglist.cpp
> --- a/source/common/scalinglist.cpp Wed Aug 13 00:19:31 2014 -0500
> +++ b/source/common/scalinglist.cpp Wed Aug 13 14:19:28 2014 +0530
> @@ -349,7 +349,7 @@
> for (int i = 0; i < count; i++)
> {
> quantCoeff[i] = s_quantScales[rem];
> - dequantCoeff[i] = s_invQuantScales[rem] << 4;
> + dequantCoeff[i] = s_invQuantScales[rem];
> }
> }
When the scaling list is enabled, we are upshifting s_quantScales[rem]
by 4 when generating the quant coeff list, and so this shift needs to be
accounted for in UNQUANT() just like how dequant() and dequant_scaling()
behave differently (the latter adds 4 to the right shift value but the
former does not).
This bug only survived this long because we always use the scaling list
array for quant, but use the scalar s_invQuantScales[rem] for dequant
when the scaling list is disabled.
a scalar (non-scaling list) version of quant might improve performance a
tad.
--
Steve Borho
More information about the x265-devel
mailing list