[x265] [PATCH] fix PSYVALUE shift overflow, Issue #180 [OUTPUT CHANGE on 12bpp]

[Steve Borho]

On 09/11, Min Chen wrote:
> # HG changeset patch
> # User Min Chen <chenm003 at 163.com>
> # Date 1442002697 18000
> # Node ID f520fd29f3d71d495e08fe96df917489348c377b
> # Parent  137854992fc614bdd6c446852528e24ed52c9991
> fix PSYVALUE shift overflow, Issue #180 [OUTPUT CHANGE on 12bpp]
> ---
>  source/common/quant.cpp |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff -r 137854992fc6 -r f520fd29f3d7 source/common/quant.cpp
> --- a/source/common/quant.cpp	Fri Sep 11 15:18:14 2015 -0500
> +++ b/source/common/quant.cpp	Fri Sep 11 15:18:17 2015 -0500
> @@ -588,7 +588,7 @@
>  #define UNQUANT(lvl)    (((lvl) * (unquantScale[blkPos] << per) + unquantRound) >> unquantShift)
>  #define SIGCOST(bits)   ((lambda2 * (bits)) >> 8)
>  #define RDCOST(d, bits) ((((int64_t)d * d) << scaleBits) + SIGCOST(bits))
> -#define PSYVALUE(rec)   ((psyScale * (rec)) >> (2 * transformShift + 1))
> +#define PSYVALUE(rec)   ((psyScale * (rec)) >> X265_MAX(0, (2 * transformShift + 1)))

this prevents a negative value, but it seems like we want really large
psy values to be clamped to a large positive value instead of 0.

>      int64_t costCoeff[trSize * trSize];   /* d*d + lambda * bits */
>      int64_t costUncoded[trSize * trSize]; /* d*d + lambda * 0    */
> 
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel

-- 
Steve Borho