[x265] [PATCH] weightanalysis: fix stack corruption due to fabsf()

[Steve Borho]

On Mon, Mar 17, 2014 at 2:18 AM,  <kavitha at multicorewareinc.com> wrote:
> # HG changeset patch
> # User Kavitha Sampath <kavitha at multicorewareinc.com>
> # Date 1395040372 -19800
> #      Mon Mar 17 12:42:52 2014 +0530
> # Node ID 99551efb1b89f6dbb00761d4003ac5927fc760b7
> # Parent  8d5deb7cafd83ac554489deb8577f905a0cda6b3
> weightanalysis: fix stack corruption due to fabsf()
>
> diff -r 8d5deb7cafd8 -r 99551efb1b89 source/encoder/slicetype.cpp
> --- a/source/encoder/slicetype.cpp      Mon Mar 17 00:47:24 2014 -0500
> +++ b/source/encoder/slicetype.cpp      Mon Mar 17 12:42:52 2014 +0530
> @@ -1303,7 +1303,9 @@
>      refMean  = (float)ref->wp_sum[0] / (fenc->lines * fenc->width) / (1 << (X265_DEPTH - 8));
>
>      /* Early termination */
> -    if (fabsf(refMean - fencMean) < 0.5f && fabsf(1.f - guessScale) < epsilon)
> +    float meanDiff = refMean < fencMean ? fencMean - refMean : refMean - fencMean;
> +    float guessVal = guessScale > 1.f ? guessScale - 1.f : 1.f - guessScale;
> +    if (meanDiff < 0.5f && guessVal < epsilon)

this is seriously ugly. I think it would be better to understand why
fabsf() seems to experience problems on Win32 Debug builds and fix the
root cause.  This is just papering over a deeper issue.

>          return;
>
>      int minoff = 0, minscale, mindenom;
> diff -r 8d5deb7cafd8 -r 99551efb1b89 source/encoder/weightPrediction.cpp
> --- a/source/encoder/weightPrediction.cpp       Mon Mar 17 00:47:24 2014 -0500
> +++ b/source/encoder/weightPrediction.cpp       Mon Mar 17 12:42:52 2014 +0530
> @@ -298,7 +298,9 @@
>
>              /* Early termination */
>              x265_emms();
> -            if (fabsf(refMean[plane] - fencMean[plane]) < 0.5f && fabsf(1.f - guessScale[plane]) < epsilon)
> +            float meanDiff = refMean[plane] < fencMean[plane] ? fencMean[plane] - refMean[plane] : refMean[plane] - fencMean[plane];
> +            float guessVal = guessScale[plane] > 1.f ? guessScale[plane] - 1.f : 1.f - guessScale[plane];
> +            if (meanDiff < 0.5f && guessVal < epsilon)
>              {
>                  SET_WEIGHT(weights[plane], 0, 1, 0, 0);
>                  continue;
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel



-- 
Steve Borho