[x264-devel] [Git][videolan/x264][master] Fix integer overflow in psy-rd

[Anton Mitrofanov (@BugMaster)]

Anton Mitrofanov pushed to branch master at VideoLAN / x264


Commits:
6c142bba by Vitaly Lysenkov at 2021-09-29T21:21:02+00:00
Fix integer overflow in psy-rd

- - - - -


1 changed file:

- encoder/rdo.c


Changes:

=====================================
encoder/rdo.c
=====================================
@@ -141,7 +141,8 @@ static inline int ssd_plane( x264_t *h, int size, int p, int x, int y )
             int dc = h->pixf.sad[size]( fdec, FDEC_STRIDE, (pixel*)x264_zero, 0 ) >> 1;
             satd = abs(h->pixf.satd[size]( fdec, FDEC_STRIDE, (pixel*)x264_zero, 0 ) - dc - cached_satd( h, size, x, y ));
         }
-        satd = (satd * h->mb.i_psy_rd * h->mb.i_psy_rd_lambda + 128) >> 8;
+        int64_t tmp = ((int64_t)satd * h->mb.i_psy_rd * h->mb.i_psy_rd_lambda + 128) >> 8;
+        satd = X264_MIN( tmp, COST_MAX );
     }
     return h->pixf.ssd[size](fenc, FENC_STRIDE, fdec, FDEC_STRIDE) + satd;
 }



View it on GitLab: https://code.videolan.org/videolan/x264/-/commit/6c142bba65bd864bab4c8d8593c510ce29d6d6d9

-- 
View it on GitLab: https://code.videolan.org/videolan/x264/-/commit/6c142bba65bd864bab4c8d8593c510ce29d6d6d9
You're receiving this email because of your account on code.videolan.org.