[x264-devel] Fix PCM cost overflow

[Anton Mitrofanov]

x264 | branch: master | Anton Mitrofanov <BugMaster at narod.ru> | Fri Dec 16 01:48:07 2011 +0400| [4c08e42504af81cdbe5789a309e868ca8eda2c1f] | committer: Jason Garrett-Glaser

Fix PCM cost overflow

> http://git.videolan.org/gitweb.cgi/x264.git/?a=commit;h=4c08e42504af81cdbe5789a309e868ca8eda2c1f
---

 encoder/analyse.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/encoder/analyse.c b/encoder/analyse.c
index 83d8b5d..7045d09 100644
--- a/encoder/analyse.c
+++ b/encoder/analyse.c
@@ -433,8 +433,10 @@ static void x264_mb_analyse_init( x264_t *h, x264_mb_analysis_t *a, int qp )
     a->i_satd_i4x4   =
     a->i_satd_chroma = COST_MAX;
 
-    /* non-RD PCM decision is inaccurate (as is psy-rd), so don't do it */
-    a->i_satd_pcm = !h->mb.i_psy_rd && a->i_mbrd ? ((uint64_t)X264_PCM_COST*a->i_lambda2 + 128) >> 8 : COST_MAX;
+    /* non-RD PCM decision is inaccurate (as is psy-rd), so don't do it.
+     * PCM cost can overflow with high lambda2, so cap it at COST_MAX. */
+    uint64_t pcm_cost = ((uint64_t)X264_PCM_COST*a->i_lambda2 + 128) >> 8;
+    a->i_satd_pcm = !h->mb.i_psy_rd && a->i_mbrd && pcm_cost < COST_MAX ? pcm_cost : COST_MAX;
 
     a->b_fast_intra = 0;
     a->b_avoid_topright = 0;