[x265] [PATCH 2 of 2] rc: tune initial predictor values for better frame size predictions in vbv lookahead

[aarthi at multicorewareinc.com]

# HG changeset patch
# User Aarthi Thirumalai
# Date 1426487738 -19800
#      Mon Mar 16 12:05:38 2015 +0530
# Node ID b4646bdaf2202748d3a3b159b9bc8e3cfcd272de
# Parent  6f159ec858f4eec484d722916c41911e3e4540f9
rc: tune initial predictor values for better frame size predictions in vbv lookahead

an overall improvement in ssim of around .05-.1 db can be seen.
Also, improves visual quality at the start of the encode.

SteamLocomotiveTrain_2560x1600_60_10bit_crop.yuv --bitrate 9000 --vbv-bufsize 9000 --strict-cbr

       Bitrate	 Y PSNR	 U PSNR	 V PSNR	 Global PSNR	 SSIM	 SSIM (dB)
before  9168.43 39.689  45.096 45.152	  41.048	 0.93176  11.66
after   9106.16  39.864 45.169	45.199	  41.194	 0.93518  11.883

diff -r 6f159ec858f4 -r b4646bdaf220 source/encoder/ratecontrol.cpp
--- a/source/encoder/ratecontrol.cpp	Tue Mar 31 22:16:21 2015 +0530
+++ b/source/encoder/ratecontrol.cpp	Mon Mar 16 12:05:38 2015 +0530
@@ -372,12 +372,17 @@
     /* Frame Predictors and Row predictors used in vbv */
     for (int i = 0; i < 4; i++)
     {
-        m_pred[i].coeff = 1.5;
+        m_pred[i].coeff = 1.0;
         m_pred[i].count = 1.0;
         m_pred[i].decay = 0.5;
         m_pred[i].offset = 0.0;
     }
-    m_pred[0].coeff = 1.0;
+    m_pred[0].coeff = m_pred[3].coeff = 0.75;
+    if (m_param->rc.qCompress >= 0.8) // when tuned for grain 
+    {
+        m_pred[1].coeff = 0.75;
+        m_pred[0].coeff = m_pred[3].coeff = 0.50;
+    }
     if (!m_statFileOut && (m_param->rc.bStatWrite || m_param->rc.bStatRead))
     {
         /* If the user hasn't defined the stat filename, use the default value */
@@ -1797,8 +1802,11 @@
         double pbits = predictSize(&m_pred[m_predType], q, (double)m_currentSatd);
         if (pbits > rce->frameSizeMaximum)
             q *= pbits / rce->frameSizeMaximum;
-
-        if (!m_isCbr || (m_isAbr && m_currentSatd >= rce->movingAvgSum && q <= q0 / 2))
+        /* To detect frames that are more complex in SATD costs compared to prev window, yet 
+         * lookahead vbv reduces its qscale by half its value. Be on safer side and avoid drastic 
+         * qscale reductions for frames high in complexity */
+        bool mispredCheck = rce->movingAvgSum && m_currentSatd >= rce->movingAvgSum && q <= q0 / 2;
+        if (!m_isCbr || (m_isAbr && mispredCheck))
             q = X265_MAX(q0, q);
 
         if (m_rateFactorMaxIncrement)