[x265] [PATCH] constants: adjust lambda tabels for 10bit encodes (fixes #55)

[Steve Borho]

On 12/03, Steve Borho wrote:
> # HG changeset patch
> # User Satoshi Nakagawa <nakagawa424 at oki.com>

Apologies to Satoshi, this is actually my patch. I accidently left a
username override in effect from when I split an earlier patch of his.

> # Date 1417624463 21600
> #      Wed Dec 03 10:34:23 2014 -0600
> # Node ID 37805c57c255d821a7521035b8b43e1f4b2b7edd
> # Parent  50d2b92ecc89bf1de974b2919ca7d0687f0f80a1
> constants: adjust lambda tabels for 10bit encodes (fixes #55)
> 
> Since samples are 10bits, where two bits of extra resolution has been added to
> add more granularity, distortion also has two extra bits. A typical resolution
> for this problem is to down-shift distortion by 2 bits everywhere, before
> adding lambda * bits to calculate RD cost. Instead, we multiply lambda by 4
> (essentially shift it up by two bits) so distortion and lambda * bits are both
> at the higher scale.
> 
> lambda2 uses the square of the up-shifted lambda, so it has the doubled up-shift
> same as the squared distortion values used for RDO.
> 
> Example output change:
> ./x265 /Volumes/video/sintel_trailer_2k_480p24.y4m o.bin --ssim --no-info
> 
> Main:          195.67 kb/s, SSIM Mean Y: 0.9833338 (17.782 dB)
> Main10 before: 363.49 kb/s, SSIM Mean Y: 0.9888182 (19.515 dB)
> Main10 after:  206.54 kb/s, SSIM Mean Y: 0.9855121 (18.390 dB)
> 
> diff -r 50d2b92ecc89 -r 37805c57c255 source/common/constants.cpp
> --- a/source/common/constants.cpp	Wed Dec 03 11:32:33 2014 +0530
> +++ b/source/common/constants.cpp	Wed Dec 03 10:34:23 2014 -0600
> @@ -27,6 +27,47 @@
>  
>  namespace x265 {
>  
> +#if HIGH_BIT_DEPTH
> +// lambda = pow(2, (double)q / 6 - 2) * (1 << (X265_DEPTH - 8));
> +double x265_lambda_tab[QP_MAX_MAX + 1] =
> +{
> +    1.0000, 1.1225, 1.2599, 1.4142, 1.5874, 
> +    1.7818, 2.0000, 2.2449, 2.5198, 2.8284, 
> +    3.1748, 3.5636, 4.0000, 4.4898, 5.0397, 
> +    5.6569, 6.3496, 7.1272, 8.0000, 8.9797, 
> +    10.0794, 11.3137, 12.6992, 14.2544, 16.0000, 
> +    17.9594, 20.1587, 22.6274, 25.3984, 28.5088, 
> +    32.0000, 35.9188, 40.3175, 45.2548, 50.7968, 
> +    57.0175, 64.0000, 71.8376, 80.6349, 90.5097, 
> +    101.5937, 114.0350, 128.0000, 143.6751, 161.2699, 
> +    181.0193, 203.1873, 228.0701, 256.0000, 287.3503, 
> +    322.5398, 362.0387, 406.3747, 456.1401, 512.0000, 
> +    574.7006, 645.0796, 724.0773, 812.7493, 912.2803, 
> +    1024.0000, 1149.4011, 1290.1592, 1448.1547, 1625.4987, 
> +    1824.5606, 2048.0000, 2298.8023, 2580.3183, 2896.3094,
> +};
> +
> +// lambda2 = pow(lambda, 2) * scale (0.85);
> +double x265_lambda2_tab[QP_MAX_MAX + 1] =
> +{
> +    0.8500, 1.0709, 1.3493, 1.7000, 2.1419, 
> +    2.6986, 3.4000, 4.2837, 5.3972, 6.8000, 
> +    8.5675, 10.7943, 13.6000, 17.1349, 21.5887, 
> +    27.2000, 34.2699, 43.1773, 54.4000, 68.5397, 
> +    86.3546, 108.8000, 137.0794, 172.7092, 217.6000, 
> +    274.1588, 345.4185, 435.2000, 548.3176, 690.8369, 
> +    870.4000, 1096.6353, 1381.6739, 1740.8000, 2193.2706, 
> +    2763.3478, 3481.6000, 4386.5411, 5526.6955, 6963.2000, 
> +    8773.0823, 11053.3910, 13926.4000, 17546.1645, 22106.7820, 
> +    27852.8000, 35092.3290, 44213.5640, 55705.6000, 70184.6580, 
> +    88427.1280, 111411.2000, 140369.3161, 176854.2561, 222822.4000, 
> +    280738.6321, 353708.5122, 445644.8000, 561477.2643, 707417.0243, 
> +    891289.6000, 1122954.5286, 1414834.0486, 1782579.2000, 2245909.0572, 
> +    2829668.0973, 3565158.4000, 4491818.1144, 5659336.1946, 7130316.8000, 
> +};
> +
> +#else /* !HIGH_BIT_DEPTH */
> +
>  // lambda = pow(2, (double)q / 6 - 2);
>  double x265_lambda_tab[QP_MAX_MAX + 1] =
>  {
> @@ -65,6 +106,8 @@
>      176854.2222, 222822.4000, 280738.6627, 353708.5368, 445644.7459
>  };
>  
> +#endif
> +
>  const uint16_t x265_chroma_lambda2_offset_tab[MAX_CHROMA_LAMBDA_OFFSET+1] =
>  {
>         16,    20,    25,    32,    40,    50,

-- 
Steve Borho