<div dir="ltr"><div>So, the general roadmap for stats collection is this:<br></div><div><br>x265 will keep all console outputs as-is, but disable cu-stats option. The cu-stats will be replaced by a more powerful per-frame CU stats in log-level 5. Now, the proposed log-levels will look like:</div><div><br>log-level 0(error) : only errors<br></div><div>log-level 1(warning) : only errors and warnings<br></div><div>log-level 2(summary) : errors, warnings, summary<br></div><div>log-level 3(info) : full x265 info, and above - default<br></div><div>log-level 4 (frame) : Basic frame-level info and above<br></div><div>log-level 5 (full) : Full frame-level info, lots of additional statistics. This option is available only if csv is also enabled. Without csv, this will default to log-level 4 outputs in console. <br><br></div><div>Comments appreciated.<br><br></div><div>Thanks,<br></div><div>Deepthi<br></div><div><br><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 28, 2015 at 8:25 PM, Steve Borho <span dir="ltr"><<a href="mailto:steve@borho.org" target="_blank">steve@borho.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 05/28, Divya Manivannan wrote:<br>
> # HG changeset patch<br>
> # User Divya Manivannan <<a href="mailto:divya@multicorewareinc.com">divya@multicorewareinc.com</a>><br>
> # Date 1432810379 -19800<br>
> # Thu May 28 16:22:59 2015 +0530<br>
> # Node ID d0b3282c8621837e76d0a09cebe22fa663eb6c29<br>
> # Parent 68d8123c5fe7dcf543d9562d1a701c7d466e0a88<br>
> stats: remove --cu-stats option<br>
><br>
> --cu-stats option collected data based on slicetype. This is removed and those data will be collected based on frame.<br>
> A warning message is displayed when --cu-stats option is given by the user.<br>
<br>
</span>nit: CTRL+E in the commit editor will reflow paragraphs to reasonable<br>
text widths<br>
<br>
Before I would queue this, I would want to see a clear road-map for how<br>
the stats collection is going to be improved, because the end-of-encode<br>
logging by slicetype is a feature and this is removing it. x264 reports<br>
this data unconditionally.<br>
<br>
I want to see a road map on where all of this is going so we can discuss<br>
the particulars before we break any backward compatibility.<br>
<br>
If we're planning to remove frame-level log messages, then we should no<br>
longer base frame-level statistics gathering on --log-level, which is<br>
only supposed to regulate console logging.<br>
<div class="HOEnZb"><div class="h5"><br>
> diff -r 68d8123c5fe7 -r d0b3282c8621 doc/reST/cli.rst<br>
> --- a/doc/reST/cli.rst Wed May 27 14:16:34 2015 +0530<br>
> +++ b/doc/reST/cli.rst Thu May 28 16:22:59 2015 +0530<br>
> @@ -124,13 +124,6 @@<br>
> is more of a problem for P frames where some blocks are much more<br>
> expensive than others.<br>
><br>
> -<br>
> -.. option:: --cu-stats, --no-cu-stats<br>
> -<br>
> - Records statistics on how each CU was coded (split depths and other<br>
> - mode decisions) and reports those statistics at the end of the<br>
> - encode. Default disabled<br>
> -<br>
> .. option:: --ssim, --no-ssim<br>
><br>
> Calculate and report Structural Similarity values. It is<br>
> diff -r 68d8123c5fe7 -r d0b3282c8621 source/CMakeLists.txt<br>
> --- a/source/CMakeLists.txt Wed May 27 14:16:34 2015 +0530<br>
> +++ b/source/CMakeLists.txt Thu May 28 16:22:59 2015 +0530<br>
> @@ -30,7 +30,7 @@<br>
> mark_as_advanced(FPROFILE_USE FPROFILE_GENERATE NATIVE_BUILD)<br>
><br>
> # X265_BUILD must be incremented each time the public API is changed<br>
> -set(X265_BUILD 60)<br>
> +set(X265_BUILD 61)<br>
> configure_file("${PROJECT_SOURCE_DIR}/<a href="http://x265.def.in" target="_blank">x265.def.in</a>"<br>
> "${PROJECT_BINARY_DIR}/x265.def")<br>
> configure_file("${PROJECT_SOURCE_DIR}/<a href="http://x265_config.h.in" target="_blank">x265_config.h.in</a>"<br>
> diff -r 68d8123c5fe7 -r d0b3282c8621 source/encoder/encoder.cpp<br>
> --- a/source/encoder/encoder.cpp Wed May 27 14:16:34 2015 +0530<br>
> +++ b/source/encoder/encoder.cpp Thu May 28 16:22:59 2015 +0530<br>
> @@ -1048,143 +1048,6 @@<br>
> #undef ELAPSED_SEC<br>
> #undef ELAPSED_MSEC<br>
> #endif<br>
> -<br>
> - if (!m_param->bLogCuStats)<br>
> - return;<br>
> -<br>
> - for (int sliceType = 2; sliceType >= 0; sliceType--)<br>
> - {<br>
> - if (sliceType == P_SLICE && !m_analyzeP.m_numPics)<br>
> - continue;<br>
> - if (sliceType == B_SLICE && !m_analyzeB.m_numPics)<br>
> - continue;<br>
> -<br>
> - StatisticLog finalLog;<br>
> - for (uint32_t depth = 0; depth <= g_maxCUDepth; depth++)<br>
> - {<br>
> - int cuSize = g_maxCUSize >> depth;<br>
> -<br>
> - for (int i = 0; i < m_param->frameNumThreads; i++)<br>
> - {<br>
> - StatisticLog& enclog = m_frameEncoder[i]->m_sliceTypeLog[sliceType];<br>
> - if (!depth)<br>
> - finalLog.totalCu += enclog.totalCu;<br>
> - finalLog.cntIntra[depth] += enclog.cntIntra[depth];<br>
> - for (int m = 0; m < INTER_MODES; m++)<br>
> - {<br>
> - if (m < INTRA_MODES)<br>
> - finalLog.cuIntraDistribution[depth][m] += enclog.cuIntraDistribution[depth][m];<br>
> - finalLog.cuInterDistribution[depth][m] += enclog.cuInterDistribution[depth][m];<br>
> - }<br>
> -<br>
> - if (cuSize == 8 && m_sps.quadtreeTULog2MinSize < 3)<br>
> - finalLog.cntIntraNxN += enclog.cntIntraNxN;<br>
> - if (sliceType != I_SLICE)<br>
> - {<br>
> - finalLog.cntTotalCu[depth] += enclog.cntTotalCu[depth];<br>
> - finalLog.cntInter[depth] += enclog.cntInter[depth];<br>
> - finalLog.cntSkipCu[depth] += enclog.cntSkipCu[depth];<br>
> - }<br>
> - }<br>
> -<br>
> - uint64_t cntInter, cntSkipCu, cntIntra = 0, cntIntraNxN = 0, encCu = 0;<br>
> - uint64_t cuInterDistribution[INTER_MODES], cuIntraDistribution[INTRA_MODES];<br>
> -<br>
> - // check for 0/0, if true assign 0 else calculate percentage<br>
> - for (int n = 0; n < INTER_MODES; n++)<br>
> - {<br>
> - if (!finalLog.cntInter[depth])<br>
> - cuInterDistribution[n] = 0;<br>
> - else<br>
> - cuInterDistribution[n] = (finalLog.cuInterDistribution[depth][n] * 100) / finalLog.cntInter[depth];<br>
> -<br>
> - if (n < INTRA_MODES)<br>
> - {<br>
> - if (!finalLog.cntIntra[depth])<br>
> - {<br>
> - cntIntraNxN = 0;<br>
> - cuIntraDistribution[n] = 0;<br>
> - }<br>
> - else<br>
> - {<br>
> - cntIntraNxN = (finalLog.cntIntraNxN * 100) / finalLog.cntIntra[depth];<br>
> - cuIntraDistribution[n] = (finalLog.cuIntraDistribution[depth][n] * 100) / finalLog.cntIntra[depth];<br>
> - }<br>
> - }<br>
> - }<br>
> -<br>
> - if (!finalLog.totalCu)<br>
> - encCu = 0;<br>
> - else if (sliceType == I_SLICE)<br>
> - {<br>
> - cntIntra = (finalLog.cntIntra[depth] * 100) / finalLog.totalCu;<br>
> - cntIntraNxN = (finalLog.cntIntraNxN * 100) / finalLog.totalCu;<br>
> - }<br>
> - else<br>
> - encCu = ((finalLog.cntIntra[depth] + finalLog.cntInter[depth]) * 100) / finalLog.totalCu;<br>
> -<br>
> - if (sliceType == I_SLICE)<br>
> - {<br>
> - cntInter = 0;<br>
> - cntSkipCu = 0;<br>
> - }<br>
> - else if (!finalLog.cntTotalCu[depth])<br>
> - {<br>
> - cntInter = 0;<br>
> - cntIntra = 0;<br>
> - cntSkipCu = 0;<br>
> - }<br>
> - else<br>
> - {<br>
> - cntInter = (finalLog.cntInter[depth] * 100) / finalLog.cntTotalCu[depth];<br>
> - cntIntra = (finalLog.cntIntra[depth] * 100) / finalLog.cntTotalCu[depth];<br>
> - cntSkipCu = (finalLog.cntSkipCu[depth] * 100) / finalLog.cntTotalCu[depth];<br>
> - }<br>
> -<br>
> - // print statistics<br>
> - char stats[256] = { 0 };<br>
> - int len = 0;<br>
> - if (sliceType != I_SLICE)<br>
> - len += sprintf(stats + len, " EncCU "X265_LL "%% Merge "X265_LL "%%", encCu, cntSkipCu);<br>
> -<br>
> - if (cntInter)<br>
> - {<br>
> - len += sprintf(stats + len, " Inter "X265_LL "%%", cntInter);<br>
> - if (m_param->bEnableAMP)<br>
> - len += sprintf(stats + len, "(%dx%d "X265_LL "%% %dx%d "X265_LL "%% %dx%d "X265_LL "%% AMP "X265_LL "%%)",<br>
> - cuSize, cuSize, cuInterDistribution[0],<br>
> - cuSize / 2, cuSize, cuInterDistribution[2],<br>
> - cuSize, cuSize / 2, cuInterDistribution[1],<br>
> - cuInterDistribution[3]);<br>
> - else if (m_param->bEnableRectInter)<br>
> - len += sprintf(stats + len, "(%dx%d "X265_LL "%% %dx%d "X265_LL "%% %dx%d "X265_LL "%%)",<br>
> - cuSize, cuSize, cuInterDistribution[0],<br>
> - cuSize / 2, cuSize, cuInterDistribution[2],<br>
> - cuSize, cuSize / 2, cuInterDistribution[1]);<br>
> - }<br>
> - if (cntIntra)<br>
> - {<br>
> - len += sprintf(stats + len, " Intra "X265_LL "%%(DC "X265_LL "%% P "X265_LL "%% Ang "X265_LL "%%",<br>
> - cntIntra, cuIntraDistribution[0],<br>
> - cuIntraDistribution[1], cuIntraDistribution[2]);<br>
> - if (sliceType != I_SLICE)<br>
> - {<br>
> - if (cuSize == 8 && m_sps.quadtreeTULog2MinSize < 3)<br>
> - len += sprintf(stats + len, " %dx%d "X265_LL "%%", cuSize / 2, cuSize / 2, cntIntraNxN);<br>
> - }<br>
> -<br>
> - len += sprintf(stats + len, ")");<br>
> - if (sliceType == I_SLICE)<br>
> - {<br>
> - if (cuSize == 8 && m_sps.quadtreeTULog2MinSize < 3)<br>
> - len += sprintf(stats + len, " %dx%d: "X265_LL "%%", cuSize / 2, cuSize / 2, cntIntraNxN);<br>
> - }<br>
> - }<br>
> - const char slicechars[] = "BPI";<br>
> - if (stats[0])<br>
> - x265_log(m_param, X265_LOG_INFO, "%c%-2d:%s\n", slicechars[sliceType], cuSize, stats);<br>
> - }<br>
> - }<br>
> }<br>
><br>
> void Encoder::fetchStats(x265_stats *stats, size_t statsSizeBytes)<br>
> @@ -1902,6 +1765,9 @@<br>
> }<br>
> else<br>
> m_param->rc.qgSize = p->maxCUSize;<br>
> +<br>
> + if (p->bLogCuStats)<br>
> + x265_log(p, X265_LOG_WARNING, "--cu-stats option is replaced by --log-level 4 option\n");<br>
> }<br>
><br>
> void Encoder::allocAnalysis(x265_analysis_data* analysis)<br>
> diff -r 68d8123c5fe7 -r d0b3282c8621 source/encoder/frameencoder.cpp<br>
> --- a/source/encoder/frameencoder.cpp Wed May 27 14:16:34 2015 +0530<br>
> +++ b/source/encoder/frameencoder.cpp Thu May 28 16:22:59 2015 +0530<br>
> @@ -904,7 +904,7 @@<br>
> // Completed CU processing<br>
> curRow.completed++;<br>
><br>
> - if (m_param->bLogCuStats || m_param->rc.bStatWrite)<br>
> + if (m_param->rc.bStatWrite)<br>
> curEncData.m_rowStat[row].sumQpAq += collectCTUStatistics(*ctu, qTreeInterCnt, qTreeIntraCnt, qTreeSkipCnt);<br>
> else if (m_param->rc.aqMode)<br>
> curEncData.m_rowStat[row].sumQpAq += calcCTUQP(*ctu);<br>
> diff -r 68d8123c5fe7 -r d0b3282c8621 source/x265cli.h<br>
> --- a/source/x265cli.h Wed May 27 14:16:34 2015 +0530<br>
> +++ b/source/x265cli.h Thu May 28 16:22:59 2015 +0530<br>
> @@ -247,7 +247,6 @@<br>
> H0("-D/--output-depth 8|10 Output bit depth (also internal bit depth). Default %d\n", param->internalBitDepth);<br>
> H0(" --log-level <string> Logging level: none error warning info debug full. Default %s\n", x265::logLevelNames[param->logLevel + 1]);<br>
> H0(" --no-progress Disable CLI progress reports\n");<br>
> - H0(" --[no-]cu-stats Enable logging stats about distribution of cu across all modes. Default %s\n",OPT(param->bLogCuStats));<br>
> H1(" --csv <filename> Comma separated log file, log level >= 3 frame log, else one line per run\n");<br>
> H0("\nInput Options:\n");<br>
> H0(" --input <filename> Raw YUV or Y4M input file name. `-` for stdin\n");<br>
> _______________________________________________<br>
> x265-devel mailing list<br>
> <a href="mailto:x265-devel@videolan.org">x265-devel@videolan.org</a><br>
> <a href="https://mailman.videolan.org/listinfo/x265-devel" target="_blank">https://mailman.videolan.org/listinfo/x265-devel</a><br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Steve Borho<br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
x265-devel mailing list<br>
<a href="mailto:x265-devel@videolan.org">x265-devel@videolan.org</a><br>
<a href="https://mailman.videolan.org/listinfo/x265-devel" target="_blank">https://mailman.videolan.org/listinfo/x265-devel</a><br>
</div></div></blockquote></div><br></div>