[x265] Enabling intra NxN partition mode when MinCbSizeY is greater than 8

[Qiwei Wen]

Dear Gurus,

As you know, the HEVC spec allows NxN partition mode for intra-coded
CUs when the CU is of size MinCbSizeY. I assume either for coding
efficiency or historical reasons, x265 won't explore PART_NxN if the
minimum CU size is not 8. I'm working on a project that involves using
x265 to prototype intra-coding algorithms and it'd be ideal for my use
case if the limitation could be lifted.

I realise that there are a few places in the code base that assume
that intra NxN implies a CU size of 8. Below is my attempt at fixing
it, the result of which is a non-local CABAC error, where the first
few intra NxN CU is coded correctly, but the CABAC state starts to
diverge to the point of making the bitstream undecodeable a few CUs
further down.

I'd appreciate any pointers on  if there are more places in the code
that makes the assumption and what the possible causes could be.

Thanks,
Qiwei

> diff --git a/source/encoder/analysis.cpp b/source/encoder/analysis.cpp
> index b219d5da4..4c42faba6 100644
> --- a/source/encoder/analysis.cpp
> +++ b/source/encoder/analysis.cpp
> @@ -580,7 +580,7 @@ uint64_t Analysis::compressIntraCU(const CUData& parentCTU, const CUGeom& cuGeom
>          checkIntra(md.pred[PRED_INTRA], cuGeom, SIZE_2Nx2N);
>          checkBestMode(md.pred[PRED_INTRA], depth);
>
> -        if (cuGeom.log2CUSize == 3 && m_slice->m_sps->quadtreeTULog2MinSize < 3)
> +        if (cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] && m_slice->m_sps->quadtreeTULog2MinSize < g_log2Size[m_param->minCUSize])
>          {
>              md.pred[PRED_INTRA_NxN].cu.initSubCU(parentCTU, cuGeom, qp);
>              checkIntra(md.pred[PRED_INTRA_NxN], cuGeom, SIZE_NxN);
> diff --git a/source/encoder/entropy.cpp b/source/encoder/entropy.cpp
> index d4bf73d53..9aecbb5e5 100644
> --- a/source/encoder/entropy.cpp
> +++ b/source/encoder/entropy.cpp
> @@ -1328,7 +1328,7 @@ void Entropy::encodeTransform(const CUData& cu, uint32_t absPartIdx, uint32_t cu
>
>      /* in each of these conditions, the subdiv flag is implied and not signaled,
>       * so we have checks to make sure the implied value matches our intentions */
> -    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> +    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
>      {
>          X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
>      }
> @@ -1463,7 +1463,7 @@ void Entropy::encodeTransformLuma(const CUData& cu, uint32_t absPartIdx, uint32_
>
>      /* in each of these conditions, the subdiv flag is implied and not signaled,
>       * so we have checks to make sure the implied value matches our intentions */
> -    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> +    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
>      {
>          X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
>      }
> diff --git a/source/encoder/search.cpp b/source/encoder/search.cpp
> index 0522f52cc..3d61caabf 100644
> --- a/source/encoder/search.cpp
> +++ b/source/encoder/search.cpp
> @@ -1541,8 +1541,8 @@ sse_t Search::estIntraPredQT(Mode &intraMode, const CUGeom& cuGeom, const uint32
>      {
>          uint32_t bmode = 0;
>
> -        if (intraMode.cu.m_lumaIntraDir[puIdx] != (uint8_t)ALL_IDX)
> -            bmode = intraMode.cu.m_lumaIntraDir[puIdx];
> +        if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
> +            bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
>          else
>          {
>              uint64_t candCostList[MAX_RD_INTRA_MODES];

-- 
Qiwei Wen
Software Developer
Exablaze Pty Ltd
✆ 0430872689
LinkedIn