[x265] Enabling intra NxN partition mode when MinCbSizeY is greater than 8

[Qiwei Wen]

Thanks Chen, will apply these suggestions as soon as I get past the cabac
issues. As I mentioned, the patch is missing some fixes and results in
CABAC errors whenever intra NxN is chosen in case that MinCbSizeY != 8.

Any suggestions on where else to look is greatly appreciated

*Qiwei Wen*
*Software Developer*
*Exablaze Pty Ltd*
✆ 0430872689
LinkedIn <https://www.linkedin.com/in/qiwei-wen-9905578a/>




On Fri, Dec 19, 2025, 2:58 AM chen <chenm003 at 163.com> wrote:

> Hi Wen,
>
> Thank for the patch.
>
> I have some comments,
>
> (1) access global variant is higher cost, how about change below?
> cuGeom.log2CUSize == g_log2Size[m_param->minCUSize]
> (1 << cuGeom.log2CUSize) == m_param->minCUSize
>
> (2) looks mixed another fix on array index typo, suggest separate patch
> m_lumaIntraDir[absPartIdx]
>
> Regards,
> Chen
>
>
> At 2025-12-18 21:26:30, "Qiwei Wen" <wenqiweiabcd at gmail.com> wrote:
> >Dear Gurus,
> >
> >As you know, the HEVC spec allows NxN partition mode for intra-coded
> >CUs when the CU is of size MinCbSizeY. I assume either for coding
> >efficiency or historical reasons, x265 won't explore PART_NxN if the
> >minimum CU size is not 8. I'm working on a project that involves using
> >x265 to prototype intra-coding algorithms and it'd be ideal for my use
> >case if the limitation could be lifted.
> >
> >I realise that there are a few places in the code base that assume
> >that intra NxN implies a CU size of 8. Below is my attempt at fixing
> >it, the result of which is a non-local CABAC error, where the first
> >few intra NxN CU is coded correctly, but the CABAC state starts to
> >diverge to the point of making the bitstream undecodeable a few CUs
> >further down.
> >
> >I'd appreciate any pointers on  if there are more places in the code
> >that makes the assumption and what the possible causes could be.
> >
> >Thanks,
> >Qiwei
> >
> >> diff --git a/source/encoder/analysis.cpp b/source/encoder/analysis.cpp
> >> index b219d5da4..4c42faba6 100644
> >> --- a/source/encoder/analysis.cpp
> >> +++ b/source/encoder/analysis.cpp
> >> @@ -580,7 +580,7 @@ uint64_t Analysis::compressIntraCU(const CUData& parentCTU, const CUGeom& cuGeom
> >>          checkIntra(md.pred[PRED_INTRA], cuGeom, SIZE_2Nx2N);
> >>          checkBestMode(md.pred[PRED_INTRA], depth);
> >>
> >> -        if (cuGeom.log2CUSize == 3 && m_slice->m_sps->quadtreeTULog2MinSize < 3)
> >> +        if (cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] && m_slice->m_sps->quadtreeTULog2MinSize < g_log2Size[m_param->minCUSize])
> >>          {
> >>              md.pred[PRED_INTRA_NxN].cu.initSubCU(parentCTU, cuGeom, qp);
> >>              checkIntra(md.pred[PRED_INTRA_NxN], cuGeom, SIZE_NxN);
> >> diff --git a/source/encoder/entropy.cpp b/source/encoder/entropy.cpp
> >> index d4bf73d53..9aecbb5e5 100644
> >> --- a/source/encoder/entropy.cpp
> >> +++ b/source/encoder/entropy.cpp
> >> @@ -1328,7 +1328,7 @@ void Entropy::encodeTransform(const CUData& cu, uint32_t absPartIdx, uint32_t cu
> >>
> >>      /* in each of these conditions, the subdiv flag is implied and not signaled,
> >>       * so we have checks to make sure the implied value matches our intentions */
> >> -    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> >> +    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
> >>      {
> >>          X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
> >>      }
> >> @@ -1463,7 +1463,7 @@ void Entropy::encodeTransformLuma(const CUData& cu, uint32_t absPartIdx, uint32_
> >>
> >>      /* in each of these conditions, the subdiv flag is implied and not signaled,
> >>       * so we have checks to make sure the implied value matches our intentions */
> >> -    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> >> +    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
> >>      {
> >>          X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
> >>      }
> >> diff --git a/source/encoder/search.cpp b/source/encoder/search.cpp
> >> index 0522f52cc..3d61caabf 100644
> >> --- a/source/encoder/search.cpp
> >> +++ b/source/encoder/search.cpp
> >> @@ -1541,8 +1541,8 @@ sse_t Search::estIntraPredQT(Mode &intraMode, const CUGeom& cuGeom, const uint32
> >>      {
> >>          uint32_t bmode = 0;
> >>
> >> -        if (intraMode.cu.m_lumaIntraDir[puIdx] != (uint8_t)ALL_IDX)
> >> -            bmode = intraMode.cu.m_lumaIntraDir[puIdx];
> >> +        if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
> >> +            bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
> >>          else
> >>          {
> >>              uint64_t candCostList[MAX_RD_INTRA_MODES];
> >
> >--
> >Qiwei Wen
> >Software Developer
> >Exablaze Pty Ltd
> >✆ 0430872689
> >LinkedIn
> >_______________________________________________
> >x265-devel mailing list
> >x265-devel at videolan.org
> >https://mailman.videolan.org/listinfo/x265-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20251219/bb402654/attachment.htm>