[x265] Enabling intra NxN partition mode when MinCbSizeY isgreater than 8

[Richard]

The problem is this 
if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
            bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
Should not be changed like this.
This if branch is for analysis-load mode, in a normal "first time" prediction this should always go to the else branch.



Original
From: Qiwei Wen <wenqiweiabcd at gmail.com>
Date: 2025-12-19 05:53
To: chen <chenm003 at 163.com>
Cc: Development for x265 <x265-devel at videolan.org>
Subject: Re: [x265] Enabling intra NxN partition mode when MinCbSizeY isgreater than 8


Thanks Chen, will apply these suggestions as soon as I get past the cabac issues. As I mentioned, the patch is missing some fixes and results in CABAC errors whenever intra NxN is chosen in case that MinCbSizeY != 8.


Any suggestions on where else to look is greatly appreciated


Qiwei Wen
Software Developer
Exablaze Pty Ltd
✆ 0430872689
LinkedIn







On Fri, Dec 19, 2025, 2:58 AM chen <chenm003 at 163.com> wrote:
Hi Wen,


Thank for the patch.


I have some comments,


(1) access global variant is higher cost, how about change below?
cuGeom.log2CUSize == g_log2Size[m_param->minCUSize]
(1 << cuGeom.log2CUSize) == m_param->minCUSize


(2) looks mixed another fix on array index typo, suggest separate patch
m_lumaIntraDir[absPartIdx]


Regards,
Chen



At 2025-12-18 21:26:30, "Qiwei Wen" <wenqiweiabcd at gmail.com> wrote:
>Dear Gurus,
>
>As you know, the HEVC spec allows NxN partition mode for intra-coded
>CUs when the CU is of size MinCbSizeY. I assume either for coding
>efficiency or historical reasons, x265 won't explore PART_NxN if the
>minimum CU size is not 8. I'm working on a project that involves using
>x265 to prototype intra-coding algorithms and it'd be ideal for my use
>case if the limitation could be lifted.
>
>I realise that there are a few places in the code base that assume
>that intra NxN implies a CU size of 8. Below is my attempt at fixing
>it, the result of which is a non-local CABAC error, where the first
>few intra NxN CU is coded correctly, but the CABAC state starts to
>diverge to the point of making the bitstream undecodeable a few CUs
>further down.
>
>I'd appreciate any pointers on  if there are more places in the code
>that makes the assumption and what the possible causes could be.
>
>Thanks,
>Qiwei
>
>> diff --git a/source/encoder/analysis.cpp b/source/encoder/analysis.cpp
>> index b219d5da4..4c42faba6 100644
>> --- a/source/encoder/analysis.cpp
>> +++ b/source/encoder/analysis.cpp
>> @@ -580,7 +580,7 @@ uint64_t Analysis::compressIntraCU(const CUData& parentCTU, const CUGeom& cuGeom
>>          checkIntra(md.pred[PRED_INTRA], cuGeom, SIZE_2Nx2N);
>>          checkBestMode(md.pred[PRED_INTRA], depth);
>>
>> -        if (cuGeom.log2CUSize == 3 && m_slice->m_sps->quadtreeTULog2MinSize < 3)
>> +        if (cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] && m_slice->m_sps->quadtreeTULog2MinSize < g_log2Size[m_param->minCUSize])
>>          {
>>              md.pred[PRED_INTRA_NxN].cu.initSubCU(parentCTU, cuGeom, qp);
>>              checkIntra(md.pred[PRED_INTRA_NxN], cuGeom, SIZE_NxN);
>> diff --git a/source/encoder/entropy.cpp b/source/encoder/entropy.cpp
>> index d4bf73d53..9aecbb5e5 100644
>> --- a/source/encoder/entropy.cpp
>> +++ b/source/encoder/entropy.cpp
>> @@ -1328,7 +1328,7 @@ void Entropy::encodeTransform(const CUData& cu, uint32_t absPartIdx, uint32_t cu
>>
>>      /* in each of these conditions, the subdiv flag is implied and not signaled,
>>       * so we have checks to make sure the implied value matches our intentions */
>> -    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
>> +    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
>>      {
>>          X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
>>      }
>> @@ -1463,7 +1463,7 @@ void Entropy::encodeTransformLuma(const CUData& cu, uint32_t absPartIdx, uint32_
>>
>>      /* in each of these conditions, the subdiv flag is implied and not signaled,
>>       * so we have checks to make sure the implied value matches our intentions */
>> -    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
>> +    if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
>>      {
>>          X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
>>      }
>> diff --git a/source/encoder/search.cpp b/source/encoder/search.cpp
>> index 0522f52cc..3d61caabf 100644
>> --- a/source/encoder/search.cpp
>> +++ b/source/encoder/search.cpp
>> @@ -1541,8 +1541,8 @@ sse_t Search::estIntraPredQT(Mode &intraMode, const CUGeom& cuGeom, const uint32
>>      {
>>          uint32_t bmode = 0;
>>
>> -        if (intraMode.cu.m_lumaIntraDir[puIdx] != (uint8_t)ALL_IDX)
>> -            bmode = intraMode.cu.m_lumaIntraDir[puIdx];
>> +        if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
>> +            bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
>>          else
>>          {
>>              uint64_t candCostList[MAX_RD_INTRA_MODES];
>
>-- 
>Qiwei Wen
>Software Developer
>Exablaze Pty Ltd
>✆ 0430872689
>LinkedIn
>_______________________________________________
>x265-devel mailing list
>x265-devel at videolan.org
>https://mailman.videolan.org/listinfo/x265-devel