[x265] Enabling intra NxN partition mode when MinCbSizeY isgreater than 8
Qiwei Wen
wenqiweiabcd at gmail.com
Fri Dec 19 09:11:39 UTC 2025
I meant 4x4 partitions, obviously, sorry for the typo!
On Fri, 19 Dec 2025 at 19:33, Richard <ccc7922 at foxmail.com> wrote:
>
> The problem is this
> if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
> bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
> Should not be changed like this.
> This if branch is for analysis-load mode, in a normal "first time" prediction this should always go to the else branch.
>
>
>
> Original
> From: Qiwei Wen <wenqiweiabcd at gmail.com>
> Date: 2025-12-19 05:53
> To: chen <chenm003 at 163.com>
> Cc: Development for x265 <x265-devel at videolan.org>
> Subject: Re: [x265] Enabling intra NxN partition mode when MinCbSizeY isgreater than 8
>
>
> Thanks Chen, will apply these suggestions as soon as I get past the cabac issues. As I mentioned, the patch is missing some fixes and results in CABAC errors whenever intra NxN is chosen in case that MinCbSizeY != 8.
>
>
> Any suggestions on where else to look is greatly appreciated
>
>
> Qiwei Wen
> Software Developer
> Exablaze Pty Ltd
> ✆ 0430872689
> LinkedIn
>
>
>
>
>
>
>
> On Fri, Dec 19, 2025, 2:58 AM chen <chenm003 at 163.com> wrote:
> Hi Wen,
>
>
> Thank for the patch.
>
>
> I have some comments,
>
>
> (1) access global variant is higher cost, how about change below?
> cuGeom.log2CUSize == g_log2Size[m_param->minCUSize]
> (1 << cuGeom.log2CUSize) == m_param->minCUSize
>
>
> (2) looks mixed another fix on array index typo, suggest separate patch
> m_lumaIntraDir[absPartIdx]
>
>
> Regards,
> Chen
>
>
>
> At 2025-12-18 21:26:30, "Qiwei Wen" <wenqiweiabcd at gmail.com> wrote:
> >Dear Gurus,
> >
> >As you know, the HEVC spec allows NxN partition mode for intra-coded
> >CUs when the CU is of size MinCbSizeY. I assume either for coding
> >efficiency or historical reasons, x265 won't explore PART_NxN if the
> >minimum CU size is not 8. I'm working on a project that involves using
> >x265 to prototype intra-coding algorithms and it'd be ideal for my use
> >case if the limitation could be lifted.
> >
> >I realise that there are a few places in the code base that assume
> >that intra NxN implies a CU size of 8. Below is my attempt at fixing
> >it, the result of which is a non-local CABAC error, where the first
> >few intra NxN CU is coded correctly, but the CABAC state starts to
> >diverge to the point of making the bitstream undecodeable a few CUs
> >further down.
> >
> >I'd appreciate any pointers on if there are more places in the code
> >that makes the assumption and what the possible causes could be.
> >
> >Thanks,
> >Qiwei
> >
> >> diff --git a/source/encoder/analysis.cpp b/source/encoder/analysis.cpp
> >> index b219d5da4..4c42faba6 100644
> >> --- a/source/encoder/analysis.cpp
> >> +++ b/source/encoder/analysis.cpp
> >> @@ -580,7 +580,7 @@ uint64_t Analysis::compressIntraCU(const CUData& parentCTU, const CUGeom& cuGeom
> >> checkIntra(md.pred[PRED_INTRA], cuGeom, SIZE_2Nx2N);
> >> checkBestMode(md.pred[PRED_INTRA], depth);
> >>
> >> - if (cuGeom.log2CUSize == 3 && m_slice->m_sps->quadtreeTULog2MinSize < 3)
> >> + if (cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] && m_slice->m_sps->quadtreeTULog2MinSize < g_log2Size[m_param->minCUSize])
> >> {
> >> md.pred[PRED_INTRA_NxN].cu.initSubCU(parentCTU, cuGeom, qp);
> >> checkIntra(md.pred[PRED_INTRA_NxN], cuGeom, SIZE_NxN);
> >> diff --git a/source/encoder/entropy.cpp b/source/encoder/entropy.cpp
> >> index d4bf73d53..9aecbb5e5 100644
> >> --- a/source/encoder/entropy.cpp
> >> +++ b/source/encoder/entropy.cpp
> >> @@ -1328,7 +1328,7 @@ void Entropy::encodeTransform(const CUData& cu, uint32_t absPartIdx, uint32_t cu
> >>
> >> /* in each of these conditions, the subdiv flag is implied and not signaled,
> >> * so we have checks to make sure the implied value matches our intentions */
> >> - if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> >> + if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
> >> {
> >> X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
> >> }
> >> @@ -1463,7 +1463,7 @@ void Entropy::encodeTransformLuma(const CUData& cu, uint32_t absPartIdx, uint32_
> >>
> >> /* in each of these conditions, the subdiv flag is implied and not signaled,
> >> * so we have checks to make sure the implied value matches our intentions */
> >> - if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> >> + if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
> >> {
> >> X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
> >> }
> >> diff --git a/source/encoder/search.cpp b/source/encoder/search.cpp
> >> index 0522f52cc..3d61caabf 100644
> >> --- a/source/encoder/search.cpp
> >> +++ b/source/encoder/search.cpp
> >> @@ -1541,8 +1541,8 @@ sse_t Search::estIntraPredQT(Mode &intraMode, const CUGeom& cuGeom, const uint32
> >> {
> >> uint32_t bmode = 0;
> >>
> >> - if (intraMode.cu.m_lumaIntraDir[puIdx] != (uint8_t)ALL_IDX)
> >> - bmode = intraMode.cu.m_lumaIntraDir[puIdx];
> >> + if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
> >> + bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
> >> else
> >> {
> >> uint64_t candCostList[MAX_RD_INTRA_MODES];
> >
> >--
> >Qiwei Wen
> >Software Developer
> >Exablaze Pty Ltd
> >✆ 0430872689
> >LinkedIn
> >_______________________________________________
> >x265-devel mailing list
> >x265-devel at videolan.org
> >https://mailman.videolan.org/listinfo/x265-devel
>
>
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel
--
Qiwei Wen
Software Developer
Exablaze Pty Ltd
✆ 0430872689
LinkedIn
More information about the x265-devel
mailing list