[x265] Enabling intra NxN partition mode when MinCbSizeY isgreater than 8
Qiwei Wen
wenqiweiabcd at gmail.com
Fri Dec 19 09:10:23 UTC 2025
Hi Richard,
I now realise that this branch is for the analysis load mode only. I
only changed that line because the way it assumes that the CU size is
8 (and therefore the 2x2 partitions that intra PUs map to can be
indexed by the PU index) jumped out to me.
Happy to lift this change out of the patch and submit a separate patch
for the analysis load mode, as Chen suggests.
However, with this patch in, it's the first pass encoding that's
missing some fixes and produces an incorrect CABAC bitstream when I
configure MinCbSizeY to be greater than 8... that's the problem I'm
trying to get past at this stage.
Thanks,
Qiwei
On Fri, 19 Dec 2025 at 19:33, Richard <ccc7922 at foxmail.com> wrote:
>
> The problem is this
> if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
> bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
> Should not be changed like this.
> This if branch is for analysis-load mode, in a normal "first time" prediction this should always go to the else branch.
>
>
>
> Original
> From: Qiwei Wen <wenqiweiabcd at gmail.com>
> Date: 2025-12-19 05:53
> To: chen <chenm003 at 163.com>
> Cc: Development for x265 <x265-devel at videolan.org>
> Subject: Re: [x265] Enabling intra NxN partition mode when MinCbSizeY isgreater than 8
>
>
> Thanks Chen, will apply these suggestions as soon as I get past the cabac issues. As I mentioned, the patch is missing some fixes and results in CABAC errors whenever intra NxN is chosen in case that MinCbSizeY != 8.
>
>
> Any suggestions on where else to look is greatly appreciated
>
>
> Qiwei Wen
> Software Developer
> Exablaze Pty Ltd
> ✆ 0430872689
> LinkedIn
>
>
>
>
>
>
>
> On Fri, Dec 19, 2025, 2:58 AM chen <chenm003 at 163.com> wrote:
> Hi Wen,
>
>
> Thank for the patch.
>
>
> I have some comments,
>
>
> (1) access global variant is higher cost, how about change below?
> cuGeom.log2CUSize == g_log2Size[m_param->minCUSize]
> (1 << cuGeom.log2CUSize) == m_param->minCUSize
>
>
> (2) looks mixed another fix on array index typo, suggest separate patch
> m_lumaIntraDir[absPartIdx]
>
>
> Regards,
> Chen
>
>
>
> At 2025-12-18 21:26:30, "Qiwei Wen" <wenqiweiabcd at gmail.com> wrote:
> >Dear Gurus,
> >
> >As you know, the HEVC spec allows NxN partition mode for intra-coded
> >CUs when the CU is of size MinCbSizeY. I assume either for coding
> >efficiency or historical reasons, x265 won't explore PART_NxN if the
> >minimum CU size is not 8. I'm working on a project that involves using
> >x265 to prototype intra-coding algorithms and it'd be ideal for my use
> >case if the limitation could be lifted.
> >
> >I realise that there are a few places in the code base that assume
> >that intra NxN implies a CU size of 8. Below is my attempt at fixing
> >it, the result of which is a non-local CABAC error, where the first
> >few intra NxN CU is coded correctly, but the CABAC state starts to
> >diverge to the point of making the bitstream undecodeable a few CUs
> >further down.
> >
> >I'd appreciate any pointers on if there are more places in the code
> >that makes the assumption and what the possible causes could be.
> >
> >Thanks,
> >Qiwei
> >
> >> diff --git a/source/encoder/analysis.cpp b/source/encoder/analysis.cpp
> >> index b219d5da4..4c42faba6 100644
> >> --- a/source/encoder/analysis.cpp
> >> +++ b/source/encoder/analysis.cpp
> >> @@ -580,7 +580,7 @@ uint64_t Analysis::compressIntraCU(const CUData& parentCTU, const CUGeom& cuGeom
> >> checkIntra(md.pred[PRED_INTRA], cuGeom, SIZE_2Nx2N);
> >> checkBestMode(md.pred[PRED_INTRA], depth);
> >>
> >> - if (cuGeom.log2CUSize == 3 && m_slice->m_sps->quadtreeTULog2MinSize < 3)
> >> + if (cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] && m_slice->m_sps->quadtreeTULog2MinSize < g_log2Size[m_param->minCUSize])
> >> {
> >> md.pred[PRED_INTRA_NxN].cu.initSubCU(parentCTU, cuGeom, qp);
> >> checkIntra(md.pred[PRED_INTRA_NxN], cuGeom, SIZE_NxN);
> >> diff --git a/source/encoder/entropy.cpp b/source/encoder/entropy.cpp
> >> index d4bf73d53..9aecbb5e5 100644
> >> --- a/source/encoder/entropy.cpp
> >> +++ b/source/encoder/entropy.cpp
> >> @@ -1328,7 +1328,7 @@ void Entropy::encodeTransform(const CUData& cu, uint32_t absPartIdx, uint32_t cu
> >>
> >> /* in each of these conditions, the subdiv flag is implied and not signaled,
> >> * so we have checks to make sure the implied value matches our intentions */
> >> - if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> >> + if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
> >> {
> >> X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
> >> }
> >> @@ -1463,7 +1463,7 @@ void Entropy::encodeTransformLuma(const CUData& cu, uint32_t absPartIdx, uint32_
> >>
> >> /* in each of these conditions, the subdiv flag is implied and not signaled,
> >> * so we have checks to make sure the implied value matches our intentions */
> >> - if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == MIN_LOG2_CU_SIZE)
> >> + if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize])
> >> {
> >> X265_CHECK(subdiv, "intra NxN requires TU depth below CU depth\n");
> >> }
> >> diff --git a/source/encoder/search.cpp b/source/encoder/search.cpp
> >> index 0522f52cc..3d61caabf 100644
> >> --- a/source/encoder/search.cpp
> >> +++ b/source/encoder/search.cpp
> >> @@ -1541,8 +1541,8 @@ sse_t Search::estIntraPredQT(Mode &intraMode, const CUGeom& cuGeom, const uint32
> >> {
> >> uint32_t bmode = 0;
> >>
> >> - if (intraMode.cu.m_lumaIntraDir[puIdx] != (uint8_t)ALL_IDX)
> >> - bmode = intraMode.cu.m_lumaIntraDir[puIdx];
> >> + if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX)
> >> + bmode = intraMode.cu.m_lumaIntraDir[absPartIdx];
> >> else
> >> {
> >> uint64_t candCostList[MAX_RD_INTRA_MODES];
> >
> >--
> >Qiwei Wen
> >Software Developer
> >Exablaze Pty Ltd
> >✆ 0430872689
> >LinkedIn
> >_______________________________________________
> >x265-devel mailing list
> >x265-devel at videolan.org
> >https://mailman.videolan.org/listinfo/x265-devel
>
>
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel
--
Qiwei Wen
Software Developer
Exablaze Pty Ltd
✆ 0430872689
LinkedIn
More information about the x265-devel
mailing list