[x265] Max TU Size

[Nicolas Morey-Chaisemartin]

If I change the max transform size to MIN(cu log2 size, 5), I end up segfaulting during quantification on a 32x32 block.
This is because in Search::codeIntraLumaQT, we get the coeff buffer this way:
         coeff_t* coeffY       = m_rqt[qtLayer].coeffRQT[0] + coeffOffsetY;

qtLayer being 3 for a 32x32 TU block.
But because m_numLayer is now computed this way:

     m_numLayers = g_log2Size[param.maxCUSize] - 2;

It's equal to 3 on 32x32 CTU. So m_rqt is long enough but not initialize for the largest TU anymore...

Nicolas


On 10/27/2014 05:08 PM, Steve Borho wrote:
> On 10/27, Nicolas Morey-Chaisemartin wrote:
>> Simply changing back the computation SEGV on master (worked on 1.3).
>>  From the look of things, the number of layers in the Search class changed from
>>      m_numLayers = g_log2Size[param.maxCUSize] - 2 + 1;
>> to
>>      m_numLayers = g_log2Size[param.maxCUSize] - 2;
>>
>> Some time ago...
>> Reverting it to the previous value fixes the issue too.
> I'm sorry, which SEGV are you referring to?  There was a bug in a
> related area last week that was only reproducible in placebo and was
> fixed by a09b45ead8e0
>
> FWIW, the code churn should be winding down in the next couple of days.
> I plan to merge to stable by tomorrow and tag a 1.4 release at the end of
> the week.
>