[x265] source/common/intrapred.cpp -- C version of angular intra performs double transpose

[Matt Johnson]

Ugh.  Sorry all, ignore this entire chain of emails.

I've just realized that the contract of the *allangs() functions is that 
they're *supposed* to return transposed results for horizontal modes (to 
make things faster in the vectorized case, I gather).  The caller is 
responsible for dealing with these transposed results accordingly; for 
example, if you want to do SATD on them, you should use a transposed 
version of your "original" pixel block, so the transposition cancels out.

-Matt

On 12/13/2013 03:06 PM, Matt Johnson wrote:
> (sorry to break this up into so many separate emails :( )
>
> The behavior has been present since the all_angs function was created in
> source/common/IntraPred.cpp in revision 2456 (6/18/13).  At that time,
> the function was called xPredIntraAngs4().
>
> I can't find a place where the horizontal mode predicted pixels are
> transposed a third time (to get the correct value), so it seems like an
> incorrect SATD value and so forth will be computed.
>
> -Matt
>
> On 12/13/2013 02:54 PM, Matt Johnson wrote:
>> It also looks like the assembly/SSE variants must be doing the same
>> thing, since commenting out the transpose in all_angs_pred_c() results
>> in a failure in the TestBench binary.
>>
>> On 12/13/2013 01:40 PM, Matt Johnson wrote:
>>> Hi all,
>>>      It looks like the allangs C variant of angular intra prediction
>>> transposes the predicted pixel matrix twice; the intra_pred_ang_c()
>>> function, which does a single mode prediction, transposes the predicted
>>> pixel matrix if necessary, then the wrapper all_angs_pred_c() transposes
>>> it again.  It seems like the all_angs_pred_c() transpose should be
>>> removed.
>>>
>>>      As a side note, should the planad_pred_c() templated function be
>>> renamed planar_pred_c()?
>>>
>>> Thanks,
>>> Matt
>>> _______________________________________________
>>> x265-devel mailing list
>>> x265-devel at videolan.org
>>> https://mailman.videolan.org/listinfo/x265-devel
>> _______________________________________________
>> x265-devel mailing list
>> x265-devel at videolan.org
>> https://mailman.videolan.org/listinfo/x265-devel
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel