[x265] [PATCH] asm : saturation bug fix for luma_vss asm routine

[Deepthi Nandakumar]

This patch is pending, right Nabajit? I havent pushed the luma_vss
/chroma_vss assembly patches or the testbench edits to luma_vss.




On Wed, Jan 29, 2014 at 1:56 PM, chen <chenm003 at 163.com> wrote:

> >@@ -5105,8 +5108,9 @@
> >     pmaddwd    m5, [r6 + 3 * 16]
>
> >     paddd      m1, m5                          ;m1=[1+2+3+4+5+6+7+8]  Row2 end
> >     psrad      m1, 6
> >-
> >-    packssdw   m0, m1
> >+    pand       m1, m7
> >+
> >+    packusdw   m0, m1
> >
> >     movlps     [r2], m0
> >     movhps     [r2 + r3], m0
> PAND + PACKUSDW may avoid overflow problem, but it is wrong way here
> as you said, you got a result value 0x8D84, it is overflow on 16bits, so
> we need to find really reason,
> I check the HM code, it use Short, so I suggest you catch input data and
> put into HM to check HM's output.
>
>
> _______________________________________________
> x265-devel mailing list
> x265-devel at videolan.org
> https://mailman.videolan.org/listinfo/x265-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20140130/7c872db7/attachment.html>