[x265] [PATCH] sse: fix data type for sse functions

[Steve Borho]

On 09/29, Deepthi Nandakumar wrote:
> So, I dont think it's fair to upgrade the datatypes of these variables just
> to satisfy debug/check code. We're unnecessarily adding cycles.
> 
> I think the best solution here might be to remove these hacky checks in
> Mode.invalidate(), and Mode.ok() and add more robust checking into
> rdcost.h.

the functions are hacky, but they cannot be replaced by checks in
rdcost.h. Their intention is not to find out-of-range function return
values, but instead to find uninitialized cost variables. The functions
did find a few bugs when they were first added, but they are mostly a
liability now that main12 has increased the dynamic range of the pixel
compare functions.

What this has all shown, though, is that any variable which needs to sum
up multiple distortion costs needs to be a uint64_t or you risk overflow
in high bit depth builds (10 or 12). Many of the Mode variables fall
into this category.

It is definitely wrong for the output of an SSE primitive to be stored
into a uint32_t. It needs to be either sse_ret_t, if the value is never
incremented after returned, else a uint64_t.

-- 
Steve Borho