[x265] bug in sao.cpp

[Steve Borho]

On 08/29, Dzung Hoang wrote:
> A bug was introduced with revision 02ec546246ad with the aim of fixing
> compiler warnings.
> 
> In the function SAO::processSaoUnitAll() in sao.cpp,
> 
>     int i; uint32_t edgeType;
> 
> was changed to
> 
>     uint32_t i; uint32_t edgeType;
> 
> This change caused a segmentation fault (on a particular test case)
> here:
> 
>     for (i = 0; i < (1 << X265_DEPTH); i++)
>         offsetBo[i] = m_clipTable[i + offset[m_tableBo[i]]];
> 
> The problem is that the expression "i + offset[m_tableBo[i]" is
> converted to unsigned type by the compiler. When this expression would
> otherwise be negative, the conversion results in a very big positive
> number, which causes the segmentation fault.
> 
> The lesson is to be very careful when changing between signed and
> unsigned types when the compiler displays a warning. Editing code to
> remove warnings sometimes do more harm than good. Perhaps the better
> approach to fixing type mismatch warnings is to convert the unsigned
> variables to signed or with type casting.
> 
> Similar code exists in SAO::processSaoUnitRow() so the same fix should
> be applied to this function as well.

Thanks, I've queued a fix which removes the global declaration of 'i' in
those two functions and declares i with the appropriate type for each
loop.

-- 
Steve Borho