[x265] bug in sao.cpp

[Dzung Hoang]

A bug was introduced with revision 02ec546246ad with the aim of fixing compiler warnings.

In the function SAO::processSaoUnitAll() in sao.cpp,

    int i;
    uint32_t edgeType;

was changed to

    uint32_t i;
    uint32_t edgeType;

This change caused a segmentation fault (on a particular test case) here:

                        for (i = 0; i < (1 << X265_DEPTH); i++)
                            offsetBo[i] = m_clipTable[i + offset[m_tableBo[i]]];

The problem is that the expression "i + offset[m_tableBo[i]" is converted to unsigned type by the compiler. When this expression would otherwise be negative, the conversion results in a very big positive number, which causes the segmentation fault.

The lesson is to be very careful when changing between signed and unsigned types when the compiler displays a warning. Editing code to remove warnings sometimes do more harm than good. Perhaps the better approach to fixing type mismatch warnings is to convert the unsigned variables to signed or with type casting.

Similar code exists in SAO::processSaoUnitRow() so the same fix should be applied to this function as well.

Best regards,
- Dzung Hoang
Zenverge, Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x265-devel/attachments/20140829/953f5469/attachment.html>