[x264-devel] Overflow within cabac treillis

[Gabriel Bouvigne]

Gabriel Bouvigne a écrit :
> Within quant_trellis_cabac, I ended up with an overflow when computing 
> cost_sig0 (l.400).
> 
> In my case, i_lambda2 is 1515192, and x264_cabac_size_decision_noup 
> returned 1449. The multiplication is thus overflowing, as cost_sig0 is a 
> 32 bits signed integer.
> 
> At this point I am not yet sure if this is a bug within the trunk cabac 
> treillis, or if I screwed up something locally.

After further investigation, it seems to be a rare but real bug, that 
occurs when you mix high Qp and bad luck.

The fix is to use uint64_t within the computation of cost_sig0.

--
Gabriel