[x264-devel] Overflow within cabac treillis

[Gabriel Bouvigne]

Within quant_trellis_cabac, I ended up with an overflow when computing 
cost_sig0 (l.400).

In my case, i_lambda2 is 1515192, and x264_cabac_size_decision_noup 
returned 1449. The multiplication is thus overflowing, as cost_sig0 is a 
32 bits signed integer.

At this point I am not yet sure if this is a bug within the trunk cabac 
treillis, or if I screwed up something locally.


--
Gabriel