[x264-devel] Overflow within cabac treillis

[Sergey A. Sablin]

Hi Gabriel,

it'll be step back, as Loren replaced 64-bit arithmetic with 32-bit to 
optimize the speed, iirc. So more likely the fix will be to low 
precision of lambda, don't know how it may change the quality though. 
Perhaps not a significant change, but anyway.

Sergey.

Gabriel Bouvigne wrote:
> Gabriel Bouvigne a écrit :
>> Within quant_trellis_cabac, I ended up with an overflow when computing 
>> cost_sig0 (l.400).
>>
>> In my case, i_lambda2 is 1515192, and x264_cabac_size_decision_noup 
>> returned 1449. The multiplication is thus overflowing, as cost_sig0 is a 
>> 32 bits signed integer.
>>
>> At this point I am not yet sure if this is a bug within the trunk cabac 
>> treillis, or if I screwed up something locally.
> 
> After further investigation, it seems to be a rare but real bug, that 
> occurs when you mix high Qp and bad luck.
> 
> The fix is to use uint64_t within the computation of cost_sig0.
> 
> --
> Gabriel
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel
>