[x264-devel] x264 quantization and AIC quastions

[Sergey A. Sablin]

decoder doesn't know anything about rounding parameters used in encoder, 
  which means encoder may use whatever it needs (check out trellis 
quantization, which uses different rounding for each coefficient)

about intra modes - that's internal encoder constants, they're not 
written into the bitstream.

Sergey.

math-BeSSeL at yandex.ru wrote:
> 
> 09.11.07, 06:15, Loren Merritt (lorenm at u.washington.edu):
> 
>> On Thu, 8 Nov 2007, math-BeSSeL at yandex.ru wrote:
>>> 1) For quantization process in the h.264 standard the following
>>> formula is used
>>>
>>> coef_q = (coef * mf + f)>> (15 + Qp/6) (*)  (for example for intra mb
>>> and coef>=0)
>>>
>>> In the x264 codec other formula is used
>>>
>>> coef_q" = ((coef + f") * mf")>> 16 (**)
>>>
>>> After the certain transformations from one it is possible to receive 
>>> another: (*) <-> (**).
>>> But there are conditions when the quantization result of the second
>>> formula (**) does not coincide with result of the first formula (*).
>>> Why this formula (**) is used?
>> Because (*) requires 32-bit math, whereas (**) can be implemented with 
>> only a 16x16>>16 multiply instruction (PMULHUW). That makes a SIMD 
>> implementation much faster.
> 
> Ок. I had such idea (4 coef in xmm register). 
> But (**) using gives an inaccuracy of calculations.
> 
> For example luma dc quantization: Qp = 29, coef(0,0) = coef_dc = 48.
> (*) -> coef_q = 1
> (**) -> coef_q" = 0.
> 
> I think, that the inaccuracy contains in f" parameter.
> For calculation f" (or "quant4_bias" in the code) the "deadzone" is used.
> I use value luma_deadzone[2] = {21, 11} -> deadzone[4] = {21, 11, 21, 11}.
> What do you think about this?
> 
> Certainly (**) using does not give visual artefacts.
> But if to compare results idct+dequantization of (*) and (**) in standart decoder, the luma component difference can <= 4 pixels (for the case coef in [-2048;2048]).  
> 
>>> 2) I_PRED_4x4_DC_LEFT / TOP, I_PRED_CHROMA_DC_LEFT / TOP,
>>> I_PRED_16x16_DC_LEFT / TOP.  I can is wrong, but the given modes are
>>> not present in the h.264 standard.
>> The standard describes a single DC function for each size, but that 
>> function does 4 different things depending on which neighbors are present. 
>> x264 splits them into 4 functions.
> 
> I understand. But the standart decoder incorrectly decodes a stream for the given modes.
> 
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel