[x264-devel] x264 quantization and AIC quastions

[math-BeSSeL at yandex.ru]

09.11.07, 06:15, Loren Merritt (lorenm at u.washington.edu):

> On Thu, 8 Nov 2007, math-BeSSeL at yandex.ru wrote:
> > 1) For quantization process in the h.264 standard the following
> > formula is used
> >
> > coef_q = (coef * mf + f)>> (15 + Qp/6) (*)  (for example for intra mb
> > and coef>=0)
> >
> > In the x264 codec other formula is used
> >
> > coef_q" = ((coef + f") * mf")>> 16 (**)
> >
> > After the certain transformations from one it is possible to receive 
> > another: (*) <-> (**).
> > But there are conditions when the quantization result of the second
> > formula (**) does not coincide with result of the first formula (*).
> > Why this formula (**) is used?
> Because (*) requires 32-bit math, whereas (**) can be implemented with 
> only a 16x16>>16 multiply instruction (PMULHUW). That makes a SIMD 
> implementation much faster.

Ок. I had such idea (4 coef in xmm register). 
But (**) using gives an inaccuracy of calculations.

For example luma dc quantization: Qp = 29, coef(0,0) = coef_dc = 48.
(*) -> coef_q = 1
(**) -> coef_q" = 0.

I think, that the inaccuracy contains in f" parameter.
For calculation f" (or "quant4_bias" in the code) the "deadzone" is used.
I use value luma_deadzone[2] = {21, 11} -> deadzone[4] = {21, 11, 21, 11}.
What do you think about this?

Certainly (**) using does not give visual artefacts.
But if to compare results idct+dequantization of (*) and (**) in standart decoder, the luma component difference can <= 4 pixels (for the case coef in [-2048;2048]).  

> > 2) I_PRED_4x4_DC_LEFT / TOP, I_PRED_CHROMA_DC_LEFT / TOP,
> > I_PRED_16x16_DC_LEFT / TOP.  I can is wrong, but the given modes are
> > not present in the h.264 standard.
> The standard describes a single DC function for each size, but that 
> function does 4 different things depending on which neighbors are present. 
> x264 splits them into 4 functions.

I understand. But the standart decoder incorrectly decodes a stream for the given modes.