[x264-devel] Re: Trellis RD

[Laurent Aimar]

On Tue, Aug 17, 2004, Loren Merritt wrote:
> Does anyone know whether trellis quantization should be applicable to
> encoding with CABAC? I tried a reasonably direct port of lavc's trellis
> algorithm, and it seems to be working, but doesn't noticably improve
> quality/bitrate.
> 
> Possible issues:
> CABAC doesn't gain anything from a run of zeros (the zero flags each use a
> separate context.)
> I had to calculate rate based on the entropy added to the CABAC model,
> not on the actual number of bits output (as that would just bias towards
> delayed outputs).
 Another possible issue: the way you compute distortion is wrong.
In h264 it's not that easy, you have to be carefull on what you use.
(Maybe you have already checked that, but in case not...)

 With:
 (Dij) = (Sourceij - Predictionij) ( 0<=i<=16, 0<=j<=16)

 You have :

 D -> DCT -> Quantization -> Coeffs -> Dequantization -> IDCT -> DeltaOut
                               |
                               +-CABAC/CAVLC.

 The DCT(D) is (C1*D*C1).*R1   (where (X .OP Y)ij = (Xij OP Yij)
 The Quantization is ./Q       (Qij = f(Qp))
 The Dequantization is .*Q
 The IDCT(Y) is C2*(Y.*R2)*C2

 *BUT* to be faster and avoid rounding problems, H264 merges .*R1./Q
in the quantization process and merges .*Q.*R2 in the dequantization
process (and in fact they even changed R2 a bit, and *64).
 So the DCT/IDCT aren't anymore real ones, quant/dequant doesn't do only
what they should.

 So computing distortion isn't as easy as in the mpeg4 trellis.

 I suggest you to read carefully http://www.vcodex.com/h264_transform.pdf

-- 
fenrir

-- 
This is the x264-devel mailing-list
If you are in trouble, please contact <postmaster at videolan.org>