[x264-devel] Potential bug in transform bypass 8x8

[Alain Champenois]

> >> This can already be done with --no-8x8dct, so I'm not sure it's worth
> >> bothering hacking into x264.

The --no-8x8dct flag will also work (lossless will become compliant) but it does 
disabled all intra8x8 modes while only intra8x8 HOR+VER really need to be turn off. 
The loss of compression when forcing -no-8x8dct remains low however (in my testing, 
0.03% instead of 0.02% for intra8x8 hor+ver only).

> > Would it not be more correct to handle this within libx264 rather than
> > externally?  The API user (like myself) may not know anything about H.264
> > lossless spec and the need to disable 8x8dct for lossless compression.
>
> Can this not be done programatically, i.e. set --no-8x8dct flags if
> the options parser encounters a lossless option?

I also see the need to make lossless compliant by default without requiring 
to specify an additionnal flag. Included is another patch which disabled 8x8dct
when lossless is enabled. 8x8dct was already disabled in lossless for a CAVLC
mode, and I just removed the IF condition to make it always disabled.

  Alain


diff -Naur a/encoder.c b/encoder.c
--- a/encoder.c
+++ b/encoder.c
@@ -629,9 +629,8 @@
         h->param.analyse.i_noise_reduction = 0;
         h->param.analyse.b_psy = 0;
         h->param.i_bframe = 0;
-        /* 8x8dct is not useful without RD in CAVLC lossless */
-        if( !h->param.b_cabac && h->param.analyse.i_subpel_refine < 6 )
-            h->param.analyse.b_transform_8x8 = 0;
+        /* disabling not compliant intra8x8 hor+ver modes */
+        h->param.analyse.b_transform_8x8 = 0;
     }
     if( h->param.rc.i_rc_method == X264_RC_CQP )
     {