[x264-devel] Potential bug in transform bypass 8x8

[Alain Champenois]

Good day x264-dev,

I was reviewing H264 transform bypass functionality (lossless) implementation in 
x264 and base on my understanding of the H264 specifications, x264 doesn't implement 
I_PRED_8x8_V or I_PRED_8x8_H modes correctly.

For all intra 8x8 predictions, the spec mentions the filtered samples should be used
but x264 is using the source samples when doing I_PRED_8x8_V or I_PRED_8x8_H 
modes in lossless (see detail explication below).


Here is the snippet of code from x264 macroblock.c handling 8x8 for lossless:

void x264_predict_lossless_8x8( x264_t *h, pixel *p_dst, int p, int idx, int i_mode, pixel edge[36] )
...
    if( i_mode == I_PRED_8x8_V )
        h->mc.copy[PIXEL_8x8]( p_dst, FDEC_STRIDE, p_src-stride, stride, 8 );
    else if( i_mode == I_PRED_8x8_H )
        h->mc.copy[PIXEL_8x8]( p_dst, FDEC_STRIDE, p_src-1, stride, 8 );
    else
        h->predict_8x8[i_mode]( p_dst, edge );
...

For I_PRED_8x8_V and I_PRED_8x8_H modes, the source samples stored in 'p_src' are used 
instead of the filtered samples stored in 'edge' which is not conform with the spec.


x264 lossless streams are compatible with libavcodec because both are doing the 
same implementation in this case (ie: using source samples instead of filtered 
samples). I've also confirmed that CoreAvc has the same implementation as libavcodec.

On the other side, JM 18.5 implementation (encoder and decoder) does match my 
understanding of the H264 spec, and when I disabled 8x8 transform during encoding 
(using --no-8x8dct), x264 streams become compatible with JM decoder (lossless). 
I've also tested the reverse (encoding with JM with 8x8 transform disabled and 
decoding with libavcodec) and it's also working.

JM and x264/libavcodec are therefore currently compatible for lossless only when 
intra 8x8 transform is disabled (tested in 420/422/444 and 8/10 bits). 


The command line I used to show the problem was:

  x264 --crf 0 --input-csp i420 --input-res 352x288 crowdrun_cif.yuv -o crowdrun_cif.264

but the issue can be reproduce with any YUV of 1 frame (MB intra 8x8 being very
common). The difference can be seen by decoding with JM 18.5 and comparing with
the source YUV.


Could someone confirmed that this is real issue of x264 (and livavcodec) not being 
compliant with the H264 spec in transform bypass 8x8.

  Regards
  Alain
 
  

## Information about transform 8x8 in Rec ITU-T H.264 (01/2012)

From the spec, the 8x8 prediction is always generated with filtered version of
the neighbouring samples. This prediction is sum with the residual process to get 
the reconstructed picture.


For all 8x8 luma blocks of the luma component of a macroblock with luma8x8BlkIdx= 0..3

 # Intra prediction
 
  - Get the 25 neighbouring samples p[x,y]                          section 8.3.2.2

  - Apply filtering on p[x,y] and return in p'[x,y]                 section 8.3.2.2.1

  - Generate the prediction from p'[x,y] and return Pred8x8L[x,y]   section 8.3.2.2.2 to 8.3.2.2.10

  - Output PredL=Pred8x8L from the intra prediction.                section 8.3.2 (8-71)

 
 # Transform coefficient
 
  - PrelL as input into the transform coefficient decoding          section 8.5

  - if(TransformBypassModeFlag==1)                                  section 8.5.3 item 3.

      - Apply the intra residual transform-bypass decoding process 
        and return rij                                              section 8.5.14

  - Sum the residual to the prediction and return uij=PrelL+rij     section 8.5.3 item 5. (8-303)

  - Do the picture reconstruction prior to the deblocking
    process with uij                                                section 8.5.13