[x264-devel] CAVLC encoding broken at high resolutions

[Martin Fiedler]

Hi,

while trying to encode video at very high resolutions (quad HD in my 
case), I noticed that x264 generates broken H.264 streams when using 
CAVLC instead of CABAC.

Here is an example input video stream (~25 MB, 100 frames, 3840x2160, 
losslessly compressed using FFV1) that exhibits the problem:
     http://www.parhelic.de/~keyj/x264_broken_cavlc_example.avi
when encoded using
     x264 --no-cabac -o blah.264 x264_broken_cavlc_example.avi
the resulting file shows visible artifacts. If --no-cabac is omitted, 
the output is just fine.

I found out that this bug has been introduced in r1327 87de234 and is 
present until today (r1713 c276662). Since r1327 introduced weighted 
P-frame prediction, I expected --weightp 0 to work around the bug, but 
this is only true for r1327. In the current revision, --weightp 0 
actually makes it worse.

I could not reproduce the bug with low resolutions (like 1920x1080) or 
with CABAC. Adding --no-mbtree also does not improve the situation.

I hope that this report is helpful :)

Best regards,

Martin Fiedler