[x264-devel] Buffer overflow in x264_plane_copy_interleave_neon

[Kirill Batuzov]

Hi,

I think I've encountered a buffer overflow error in function 
x264_plane_copy_interleave_neon. It causes latest x264 to segfault
under QEMU user emulation or Valgrind.

x264_plane_copy_interleave_neon copies bytes in blocks of 16. If width
is not multiple of 16 it rounds it up (explicitly on the first loop
iteration, implicitly on the following iterations). This results in some
extra bytes being copied.

Usually nothing happens because it's only few bytes and there are some 
extra space in destination and some data following the source. In my 
case one of the source planes was near the page boundary and caused 
segmentation fault.

For comparrison, the same function on x86 has a wrapper written in C
which ensures that initial pointer is aligned and the length is a
multiple of 16. On ARM, assembler function is used durectly.

Relevant part of Valgrind output:

==7862== Invalid read of size 8
==7862==    at 0xE46D8: x264_plane_copy_interleave_neon (in /home/batuzovk/x264/x264)
==7862==  Address 0x516bffa is in a r-- mapped file /home/batuzovk/x264/x-class.y4m segment
==7862== 
==7862== 
==7862== Process terminating with default action of signal 11 (SIGSEGV)
==7862==  Access not within mapped region at address 0x516C000
==7862==    at 0xE46D8: x264_plane_copy_interleave_neon (in /home/batuzovk/x264/x264)

Backtrace (from QEMU with attached GDB):

(gdb) bt
#0  0x000e4f40 in x264_plane_copy_interleave_neon ()
#1  0x000180e8 in x264_frame_copy_picture ()
#2  0x0002e3d8 in x264_encoder_encode ()
#3  0x0019c29c in object.8955 ()

Source video:
http://www.nasa.gov/downloadable/videos/x-class_flare_outburst.mp4
converted to yuv4mpeg2 format before encoding
Resulting header:
YUV4MPEG2 W720 H720 F25:1 Ip A1:1 C420mpeg2 XYSCSS=420MPEG2

x264 options:
x264 --crf 24 --preset fast --threads 1 x-class.y4m -o x-class.264

I think that some other NEON functions may be affected by the same
issue. x264_plane_copy_deinterleave_neon is a prime suspect because it
looks exactly the same as x264_plane_copy_interleave_neon. Unfortunately
I do not have an input that triggers a error there.

-- 
Kirill