[x264-devel] AddressSanitizer: heap-buffer-overflow at encoder/me.c:406 with 4 threads

[Hongxu Chen]

Hi,

    On my machine, when running x264 with *4 threads* (sandbox version, git
HEAD d4099dd4c722f52c4f3c14575d7d39eb8fadb97f), it may cause
a heap-buffer-overflow (read) at encoder/me.c:406.

$ ./x264-asan/install/bin/x264 --threads 4 --quiet --output /dev/null
hbo_me.c:406_1
=================================================================
==23250==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x621000030904 at pc 0x000000755471 bp 0x7fadac672df0 sp 0x7fadac672de8
READ of size 2 at 0x621000030904 thread T4
    #0 0x755470 in x264_8_me_search_ref
/home/hongxu/work/x264/x264-asan/encoder/me.c:406:13
    #1 0x6df538 in mb_analyse_inter_p16x16
/home/hongxu/work/x264/x264-asan/encoder/analyse.c:1275:13
    #2 0x6df538 in x264_8_macroblock_analyse
/home/hongxu/work/x264/x264-asan/encoder/analyse.c:3026
    #3 0x59f274 in slice_write
/home/hongxu/work/x264/x264-asan/encoder/encoder.c:2775:9
    #4 0x591bee in slices_write
/home/hongxu/work/x264/x264-asan/encoder/encoder.c:3116:13
    #5 0x5a9fdf in threadpool_thread_internal
/home/hongxu/work/x264/x264-asan/common/threadpool.c:69:20
    #6 0x5fefeb in x264_stack_align
(/home/hongxu/work/x264/x264-asan/install/bin/x264+0x5fefeb)

0x621000030904 is located 2 bytes to the right of 4098-byte region
[0x62100002f900,0x621000030902)
allocated by thread T0 here:
    #0 0x4d4948 in __interceptor_memalign.localalias.1
(/home/hongxu/work/x264/x264-asan/install/bin/x264+0x4d4948)
    #1 0x548fcc in x264_malloc
/home/hongxu/work/x264/x264-asan/common/base.c:124:21

Thread T4 created by T0 here:
    #0 0x4377a0 in pthread_create
(/home/hongxu/work/x264/x264-asan/install/bin/x264+0x4377a0)
    #1 0x5a96e9 in x264_8_threadpool_init
/home/hongxu/work/x264/x264-asan/common/threadpool.c:111:13

SUMMARY: AddressSanitizer: heap-buffer-overflow
/home/hongxu/work/x264/x264-asan/encoder/me.c:406:13 in x264_8_me_search_ref
Shadow bytes around the buggy address:
  0x0c427fffe0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fffe0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fffe0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fffe100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fffe110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c427fffe120:[02]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fffe130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fffe140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fffe150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fffe160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fffe170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==23250==ABORTING
[1]    23250 abort      ./x264-asan/install/bin/x264 --threads 4 --quiet
--output /dev/null

  When running with 1,2,3 threads, it crashes at different places such as
encoder/me.c:244.
  When running with 5 or more threads, everything seems fine.

Best Regards,
Hongxu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190409/fba98a39/attachment.html>