[x264-devel] AddressSanitizer: heap-buffer-overflow at encoder/me.c:244 with 4 threads

[Hongxu Chen]

FYI, here is another POC that can trigger the crash when 1, 2, 3, 4 threads
are specified on my machine.

Best Regards,
Hongxu


On Mon, Apr 8, 2019 at 2:35 PM Hongxu Chen <leftcopy.chx at gmail.com> wrote:

> Hi,
>
>     On my machine, when running x264 (sandbox version, git HEAD
> d4099dd4c722f52c4f3c14575d7d39eb8fadb97f) with 4 threads, it may cause
> a heap-buffer-overflow at encoder/me.c:244.
>
> $ ./x264-asan/install/bin/x264 --threads 4 --quiet --output /dev/null
> x264_poc
> =================================================================
> ==32524==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x621000031d20 at pc 0x00000074ce72 bp 0x7fe714073df0 sp 0x7fe714073de8
> READ of size 2 at 0x621000031d20 thread T2
>     #0 0x74ce71 in x264_8_me_search_ref
> /home/hongxu/work/x264/x264-asan/encoder/me.c:244:21
>     #1 0x6df538 in mb_analyse_inter_p16x16
> /home/hongxu/work/x264/x264-asan/encoder/analyse.c:1275:13
>     #2 0x6df538 in x264_8_macroblock_analyse
> /home/hongxu/work/x264/x264-asan/encoder/analyse.c:3026
>     #3 0x59f274 in slice_write
> /home/hongxu/work/x264/x264-asan/encoder/encoder.c:2775:9
>     #4 0x591bee in slices_write
> /home/hongxu/work/x264/x264-asan/encoder/encoder.c:3116:13
>     #5 0x5a9fdf in threadpool_thread_internal
> /home/hongxu/work/x264/x264-asan/common/threadpool.c:69:20
>     #6 0x5fefeb in x264_stack_align
> (/home/hongxu/work/x264/x264-asan/install/bin/x264+0x5fefeb)
>
> 0x621000031d20 is located 30 bytes to the right of 4098-byte region
> [0x621000030d00,0x621000031d02)
> allocated by thread T0 here:
>     #0 0x4d4948 in __interceptor_memalign.localalias.1
> (/home/hongxu/work/x264/x264-asan/install/bin/x264+0x4d4948)
>     #1 0x548fcc in x264_malloc
> /home/hongxu/work/x264/x264-asan/common/base.c:124:21
>
> Thread T2 created by T0 here:
>     #0 0x4377a0 in pthread_create
> (/home/hongxu/work/x264/x264-asan/install/bin/x264+0x4377a0)
>     #1 0x5a96e9 in x264_8_threadpool_init
> /home/hongxu/work/x264/x264-asan/common/threadpool.c:111:13
>
> SUMMARY: AddressSanitizer: heap-buffer-overflow
> /home/hongxu/work/x264/x264-asan/encoder/me.c:244:21 in x264_8_me_search_ref
> Shadow bytes around the buggy address:
>   0x0c427fffe350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> =>0x0c427fffe3a0: 02 fa fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe3b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe3c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe3d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe3e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe3f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> Shadow byte legend (one shadow byte represents 8 application bytes):
>   Addressable:           00
>   Partially addressable: 01 02 03 04 05 06 07
>   Heap left redzone:       fa
>   Freed heap region:       fd
>   Stack left redzone:      f1
>   Stack mid redzone:       f2
>   Stack right redzone:     f3
>   Stack after return:      f5
>   Stack use after scope:   f8
>   Global redzone:          f9
>   Global init order:       f6
>   Poisoned by user:        f7
>   Container overflow:      fc
>   Array cookie:            ac
>   Intra object redzone:    bb
>   ASan internal:           fe
>   Left alloca redzone:     ca
>   Right alloca redzone:    cb
> ==32524==ABORTING
> [1]    32524 abort      ./x264-asan/install/bin/x264 --threads 4 --quiet
> --output /dev/null x264_poc
>
> Till now, I can only see the crash stably when x264 is compiled with
> AddressSanitizer and run with 4 threads.
>
> Best Regards,
> Hongxu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190408/046cf7e7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: x264_poc_1
Type: application/octet-stream
Size: 11415 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190408/046cf7e7/attachment.obj>