[x264-devel] AddressSanitizer: heap-buffer-overflow at encoder/me.c:406 with 4 threads

[BugMaster]

On Tue, 9 Apr 2019 16:01:31 +0800, Hongxu Chen wrote:
> Hi,

>     On my machine, when running x264 with 4 threads (sandbox
> version, git HEAD d4099dd4c722f52c4f3c14575d7d39eb8fadb97f), it may
> cause a heap-buffer-overflow (read) at encoder/me.c:406.


> $ ./x264-asan/install/bin/x264 --threads 4 --quiet --output /dev/null hbo_me.c:406_1
> =================================================================
> ==23250==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x621000030904 at pc 0x000000755471 bp 0x7fadac672df0 sp 0x7fadac672de8
> READ of size 2 at 0x621000030904 thread T4
>     #0 0x755470 in x264_8_me_search_ref
> /home/hongxu/work/x264/x264-asan/encoder/me.c:406:13
>     #1 0x6df538 in mb_analyse_inter_p16x16
> /home/hongxu/work/x264/x264-asan/encoder/analyse.c:1275:13
>     #2 0x6df538 in x264_8_macroblock_analyse
> /home/hongxu/work/x264/x264-asan/encoder/analyse.c:3026
>     #3 0x59f274 in slice_write
> /home/hongxu/work/x264/x264-asan/encoder/encoder.c:2775:9
>     #4 0x591bee in slices_write
> /home/hongxu/work/x264/x264-asan/encoder/encoder.c:3116:13
>     #5 0x5a9fdf in threadpool_thread_internal
> /home/hongxu/work/x264/x264-asan/common/threadpool.c:69:20
>     #6 0x5fefeb in x264_stack_align
> (/home/hongxu/work/x264/x264-asan/install/bin/x264+0x5fefeb)

> 0x621000030904 is located 2 bytes to the right of 4098-byte region [0x62100002f900,0x621000030902)
> allocated by thread T0 here:
>     #0 0x4d4948 in __interceptor_memalign.localalias.1
> (/home/hongxu/work/x264/x264-asan/install/bin/x264+0x4d4948)
>     #1 0x548fcc in x264_malloc
> /home/hongxu/work/x264/x264-asan/common/base.c:124:21

> Thread T4 created by T0 here:
>     #0 0x4377a0 in pthread_create
> (/home/hongxu/work/x264/x264-asan/install/bin/x264+0x4377a0)
>     #1 0x5a96e9 in x264_8_threadpool_init
> /home/hongxu/work/x264/x264-asan/common/threadpool.c:111:13

> SUMMARY: AddressSanitizer: heap-buffer-overflow
> /home/hongxu/work/x264/x264-asan/encoder/me.c:406:13 in x264_8_me_search_ref
> Shadow bytes around the buggy address:
>   0x0c427fffe0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c427fffe110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>>0x0c427fffe120:[02]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>   0x0c427fffe170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> Shadow byte legend (one shadow byte represents 8 application bytes):
>   Addressable:           00
>   Partially addressable: 01 02 03 04 05 06 07 
>   Heap left redzone:       fa
>   Freed heap region:       fd
>   Stack left redzone:      f1
>   Stack mid redzone:       f2
>   Stack right redzone:     f3
>   Stack after return:      f5
>   Stack use after scope:   f8
>   Global redzone:          f9
>   Global init order:       f6
>   Poisoned by user:        f7
>   Container overflow:      fc
>   Array cookie:            ac
>   Intra object redzone:    bb
>   ASan internal:           fe
>   Left alloca redzone:     ca
>   Right alloca redzone:    cb
> ==23250==ABORTING
> [1]    23250 abort      ./x264-asan/install/bin/x264 --threads 4 --quiet --output /dev/null

>   When running with 1,2,3 threads, it crashes at different places such as encoder/me.c:244.
>   When running with 5 or more threads, everything seems fine.

> Best Regards,
> Hongxu

Hi.

Do this happen only with interlaced encoding?
Can you test with attached patch. It should fix heap-buffer-overflow
(read) with interlaced encoding.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-heap-buffer-overflow-read-detected-by-ASan-with-.patch
Type: application/octet-stream
Size: 1524 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20190412/45b199fe/attachment.obj>