[x264-devel] x264 seg fault issue when encode 4K video

[홍석진]

Hi. 
​
I have a problem when I try to encode some 4K videos with libx264.
​
I'm using x264 options as below
-c:v libx264 -profile:v high -x264opts bframes=2:b_adapt=2:rc_lookahead=48:keyint=60:keyint-min=30 -threads 21 -pix_fmt yuv420p -vf scale=3840:2160 -f mp4.​
​
Actually, I tried to encode by changing arguments as below
- change number of threads (21 -> 16 or 10)
- add force_key_frames - delete bframe options
- change b_adapt (2 -> 1, it means that X264_B_ADAPT_TRELLIS to X264_B_ADAPT_FAST as you know)
​
Changing b_adapt option 2 to 1 works well.
Otherwise sometimes it occurs same seg fault issues.
​
I attached back trace info
#0 0x0000000001f5a721 in x264_slicetype_frame_cost () #1 0x0000000001f5c89e in x264_slicetype_analyse () #2 0x0000000001ef9be6 in x264_stack_align () #3 0x0000000001ef201c in x264_lookahead_slicetype_decide () #4 0x0000000001ef21d9 in x264_lookahead_thread () #5 0x00007ffff72a6e25 in start_thread () from /usr/lib64/libpthread.so.0 #6 0x00007ffff660034d in clone () from /usr/lib64/libc.so.6​
I tried to analyze x264 codes and found something weird situation.
​
​
static int slicetype_frame_cost( x264_t *h, x264_mb_analysis_t *a, x264_frame_t **frames, int p0, int p1, int b ) /* For each list, check to see whether we have lowres motion-searched this reference frame before. */ x264_log( h, X264_LOG_INFO, "[slicetype_frame_cost] p0: %d, p1: %d, b: %d, b-p0-1: %d, p1-b-1: %d\n", p0, p1, b, b-p0-1, p1-b-1); do_search[0] = b != p0 && fenc->lowres_mvs[0][b-p0-1][0][0] == 0x7FFF; do_search[1] = b != p1 && fenc->lowres_mvs[1][p1-b-1][0][0] == 0x7FFF;​
I think this problem is array index out of range exception in x264_slicetype_frame_cost function.
Seg fault occurs when function parameters b=48, p0=44 and p1=48.
It means [b-p0-1] is 3. But fenc->lowres_mvs[0][3] is not allocated.
​
​
cur_nonb = i; // before segmentation fault, i is 47. (i = number of frames in frame buffer - 1) while( IS_X264_TYPE_B( frames[cur_nonb]->i_type ) && cur_nonb > 0 ) cur_nonb--;​
I understood p0 means the index of non-bframe (reverse position) in frames, macroblock_tree function.
​
Before seg fault, frame types of frames changed 1530000000000000000000000000000000000000000000000 to 153553553553553553553553553553553553355333533555.
So that, cur_nonb is calculated 44. After that b-p0-1 be 3.
Except in this case, b-p0-1 and p1-b-1 are always calculated -1 to 2.
​
When I changed frames->i_type X264_TYPE_B to X264_TYPE_P forced if greater than 2 consecutive b frames are exist, it solved.
But I'm not sure whether can be used or not because this solution is not cool.
(Anyway, because of this issue I changed b-adapt option 2 to 1)
​
​
To sum up, seg fault occurs when encode some 4K contents.
By changing x264 options, it can be avoided (especially b-adapt option).
​
Please let me know if you need more information.
​
Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20180702/d2f9cccd/attachment-0001.html>