[x264-devel] x264 seg fault issue when encode 4K video

[BugMaster]

On Mon, 02 Jul 2018 16:37:36 +0900, 홍석진 wrote:
> Hi.
>
> I have a problem when I try to encode some 4K videos with libx264.
>
> I'm using x264 options as below
> -c:v libx264 -profile:v high -x264opts bframes=2:b_adapt=2:rc_lookahead=48:keyint=60:keyint-min=30 -threads 21 -pix_fmt yuv420p -vf scale=3840:2160 -f mp4.?
>
> Actually, I tried to encode by changing arguments as below
> - change number of threads (21 -> 16 or 10)
> - add force_key_frames - delete bframe options
> - change b_adapt (2 -> 1, it means that X264_B_ADAPT_TRELLIS to X264_B_ADAPT_FAST as you know)
>
> Changing b_adapt option 2 to 1 works well.
> Otherwise sometimes it occurs same seg fault issues.
>
> I attached back trace info
> #0  0x0000000001f5a721 in x264_slicetype_frame_cost ()
> #1  0x0000000001f5c89e in x264_slicetype_analyse ()
> #2  0x0000000001ef9be6 in x264_stack_align ()
> #3  0x0000000001ef201c in x264_lookahead_slicetype_decide ()
> #4  0x0000000001ef21d9 in x264_lookahead_thread ()
> #5  0x00007ffff72a6e25 in start_thread () from /usr/lib64/libpthread.so.0
> #6  0x00007ffff660034d in clone () from /usr/lib64/libc.so.6?
> I tried to analyze x264 codes and found something weird situation.
>
>
> static int slicetype_frame_cost( x264_t *h, x264_mb_analysis_t *a,
>                                  x264_frame_t **frames, int p0, int p1, int b )
> /* For each list, check to see whether we have lowres motion-searched this reference frame before. */
>         x264_log( h, X264_LOG_INFO, "[slicetype_frame_cost] p0: %d, p1: %d, b: %d, b-p0-1: %d, p1-b-1: %d\n", p0, p1, b, b-p0-1, p1-b-1);
>         do_search[0] = b != p0 && fenc->lowres_mvs[0][b-p0-1][0][0] == 0x7FFF;
>         do_search[1] = b != p1 && fenc->lowres_mvs[1][p1-b-1][0][0] == 0x7FFF;?
> I think this problem is array index out of range exception in x264_slicetype_frame_cost function.
> Seg fault occurs when function parameters b=48, p0=44 and p1=48.
> It means [b-p0-1] is 3. But fenc->lowres_mvs[0][3] is not allocated.
>
>
>
> cur_nonb = i; // before segmentation fault, i is 47. (i = number of frames in frame buffer - 1)
>         while( IS_X264_TYPE_B( frames[cur_nonb]->i_type ) && cur_nonb > 0 )
>             cur_nonb--;?
> I understood p0 means the index of non-bframe (reverse position) in frames, macroblock_tree function.
>
> Before seg fault, frame types of frames changed 1530000000000000000000000000000000000000000000000 to 153553553553553553553553553553553553355333533555.
> So that, cur_nonb is calculated 44. After that b-p0-1 be 3.
> Except in this case, b-p0-1 and p1-b-1 are always calculated -1 to 2.
>
> When I changed frames->i_type X264_TYPE_B to X264_TYPE_P forced if greater than 2 consecutive b frames are exist, it solved.
> But I'm not sure whether can be used or not because this solution is not cool.
> (Anyway, because of this issue I changed b-adapt option 2 to 1)
>
>
> To sum up, seg fault occurs when encode some 4K contents.
> By changing x264 options, it can be avoided (especially b-adapt option).
>
> Please let me know if you need more information.
>
> Regards.
>

Hi.

Do you force frame types in you app using libx264? Because it looks
like bug is cased by 3 B-frames sequence when only 2 B-frames was
allowed (bframes=2 option).

Also can you try attached patch to see if it fix your seg fault? That
is only possible that can in theory cause such problems at 4K content.

If it doesn't help than it would be good to see h->param.i_bframe,
frm->i_type and frm->i_forced_type where `frm` is elements
of frames[0..num_frames-1].
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-possible-slicetype_path_cost-overflow-at-high-re.patch
Type: application/octet-stream
Size: 2790 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20180702/5286a324/attachment.obj>