[x265] x265 crashes/gets stuck when giving more than '--slices 16'

[Michael Lackner]

Greetings,

During my experimentation with creating a x265-based benchmark, I found out that x265
segfaults and/or freezes when giving it more than 16 frame slices to encode (e.g. --slices
17 or --slices 20 or --slices 32 or similar).

It seems easy to reproduce as well. In my case, I'm feeding raw 16-bit YUV 4:4:4 content
to it (16-bit per channel, 48-bit per pixel, no chroma subsampling). The resolution is
8192×3428, an upscale of the free movie "Tears of Steel".

It works perfectly fine with <=16 slices on all my systems though, here's the ones I've
tested it on:

Environments / software versions:

  * x265 2.2+22-20217c8af8ac

     CentOS 6.8 Linux x86_64, x265 built by GCC 6.2.0 + yasm 1.3.0
      => Segmentation fault

  * x265 2.3+9-820f4327ddac

    o CentOS 6.8 Linux x86_64, x265 built by GCC 6.2.0 + yasm 1.3.0
      => Segmentation fault

    o FreeBSD 10.3 UNIX x86_64, x265 built by clang 3.8.1 + yasm 1.3.0
      => Process locks up in STOP state and becomes unkillable, even
         to SIGKILL

    o Windows XP Professional x64 Edition (NT5.2), x265 built by
      MSVC2010 SP1, cl.exe 16.00.40219.01 + yasm 1.3.0
      => Terminates with %ERRORLEVEL% still set to 0, so this crash is
      completely uncaught

As said, it works fine on all platforms with --slices [1..16].

Here's some sample output of the crash from Linux, it's supposed to be a 2-pass encode:

29163 Segmentation fault      (core dumped) nice -n19 x265 ./raw8k.yuv --frames 500
--input-depth 16 --dither --input-res 8192x3428 --input-csp i444 -D 10 --fps 24 --slices
20 -p veryslow --pmode --pme --open-gop --ref 6 --bframes 16 --b-pyramid --weightb
--max-merge 5 --b-intra --bitrate 50000 --rect --amp --aq-mode 2 --no-sao --qcomp 0.75
--no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4
--tu-intra-depth 4 --ctu 32 --max-tu-size 32 --pass 1 --slow-firstpass --stats v.stats
--sar 1 --range full -o pass1.h265

I don't have any debug builds of x265 right now and I don't really know how to even do any
debugging, but if you can tell me if you need anything, I can always try to build a debug
version to generate more helpful output / dump files.

Or maybe 16 frame slices are supposed to be the maximum, but it's just not handled
correctly yet?!

Is there any information available on that behavior?

-- 
Michael Lackner
Lehrstuhl für Informationstechnologie (CiT)
Montanuniversität Leoben
Tel.: +43 (0)3842/402-1505 | Mail: michael.lackner at unileoben.ac.at
Fax.: +43 (0)3842/402-1502 | Web: http://institute.unileoben.ac.at/infotech