[x264-devel] [patch] Segfault with sliced-threads and height less than 50

[Justin Greer]

Hey all, Justin from Zencoder here...

We noticed an issue on files that have the zerolatency tuning applied 
and which are very short.  I narrowed it down to a bug in 
x264_validate_parameters which can set the number of i_threads to zero 
if the height is less than 50, here:

http://git.videolan.org/?p=x264.git;a=blob;f=encoder/encoder.c;h=b1a99e6ece168967be7d55f849602a0e63186f6c;hb=0c7dab9c2a106ce3ee5d6ad7282afb49e1cc3954#l486

That means the initialization of p_bitstream never happens, since this 
loop doesn't run:

http://git.videolan.org/?p=x264.git;a=blob;f=encoder/encoder.c;h=b1a99e6ece168967be7d55f849602a0e63186f6c;hb=0c7dab9c2a106ce3ee5d6ad7282afb49e1cc3954#l1255

And then when encode() calls x264_encoder_headers(), and it calls 
bs_init(), p_data doesn't exist so s->p gets set to zero and then 
accessing it to get cur_bits triggers the segfault:

http://git.videolan.org/?p=x264.git;a=blob;f=common/bitstream.h;h=1a153380c99c3031cb054e2473d1039a59f538c6;hb=0c7dab9c2a106ce3ee5d6ad7282afb49e1cc3954#l85

Looks like this simple patch (attached) should fix it, setting a lower 
bound on i_threads of 1.  Two test cases are also attached -- just basic 
y4m header files (one at height 50, the other at height 48) that are 
enough to display the bug.

The test case run logs are below.  Let me know if you need any more info.

-- Justin


Running the test cases:

$ ./x264 --sliced-threads -o video.mp4 /tmp/320x50.y4m

y4m [info]: 320x50p 1:1 @ 30/1 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 1.1

$ ./x264 --sliced-threads -o video.mp4 /tmp/320x48.y4m

y4m [info]: 320x48p 1:1 @ 30/1 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 1.1
Segmentation fault: 11


AFTER PATCH:

$ ./x264 --sliced-threads -o video.mp4 /tmp/320x48.y4m
y4m [info]: 320x48p 1:1 @ 30/1 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 1.1


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: x264_min_threads_patch.dat
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20120106/de91755b/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 320x48.y4m
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20120106/de91755b/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 320x50.y4m
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20120106/de91755b/attachment-0002.ksh>