[x264-devel] Yet another updated threaded slicetype patch (v14)

[Steven Walters]

>
> Your theory makes sense, but there are hard bounds checking in slicetype.c:
>
>    for( j = 0; h->lookahead->next.list[j] && h->lookahead->next.list[j]->i_type == X264_TYPE_AUTO && j <= X264_BFRAME_MAX*4+2; j++ )
>
> and frames is inited with:
>
>   x264_frame_t *frames[X264_BFRAME_MAX*4+3] = { NULL, };
>
> So we are sure we will never exceed our array size.
>

X264_FRAME_MAX * 4 + 3 evaluates out to 67. which would be index point 66.
frame_cost is called b=66 before jumping to a call with b=68,
so some logic to prevent going beyond the index 66 is apparently failing.

> Just to confirm, you get the same crash above at the same settings with this patch AND lookahead==0?
>
blah, human error on my side, on lookahead 0 it actually only gets as
high as b=62 on all the settings i tried now,
so i must of increased it accidentally on a prior test when i changed
some other parameters.
--lookahead 0 is fine.

attached is a zip compressed display of gdb's dump that i piped to a file.
i added some printfs to the code so i could narrow things down previously by
printing out the current p0,p1,b values of frame_cost and the maximum
seem values
and frames[b] to make sure it's pointing to 'real' space.
be careful it extracts out to about 4 MBs of text.

> "thread apply all bt" doesn't work?
> I don't have Windows running to check if that works or not, but I think it should.
this apparently works and is included in the above output.
(despite the output looking fairly useless to even me)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out.zip
Type: application/zip
Size: 91843 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/x264-devel/attachments/20090414/e0dbd60a/attachment-0001.zip>