[vlc-devel] [HELP] Tracking down an audio transcoding segfault with the mpgatofixed32 filter

[Antoine Cellerier]

On Fri, Jun 20, 2008, Kevin DuBois wrote:
>    I have fixed the problem. The patch is attached. The segfault was caused
>    by a too small conversion buffer being specified in mpgatofixed32 due to
>    an incorrect value in p_filter->fmt_out.audio.i_bitspersample. Namely, if
>    mpga->s16l is requested, , a filter that is mgpa to fl32 is returned, and
>    transcode is aware of that the output is fl32, although it requested
>    s16l.  This is the reason for the "FIXME FIXME ..." in transcode.c:1212,
>    that code will account for a filter returning a format that it hasnt
>    requested. This also caused a segfault in mpgatofixed32 because it relies
>    on p_filter->fmt_out.audio.i_bitspersample being correct when it allocates
>    its buffers. p_filter->fmt_out.audio.i_bitspersample incorrectly held the
>    value 16, which is only half the buffer size requested.
> 
>     I modified the  filters chain to account for this difference. This is
>    pretty much the exact same thing the old filters code did. My next step is
>    to revamp the filters/transcode code so that any number of audio filters
>    can be requested, and the filter chain will intelligently hook them up
>    together, and account for any format differences that arise.

Thanks for the patch. I'll apply it right away.

-- 
Antoine Cellerier
dionoea