[vlc-devel] VLC avcodec.c InitAudioDec function failed will lead extradata be freed two times

[Ximing Cheng]

I found a crash when VLC  avcodec.c (/vlc/modules/codec/avcodec/avcodec.c)
call

*static int OpenDecoder( vlc_object_t *p_this )*

OpenDecoder will call:

*ret = InitAudioDec( p_dec, avctx, p_codec );*

InitAudioDec will call

*if( OpenAudioCodec( p_dec ) < 0 )*
*{*
*    av_free( p_context->extradata );*
*    free( p_sys );*
*    return VLC_EGENERIC;*
*}*

if OpenAudioCodec failed VLC will free the buffer of p_context->extradata,
and after av_free the p_context->extradata is invalid but the pointer is
not NULL

but in OpenDecoder function if InitAudioDec failed VLC will call this:

*    switch( i_cat )*
*    {*
*        case VIDEO_ES:*
*            ret = InitVideoDec( p_dec, avctx, p_codec );*
*            break;*
*        case AUDIO_ES:*
*            ret = InitAudioDec( p_dec, avctx, p_codec );*
*            break;*
*        case SPU_ES:*
*            ret = InitSubtitleDec( p_dec, avctx, p_codec );*
*            break;*
*        default:*
*            ret = VLC_EGENERIC;*
*    }*

*    if( ret != VLC_SUCCESS )*
*    {*
*        avcodec_free_context( &avctx );*
*        return ret;*
*    }*

in avcodec_free_context will call this:

*av_freep(&avctx->extradata);*

in av_freep the extradata is not NULL, but the pointer is invalid.
so, if audio decoder init failed VLC will free extradata two times, and
that will cause a crash in sometime.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/vlc-devel/attachments/20150317/88dd7885/attachment.html>