[vlc-devel] [PATCH v3] omxil: Feed H264 in annex b format

[Laurent Aimar]

Hi,

> @@ -1155,6 +1187,24 @@ static picture_t *DecodeVideo( decoder_t *p_dec, block_t **pp_block )
>              block_Release(p_block);
>          }
>  
> +        /* Convert H.264 NAL format to annex b */
> +        if( p_sys->i_nal_size_length )
> +        {
> +            /* This only works for NAL sizes 3-4 */
> +            int i_len = p_header->nFilledLen, i;
> +            uint8_t* ptr = p_header->pBuffer;
> +            while( i_len >= p_sys->i_nal_size_length )
> +            {
> +                int nal_len = 0;
> +                for( i = 0; i < p_sys->i_nal_size_length; i++ ) {
> +                    nal_len = (nal_len << 8) | ptr[i];
> +                    ptr[i] = 0;
> +                }
nal_len can overflow when p_sys->i_nal_size_length >= 4.

> +                ptr[p_sys->i_nal_size_length - 1] = 1;
> +                ptr   += nal_len + 4;
> +                i_len -= nal_len + 4;
and that will badly corrupt the value of i_len.

-- 
fenrir