[vlc-devel] [PATCH v3] omxil: Feed H264 in annex b format

[Martin Storsjö]

On Wed, 28 Sep 2011, Laurent Aimar wrote:

> Hi,
>
>> @@ -1155,6 +1187,24 @@ static picture_t *DecodeVideo( decoder_t *p_dec, block_t **pp_block )
>>              block_Release(p_block);
>>          }
>>
>> +        /* Convert H.264 NAL format to annex b */
>> +        if( p_sys->i_nal_size_length )
>> +        {
>> +            /* This only works for NAL sizes 3-4 */
>> +            int i_len = p_header->nFilledLen, i;
>> +            uint8_t* ptr = p_header->pBuffer;
>> +            while( i_len >= p_sys->i_nal_size_length )
>> +            {
>> +                int nal_len = 0;
>> +                for( i = 0; i < p_sys->i_nal_size_length; i++ ) {
>> +                    nal_len = (nal_len << 8) | ptr[i];
>> +                    ptr[i] = 0;
>> +                }
> nal_len can overflow when p_sys->i_nal_size_length >= 4.

Hmm, yes. So one could change nal_len to uint32_t, and only do this for 0 
< i_nal_size_length <= 4. And once the nal_len is parsed, break if it is 
larger than some sanity limit (INT_MAX, p_header->pFilledLen, or something 
else)?

// Martin