[x264-devel] Behaviour of Annex B encoding: bug or not?
Philip Spencer
pspencer at fields.utoronto.ca
Thu Jan 7 20:08:52 CET 2010
> You're reading it wrong then. Escaping (by inserting emulation
> prevention bytes) is defined for NAL units in section 7.3.1. Annex B
> only defines further encapsulation of these bits but doesn't change the
> actual representation. /Gil
Thanks for the clarification!
If it's defined in the spec itself, not in Annex B, that means my initial
suspicion and attempted patch were wrong. Obviously the interoperability
problems we're having are due to either
(1) A buggy brand of hardware
(2) A mass misunderstanding of the spec in the marketplace or
(3) Hardware which somehow thinks the software program I'm using to
talk to it is broken and drops emulation prevention bytes because it
thinks the software can't handle it.
I originally thought (1) was unlikely, since I know the unit interoperates
well with a wide range of other hardware models, and a unit which doesn't
escape would be incompatible with one that does.
However, on examining the traffic actually GENERATED by this hardware
unit, I found no packets where escaping was ever needed, so that makes it
quite likely the unit is buggy and nobody's ever noticed before because
the traffic patterns don't usually need escaping.
It's only because x264 generates more complete session information, which
includes several near-zero multi-byte integer parameters that result in
escape-required patterns of traffic, that we are seeing the problem.
It doesn't occur when the units talk to each other.
I guess my approach should be not to alter x264 (since it's doing the
right thing) but rather to add a flag in the ekiga/opal software so that,
if it's dealing with buggy hardware, it calls the x264 library as usual to
generate the NALs but then strips out the emulation prevention bytes
before putting the packet on the wire.
If I get time I'll see if I can find other RTP streams in the wild, from
other hardware or software, and see whether or not this misreading of the
spec is widespread or limited to our hardware and the wireshark network
analyzer. If I find it's widespread I'll post back here again to see if
you all think it's widespread enough to warrant adding a flag to request
that x264 not do escaping.
I'll also post to the wireshark people so they can fix their analyzer. As
for our LifeSize Room hardware unit, I suspect getting the manufacturer to
fix their product is unlikely as they won't want to risk breaking
interoperability with their older units, but I'll try anyway.
Thanks everyone.
- Philip
On Wed, 6 Jan 2010, Gil Pedersen wrote:
> On 06/01/2010, at 01.36, Alex Giladi wrote:
>
>> Gil,
>> Quoting section 3 of the RFC: "This payload specification can only be
>> used to carry the "naked" H.264 NAL unit stream over RTP, and not the
>> bitstream format discussed in Annex B of H.264". Escaping is defined
>> in Annex B (section B.1).
>> Haven't played with this (I work with plain old MPEG-2 TS), but this
>> is my interpretation of the specs.
>
> You're reading it wrong then. Escaping (by inserting emulation
> prevention bytes) is defined for NAL units in section 7.3.1. Annex B
> only defines further encapsulation of these bits but doesn't change the
> actual representation. /Gil
>
>> Also, there is no "de-facto standard": there is only compliance or
>> lack of compliance.
>> Alex.
>>
>> On Tue, Jan 5, 2010 at 5:44 PM, Gil Pedersen <gil at cmi.aau.dk> wrote:
>>> On 05/01/2010, at 20.49, Alex Giladi wrote:
>>>
>>>> RTP payload (at RFC 3984) uses RBSP. Maybe it's worth adding "rtp" as
>>>> an output option?
>>>> --ag
>>>
>>> Is this true? I see no mention of RBSP in the RFC. The RFC 3984 payload is based on NAL units, which according to the H.264 spec consists of a 1-byte header followed by RBSP _and_ the necessary emulation bytes. Further, as mentioned, the H.264 reference encoder seems to output RTP packets with the emulation bytes.
>>>
>>> As far as I can tell any decoder that expects raw RBSP + 1 byte header is broken and x264 should never allow this to be generated unless it can be proven that it's the de-facto standard.
>>>
>>> /Gil
>>>
>>>> On Tue, Jan 5, 2010 at 2:04 PM, Jason Garrett-Glaser
>>>> <darkshikari at gmail.com> wrote:
>>>>> On Tue, Jan 5, 2010 at 1:41 PM, Philip Spencer
>>>>> <pspencer at fields.utoronto.ca> wrote:
>>>>>> First a disclaimer: I know very little about the H.264 spec, and am just
>>>>>> trying to resolve an interoperability issue we are having, so please bear
>>>>>> with me if I misstate or misunderstand something.
>>>>>>
>>>>>> In the routine x264_nal_encode (in common/common.c), the flag b_annexb
>>>>>> controls whether or not to add a NAL start code (00 00 00 01) to the
>>>>>> beginning of the packet, but does not control whether or not to do the
>>>>>> escaping of sequences of the form 00 00 00/1/2/3 by inserting a 03 byte into
>>>>>> the third position -- that escaping is ALWAYS done, even if b_annexb is not
>>>>>> set.
>>>>>>
>>>>>> Is this a bug, or is it meant to be that way? (I don't have access to the
>>>>>> text of Annex B).
>>>>>
>>>>> We didn't make it an option because we didn't know of any devices that
>>>>> expected non-escaped bytestreams. All containers we knew that didn't
>>>>> want Annex-B startcodes still expected escaped NAL units.
>>>>>
>>>>>> It certainly breaks interoperability with several devices. In particular,
>>>>>> x264 cannot be used with the Ekiga softphone application to communicate with
>>>>>> the "LifeSize Room" brand of videoconferencing equipment: that device
>>>>>> expects non-Annex-B packets over RTP, and cannot handle the extra 03 byte
>>>>>> that is inserted. The result is that the session parameters (like stream
>>>>>> resolution and other such settings, which often contain multiple zero bytes
>>>>>> in a row) get completely garbled because of the Annex-B bytestream encoding.
>>>>>>
>>>>>> Also, if one attempts to sniff the network traffic with software such as
>>>>>> WireShark, it too chokes on the unexpected 03 bytes in RTP packets.
>>>>>>
>>>>>> It would seem to me that this is a bug: if Annex B bytestream encoding is
>>>>>> not desired, such as for an RTP packet, then the extra escape bytes should
>>>>>> not be inserted.
>>>>>>
>>>>>> On our system, I have applied the patch below to common.c and then
>>>>>> H.264 connectivity to the LifeSize Room videoconferencing equipment works
>>>>>> just fine.
>>>>>>
>>>>>> On the other hand, from a quick glance at the source code of the reference
>>>>>> encoder/decoder, it seems that it behaves the same way as x264: always
>>>>>> inserts the escape byte. Is this a bug in the reference encoder/decoder too,
>>>>>> or does the text of Annex B specify that ALL H.264 streams should have the
>>>>>> extra bytes inserted, even when bytestream encoding is not being used?
>>>>>>
>>>>>> In the latter case, then obviously the LifeSize brand videoconference units
>>>>>> are buggy, but since I know they interperate will over H.264 with a wide
>>>>>> range of units from other manufacturers there must be a lot of buggy devices
>>>>>> out there -- would it be worth adding an extra flag to x264 that says "do
>>>>>> bytestream encoding only in Annex B mode, for compatibility with
>>>>>> devices that cannot handle it in RTP mode"?
>>>>>
>>>>> The best solution here is probably to add another parameter to control
>>>>> the escaping. b_escaped_nals or similar. In the case of
>>>>> b_escaped_nals == 0, a straight memcpy would be used.
>>>>>
>>>>> Dark Shikari
>>>>> _______________________________________________
>>>>> x264-devel mailing list
>>>>> x264-devel at videolan.org
>>>>> http://mailman.videolan.org/listinfo/x264-devel
>>>>>
>>>> _______________________________________________
>>>> x264-devel mailing list
>>>> x264-devel at videolan.org
>>>> http://mailman.videolan.org/listinfo/x264-devel
>>>
>>> _______________________________________________
>>> x264-devel mailing list
>>> x264-devel at videolan.org
>>> http://mailman.videolan.org/listinfo/x264-devel
>>>
>> _______________________________________________
>> x264-devel mailing list
>> x264-devel at videolan.org
>> http://mailman.videolan.org/listinfo/x264-devel
>
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel
>
--------------------------------------------+-------------------------------
Philip Spencer pspencer at fields.utoronto.ca | Director of Computing Services
Room 336 (416)-348-9710 ext3036 | The Fields Institute for
222 College St, Toronto ON M5T 3J1 Canada | Research in Mathematical Sciences
More information about the x264-devel
mailing list