[x264-devel] Behaviour of Annex B encoding: bug or not?

[Jason Garrett-Glaser]

On Tue, Jan 5, 2010 at 1:41 PM, Philip Spencer
<pspencer at fields.utoronto.ca> wrote:
> First a disclaimer: I know very little about the H.264 spec, and am just
> trying to resolve an interoperability issue we are having, so please bear
> with me if I misstate or misunderstand something.
>
> In the routine x264_nal_encode (in common/common.c), the flag b_annexb
> controls whether or not to add a NAL start code (00 00 00 01) to the
> beginning of the packet, but does not control whether or not to do the
> escaping of sequences of the form 00 00 00/1/2/3 by inserting a 03 byte into
> the third position -- that escaping is ALWAYS done, even if b_annexb is not
> set.
>
> Is this a bug, or is it meant to be that way? (I don't have access to the
> text of Annex B).

We didn't make it an option because we didn't know of any devices that
expected non-escaped bytestreams.  All containers we knew that didn't
want Annex-B startcodes still expected escaped NAL units.

> It certainly breaks interoperability with several devices. In particular,
> x264 cannot be used with the Ekiga softphone application to communicate with
> the "LifeSize Room" brand of videoconferencing equipment: that device
> expects non-Annex-B packets over RTP, and cannot handle the extra 03 byte
> that is inserted. The result is that the session parameters (like stream
> resolution and other such settings, which often contain multiple zero bytes
> in a row) get completely garbled because of the Annex-B bytestream encoding.
>
> Also, if one attempts to sniff the network traffic with software such as
> WireShark, it too chokes on the unexpected 03 bytes in RTP packets.
>
> It would seem to me that this is a bug: if Annex B bytestream encoding is
> not desired, such as for an RTP packet, then the extra escape bytes should
> not be inserted.
>
> On our system, I have applied the patch below to common.c and then
> H.264 connectivity to the LifeSize Room videoconferencing equipment works
> just fine.
>
> On the other hand, from a quick glance at the source code of the reference
> encoder/decoder, it seems that it behaves the same way as x264: always
> inserts the escape byte. Is this a bug in the reference encoder/decoder too,
> or does the text of Annex B specify that ALL H.264 streams should have the
> extra bytes inserted, even when bytestream encoding is not being used?
>
> In the latter case, then obviously the LifeSize brand videoconference units
> are buggy, but since I know they interperate will over H.264 with a wide
> range of units from other manufacturers there must be a lot of buggy devices
> out there -- would it be worth adding an extra flag to x264 that says "do
> bytestream encoding only in Annex B mode, for compatibility with
> devices that cannot handle it in RTP mode"?

The best solution here is probably to add another parameter to control
the escaping.  b_escaped_nals or similar.  In the case of
b_escaped_nals == 0, a straight memcpy would be used.

Dark Shikari