[x264-devel] Opaque pointer for nalu processing?

[Jason Garrett-Glaser]

On Wed, Aug 8, 2012 at 3:40 AM, Killian De Smedt <killiands at gmail.com> wrote:
>
>
> On 8 August 2012 11:52, Jason Garrett-Glaser <jason at x264.com> wrote:
>>
>> > Now I am trying to do the reordering of the nalu's, I assume this is
>> > necessary (annex b used)?
>>
>> Technically yes, but some decoders might work anyways.  I don't think
>> libavcodec does, but it shouldn't be horribly difficult to improve it
>> to support that.
>>
>> Even if you don't reorder them server-side, for example, you could
>> send out the NALUs one by one, then reorder them on the client and
>> send the whole frame to decode at once. Some example guessy numbers:
>>
>> Example without nalu_process: 20ms encode, 30ms network send, 5ms decode
>> (55ms)
>> Example with nalu_process: 2ms encode before NALUs are output, 30ms
>> network send, 5ms decode (37ms)
>> Example with nalu_process, no reordering: 2ms encode before NALUs are
>> output, 30ms network send (decode done at the same time) 0.5ms decode
>> finish after network send (32.5ms)
>
>
> My original idea was to send out nalu's unordered and reorder them on the
> client before decoding, but it would be even better if the decoder could
> manage that for me, I might look into libavcodec for that, thank you.
>>
>>
>> > Currently (for debugging) I am doing this in the
>> > same app and write the final stream to a file that i try to play with
>> > the
>> > totem movie player. This works fine for the encoder_encode version.
>> >
>> > My current algorithm is as follows:
>> > if type is not 1,2,3,4,5 (slices) : write to file immediately, I assume
>> > headers are sent synchronously in-order before other nalu's, otherwise
>> > this
>> > could not work, because the mb_start/finish values make no sense for
>> > them.
>> > Otherwise, buffer until all chunks are received, order by i_mb_start and
>> > write away to file in that order.
>>
>> Are you sure all the slices are for the same frame?  If you end up
>> with slices from two different frames, this could happen.
>>
>
> Yes, but I just found my (quite stupid) error, I created a buffer of the
> recommended size in x264.h (nal->i_payload*3/2 + 5 + 16.), but forget to
> update the size I pass to my serializer after the nal_encode, so I ended up
> writing a lot of rubbish padding. I now send the nal->i_payload (after
> encoding) to my serializer and everything works as expected.
>
> I had one last question. this is a low latency project for
> computer-generated frames, and it could be that after a frame gets encoded
> it takes some time before I receive another frame. Therefore it is essential
> for me to know (asap) when a frame is fully encoded/received without having
> to wait on another frame.
>
> Currently I still look at the nals returned by encoder_encode and use the
> largest last_mb value from that to signal what is the last slice of my
> frame. This works for now, but is problematic for two reasons, it does not
> really follow x264 rules (nals could be invalid when using nalu_process) and
> it introduces a further useless message. Is there a way I could know if a
> specific nalu is the last slice in a frame? It is certainly not direct
> present in the nal_t, but could I possibly derive it from the last_mb value
> or the startcode?

What about just looking for when x264_encoder_encode returns?

Jason