[vlc-devel] a few questions about module behaviour

[Tomas Carnecky]

I'm getting nowhere in the IRC channels, nobody's helping me, and if
they try, they know absolutely nothing about how vlc works or how to
write modules.

Basically, I want to write a simple access/demux module. This module
should produce a perfectly fine video and audio elementary stream as
long as it's loaded, the idea is that the user starts vlc using that
module, creates a sout and leaves it running for a potentially long time.

The input data is a bytestream with a very simple payload format, very
easy to parse. The packet payload contains either raw I420 frames or
sound samples, each with an absolute timestamp (usec resolution) when it
was captured. It is very important to understand that there can be (and
will be) gaps between two video frames or two packets that carry sound
samples, so somebody has to pay attention to the timing information and
duplicate/drop frames/samples if necessary.

The video resolution or sound format of the input stream can change at
any time. I have taken care of the video problem by making the output ES
fixed size and allocating a canvas of that size. When an input frame
comes in, it is copied into the canvas, and than the canvas is
duplicated into the output ES. That works reasonably well, I'll have to
figure out something for the changing sound format (but that's less of
an issue right now).

The biggest problem is getting the timing right. vlc prints out cryptic
messages and I have absolutely no idea what they mean or what vlc is
trying to tell me. As mentioned earlier, each packet in the input stream
has a timestamp, it really is gettimeofday() at the point when the
packet was created (the video frame captured). It would seem easy to
save the timestamp of the first packet and later calculate the diff
between the first packet's and current packet's timestamps to get PTS of
the block that needs to be passed to the output ES. It would seem ...

In Open() I create a thread that reads from the network socket, chops
the data up into packets and passes those to a fifo. Demux() then takes
the packets from the fifo, duplicates/drops data if necessary and passes
it to the output ES.

Demux() works like this (for video frames):

if (packets-in-fifo) {
    packet = next-packet-from-fifo()
    pts = mdate-of-first-packet + (packet->time - first_packet->time)
    /* now we know what PTS (in vlc's scale) this frame has.
       If there's a gap between the last frame that was put into
       the output ES and 'pts', we need to duplicate frames */
    while (date_Get(video.pts) < pts) {
        copy = block_Duplicate(canvas)
        copy->pts = date_Increment(video.pts, 1)
        es_out_send(copy)
    }
    merge-packet-into-canvas()
} else {
    /* no packets in fifo, duplicate canvas */
    now = mdate()
    while (date_Get(video.pts) < now) {
        copy = block_Duplicate(canvas)
        copy->pts = date_Increment(video.pts, 1)
        es_out_send(copy)
    }
}

It sort-of works if I send out only video frames, but once I add audio
vlc goes crazy and complains about bad timing, like:
[...] audio drift is too big (236985), dropping buffer
[...] late picture skipped (2240)

Once thing that really amazed me is when I changed Demux() to only do:
  copy = block_Duplicate(canvas)
  copy->pts = date_Increment(video.pts, 1)
  es_out_send(copy)
  return 1;
I still saw 'late picture skipped' messages. I don't understand, I can
hardly push out frames faster than that, or is there a scheduling
problem somewhere?

Does anyone have any suggestions what could be wrong with that code? Or
how to get the timing right? I tried asking in the IRC channel, but
nobody was able to answer me even an easy question like 'does vlc
automatically duplicate/drop frames?'. So I try it here, maybe I'll have
more luck.

My code has lots of msg_Dbg() in it so I won't post it here, but if you
really want I guess I could upload it somewhere.

thanks
tom