[vlc-devel] Problems with synchronizing video and audio using RTP (live555)

[Jami Pekkanen]

On Wed, 2009-06-17 at 17:58 +0300, Jami Pekkanen wrote:
> Hello,
> 
> I have a setup with a homebrew RTP client sending audio (AMR) and video
> (MP4-ES) stream and a VLC (current trunk and live555 from yesterday)
> instance playing it using a SDP file to initialize the playing (ie $ vlc
> stream.sdp). When playing the streams separately, it works fine, but
> when playing them at the same time, the synchronization fails (constant
> "PTS is out of range" for audio). This leads to video playing fine, but
> with audio mostly missing. It does work if the playing starts from the
> start of the stream, but if too much frames are dropped, this falls out
> of sync also.

I think I found something on this. I'm sending the stream without RTCP
nor RTSP. Live555 doesn't want to calculate timings if it doesn't get
RTP-Info from RTSP, so I hacked the attached npthack.patch to live555.
It's probably quite ugly and I have no idea about how it fits in to
bigger picture, but it seems to give correct timings for the frames.
Even without the patch, VLC also seems to be able to handle
getNormalPlaytime returning 0.0 for all of the frames, but it doesn't
help with the behavior I'm now describing.

When I played the stream with that patch applied, I noticed that it
seems to read only one audio buffer for every video buffer although I'm
sending about three audio buffers for each video buffer. Thus about 2/3
of the audio is left in to (network?) stack and the audio drifts out of
sync very fast. I've attached a log[1] that shows this behavior.

As far as I understand, live555.cpp:Demux seems to behave so that for
each Demux call it retrieves one buffer for each substream. Should this
be fixed?

To reproduce the behavior, download input.1, input.2, input.sdp and
repro.sh, or repro.tar.gz which contains them all from 
http://jami.gotdns.org/amrrtp/ and run repro.sh. You'll need rtptools
from http://www.cs.columbia.edu/irt/software/rtptools/ in your path.

- Jami Pekkanen

[1] http://jami.gotdns.org/amrrtp/output.log
-------------- next part --------------
A non-text attachment was scrubbed...
Name: npthack.patch
Type: text/x-patch
Size: 1146 bytes
Desc: not available
URL: <http://mailman.videolan.org/pipermail/vlc-devel/attachments/20090622/7025de19/attachment.bin>