[vlc] Re: More elegant handling of large VLM playlists
Derk-Jan Hartman
hartman at videolan.org
Tue Nov 8 16:21:08 CET 2005
On 8-nov-2005, at 5:28, Warren Young wrote:
> We use VLM to serve MPEG-1 and MPEG-2 files over RTSP. Currently,
> there are a few things that are less than ideal:
>
> 1. It takes too long to start vlc when you use the --vlm-conf flag
> to set it up to serve a bunch of MPEG files. It seems that it
> demuxes all files it will be serving. During the time this takes,
> the RTSP port is listening, but vlc is busy doing this demux loop,
> so it does not respond to streaming requests.
I'm not very knowledgeable about this area, but i'll try to explain
what i know about this.
> Why does it: a) open the RTSP port before it's ready to answer
> requests; and b) bother to demux all files in the first place?
a: I guess no one ever tested with this many files
b: To determine the streams properties (video, audio tracks, length).
MPEG PS can "insert Elementry Streams" basically anywhere in the
stream, but RTSP needs to know about ALL tracks before setting up the
stream in order to construct the SDP.
> I added some timing debug code [*] to vlc running on a test
> server. I found that it took about 200 seconds to start up when
> given about 12 GB of MPEG to manage. This comes out to 60 MB/sec,
> which is quite reasonable for a disk-bound process. (We have 4
> spindles in RAID-5 on this system.)
>
> The problem is, many of our production servers hold hundreds of GB
> of MPEGs. If we blindly assume that our production systems are as
> fast as our test system it will take about half an hour per 100 GB
> of data to start VLM! Way too slow.
Yes i can see that.
> 2. In doing the above test, I had to make the following patch to
> keep the number of messages emitted at warning level (--verbose 1)
> under control. We have some MPEG streams that vlc's demuxer
> doesn't like:
>
> --- modules/demux/ps.c.orig 2005-06-25 07:43:06.000000000 -0600
> +++ modules/demux/ps.c 2005-11-07 20:22:35.000000000 -0700
> @@ -177,7 +177,14 @@
> }
> else if( i_ret == 0 )
> {
> - msg_Warn( p_demux, "garbage at input" );
> + /* failure to resynch, so warn if this is the first
> failure in
> + this file */
> + static stream_t* last_stream = 0;
> + if (last_stream != p_demux->s) {
> + msg_Warn( p_demux, "garbage in stream '%s'",
> + p_demux->psz_path );
> + last_stream = p_demux->s;
> + }
> return 1;
> }
>
> You might consider applying this patch to vlc. There's not much
> point in emitting this message thousands of times when there's a
> problem with a stream.
mmm, i'm not sure about this one. This is a sync failure, and CAN
occur multiple times within a file. It is an error and should be
reported whenever it occurs if you ask me.
> 3. The problem I referred to above seems to be an oversensitivity
> in the PS demuxer as to what it considers "garbage". I have
> streams here that will play quite well with all of the many
> hardware and software MPEG decoders we have (including VLC itself),
> if you play the file directly. But if you serve these files through
> RTSP with VLM, various problems happen -- for example, missing
> video, or MPEG artifacts that don't appear when playing the file
> directly. The symptoms vary by RTSP client (we've tried QuickTime
> and VLC for Windows) and by stream, but what doesn't vary is that
> the problem streams fail on both players, while all other streams
> play with both players. This plus the strong correlation with "PS
> stream garbage" warnings from VLM suggest that VLM is where the
> problem lies.
I think the problem here is that because of the loading of all the
files in the VLM, the synchronization code becomes overloaded,
causing data to be late and requiring the server to resync the
stream. If the load on the server is low, you should not run into
this if you don't run into it during normal VLC playback i think.
Is all your MPEG data in true PS?
You might consider streaming in muxed form, with VLM instead of in
seperate tracks. Then less pre-demuxing might be required, but i'm
not sure, i haven't tested that.
> I've been able to fix two such streams by remuxing them. The new
> streams do not trip the vlc demux garbage detector, and they stream
> over RTSP correctly. So, we could conceivably just remux all of
> the problem streams, but it would be better if we could make vlc as
> robust in this regard as all the other MPEG decoders we've tested.
> It's got to be some weakness in the demuxer.
No, the problem is that a lot of MPEG data out there is as invalid
and broken as hell. Try passing your files trough ProjectX analyzer
and you will see that these streams have issues. The fact that most
players don't care about this as much as VLC has also resulted in the
fact that there are so many of those files out there. VLC is very
peculiar about PTS/PCR for instance. It needs to be like this,
because that's what makes VLC a good MPEG streamer. it actually uses
the information of the MPEG files to do MPEG streaming in the way
MPEG streaming was designed.
>
> I may be able to help with fixing some of these things. This part
> of VLC is just barely within my expertise: I can handle MPEG code
> down to the stream level. (I have a copy of ISO 13818-1, plus
> several MPEG books.) Beyond that, I'm useless. If someone more
> clueful wants to tackle these issues, that'd be great. I can
> supply test clips and such if you need them. Otherwise, I'd
> appreciate some guidance on where to start hacking within the code.
I think most is just a matter of VLM optimization. For instance the
pre-demux on load might be something that could be cached in one way
or another. This is what most servers do as well. When they add the
file to the "database" they analyse it and extract the required
information for the SDP/RTSP session and cache it. VLC currently is
unable to do so.
> [*] My timing test patch, in case anyone's interested:
>
> --- src/misc/vlm.c.orig 2005-06-25 07:43:02.000000000 -0600
> +++ src/misc/vlm.c 2005-11-07 20:40:53.000000000 -0700
> @@ -116,8 +116,10 @@
> {
> vlm_message_t *p_message = NULL;
> char *psz_buffer = NULL;
> + time_t start = time(0);
>
> - msg_Dbg( p_this, "loading vlm conf ..." );
> + msg_Warn( p_this, "loading vlm conf at %s...",
> + asctime(localtime(&start)));
> asprintf(&psz_buffer, "load %s", psz_vlmconf );
> if( psz_buffer )
> {
> @@ -128,6 +130,8 @@
> free(p_message);
> free(psz_buffer);
> }
> + msg_Warn( p_this, "...finished loading vlm conf, %d
> seconds.",
> + time(0) - start);
> }
> free(psz_vlmconf);
>
> --
> This is the vlc mailing-list, see http://www.videolan.org/vlc/
> To unsubscribe, please read http://www.videolan.org/support/lists.html
>
>
--
This is the vlc mailing-list, see http://www.videolan.org/vlc/
To unsubscribe, please read http://www.videolan.org/support/lists.html
More information about the vlc
mailing list