[vlc] More elegant handling of large VLM playlists

[Warren Young]

We use VLM to serve MPEG-1 and MPEG-2 files over RTSP.  Currently, there 
are a few things that are less than ideal:


1. It takes too long to start vlc when you use the --vlm-conf flag to 
set it up to serve a bunch of MPEG files.  It seems that it demuxes all 
files it will be serving.  During the time this takes, the RTSP port is 
listening, but vlc is busy doing this demux loop, so it does not respond 
to streaming requests.

Why does it: a) open the RTSP port before it's ready to answer requests; 
and b) bother to demux all files in the first place?

I added some timing debug code [*] to vlc running on a test server.  I 
found that it took about 200 seconds to start up when given about 12 GB 
of MPEG to manage.  This comes out to 60 MB/sec, which is quite 
reasonable for a disk-bound process.  (We have 4 spindles in RAID-5 on 
this system.)

The problem is, many of our production servers hold hundreds of GB of 
MPEGs.  If we blindly assume that our production systems are as fast as 
our test system it will take about half an hour per 100 GB of data to 
start VLM!  Way too slow.


2. In doing the above test, I had to make the following patch to keep 
the number of messages emitted at warning level (--verbose 1) under 
control.  We have some MPEG streams that vlc's demuxer doesn't like:

--- modules/demux/ps.c.orig     2005-06-25 07:43:06.000000000 -0600
+++ modules/demux/ps.c  2005-11-07 20:22:35.000000000 -0700
@@ -177,7 +177,14 @@
      }
      else if( i_ret == 0 )
      {
-        msg_Warn( p_demux, "garbage at input" );
+        /* failure to resynch, so warn if this is the first failure in
+           this file */
+        static stream_t* last_stream = 0;
+        if (last_stream != p_demux->s) {
+            msg_Warn( p_demux, "garbage in stream '%s'",
+                    p_demux->psz_path );
+            last_stream = p_demux->s;
+        }
          return 1;
      }

You might consider applying this patch to vlc.  There's not much point 
in emitting this message thousands of times when there's a problem with 
a stream.


3. The problem I referred to above seems to be an oversensitivity in the 
PS demuxer as to what it considers "garbage".  I have streams here that 
will play quite well with all of the many hardware and software MPEG 
decoders we have (including VLC itself), if you play the file directly. 
  But if you serve these files through RTSP with VLM, various problems 
happen -- for example, missing video, or MPEG artifacts that don't 
appear when playing the file directly.  The symptoms vary by RTSP client 
(we've tried QuickTime and VLC for Windows) and by stream, but what 
doesn't vary is that the problem streams fail on both players, while all 
other streams play with both players.  This plus the strong correlation 
with "PS stream garbage" warnings from VLM suggest that VLM is where the 
problem lies.

I've been able to fix two such streams by remuxing them.  The new 
streams do not trip the vlc demux garbage detector, and they stream over 
RTSP correctly.  So, we could conceivably just remux all of the problem 
streams, but it would be better if we could make vlc as robust in this 
regard as all the other MPEG decoders we've tested.  It's got to be some 
weakness in the demuxer.


I may be able to help with fixing some of these things.  This part of 
VLC is just barely within my expertise: I can handle MPEG code down to 
the stream level.  (I have a copy of ISO 13818-1, plus several MPEG 
books.)  Beyond that, I'm useless.  If someone more clueful wants to 
tackle these issues, that'd be great.  I can supply test clips and such 
if you need them.  Otherwise, I'd appreciate some guidance on where to 
start hacking within the code.



[*] My timing test patch, in case anyone's interested:

--- src/misc/vlm.c.orig 2005-06-25 07:43:02.000000000 -0600
+++ src/misc/vlm.c      2005-11-07 20:40:53.000000000 -0700
@@ -116,8 +116,10 @@
      {
          vlm_message_t *p_message = NULL;
          char *psz_buffer = NULL;
+        time_t start = time(0);

-        msg_Dbg( p_this, "loading vlm conf ..." );
+        msg_Warn( p_this, "loading vlm conf at %s...",
+                asctime(localtime(&start)));
          asprintf(&psz_buffer, "load %s", psz_vlmconf );
          if( psz_buffer )
          {
@@ -128,6 +130,8 @@
              free(p_message);
              free(psz_buffer);
          }
+        msg_Warn( p_this, "...finished loading vlm conf, %d seconds.",
+                time(0) - start);
     }
     free(psz_vlmconf);

-- 
This is the vlc mailing-list, see http://www.videolan.org/vlc/
To unsubscribe, please read http://www.videolan.org/support/lists.html