[vlc-devel] [PATCH 0/1] Improve SFTP network performance with buffer

[Martin Disch]

Thank you for your extensive reply, that was very insightful.

On Wed, Feb 14, 2018 at 6:51 PM, Rémi Denis-Courmont <remi at remlab.net> wrote:
> Normally, VLC uses whatever it needs. If the stream plays back at 10 Mbps, VLC
> will use roughly that, after a short initial prebuffering spike at maximum
> bandwidth.

That's what I would have expected, but somehow it doesn't seem to be
the case. VLC 3.0 without any special configuration consistently
doesn't manage to play back a ~12 Mbit/s file from a server I have a
great connection to. I would often run into the situation that VLC
would stop to fetch some more data during playback, when I could
easily download the file at 250 Mbit/s.

>> Data is read into that buffer and the amount requested by the player is
>> copied from there to the buffer provided in the Read() call.
>
> The prefetch filter already does that. You can achieve the same thing with
> zero lines of code, by tweaking the prefetch parameters.

That's exactly what I was afraid of. I did some searches on the forums
concerning SFTP performance and the prefetch module did not turn up
(SFTP is not a topic that is much discussed). So instead I went and
reinvented the wheel like a fool :)

> That is why the prefetch module makes relative small requests by default
> (16 KiB), but then keeps asking for data until buffers are full, to reduce the
> risk of TCP-level congestion and back-off.

That makes perfect sense, but I get a strange result. I tried the
prefetch module just now, using a larger than default buffer (500 MiB)
and leaving the other parameters unchanged. Something definitely
happens, since VLC is indeed using the additional memory. But
surprisingly, it doesn't produce a spike in network utilization to
fill up the buffer. Instead, the buffer slowly increases in size to
its maximum capacity as content is played. And it appears that it is
not really prebuffering, but rather post-buffering (if that's a
thing). I can skip backwards very quickly (which I can't with a small
buffer), making it look like this part is already in memory. But it
doesn't help at all with the stutters, they still happen in places
where the video's bitrate increases.
When having a large buffer on its own didn't help, I set the read size
for the prefetch module to an unreasonably high value. To my great
delight, that changed everything. Now I'm seeing very high network
usage (50 - 60 Mbit/s) until the buffer is full. So despite you
pointing out the opposite, it looks like the size of application reads
does matter in this case, assuming that's what the read size option
translates to. Which is strange, since your argument makes perfect
sense.

> While bandwidth is far more important than latency, the latency potentially
> incurred by blocking reads of 1 MiB is too high to be generally acceptable,
> not to even mention 10 MiB or 100 MiB. If just wouldn't do if it took one
> minute before VLC started playing.

There won't be a blocking read. libssh2_sftp_read() won't try to
return 1 MiB, even if that's what we asked for, because that would
block the socket. Instead, what it usually returned in my testing were
30 kB reads, which is a good size.
This is probably also why using a very high read size in the prefetch
module doesn't cause a problem, but allows VLC to read at a much
higher rate than normal.

> The prefetch buffer is already 16 MiB by default. So adding a 1 MiB buffer in
> front of it is not going to do much good.

That seems logical, but I'm still surprised by the fact that my 1 MiB
buffer hack makes such a difference (without using prefetch, that is).
But now that I figured out how to configure the prefetch filter to
give me the same result, that's a much better solution than
complicating the SFTP module.

Thank you again, consider this issue closed.

--
Martin Disch
martindisch at gmail.com