[vlc-devel] [PATCH 0/1] Improve SFTP network performance with buffer

[Rémi Denis-Courmont]

Le mercredi 14 février 2018, 16:05:00 EET Martin Disch a écrit :
> Streaming video files over SFTP does not currently work well when the source
> is not on the local area network, but somewhere on the internet where there
> is some latency instead. Playback often stops to load more data and when
> looking at the bandwidth usage, VLC doesn't use more than 8-10 Mbit/s, even
> though my connection can offer more than 20 times that.

Normally, VLC uses whatever it needs. If the stream plays back at 10 Mbps, VLC 
will use roughly that, after a short initial prebuffering spike at maximum 
bandwidth.

Multimedia streams at 200 Mbps are rather uncommon so far.

> I've taken a look at the code and think I found the issue. The SFTP module
> implements the Read() function for access to the next N bytes of a file.
> When called, it uses libssh2 to get that number of bytes. Unfortunately,
> the amount requested by VLC is relatively small, at least smaller than what
> would be possible to fetch with libssh2_sftp_read() without blocking the
> socket.

Authenticated transport protocols such as TLS and SSH use framing, and do not 
allow reading an incomplete frame. This is not at all like raw 
unauthenticated/unencrypted TCP.

As suchm the size of application reads should not matter at all with TLS and 
SSH. The protocol layer will always release one frame after it has been able 
to decrypt and authenticate it. Then the internal buffer of the protocol 
implementation will split the frames to match application reads.

What matters is *not* the size of the application reads, it is the size of the 
receive window and/or the internal protocol buffer. The window needs to be no 
smaller than the playback bandwidth multipled by the round trip time, so that 
streaming does not stall.

> Not requesting that many bytes in one call of Read() might be ok
> for devices that have relatively low latency due to being attached to the
> local machine, but when these requests go over a network with some latency,
> it's less than ideal.

It is the other way around. The more bytes you request from the previous 
layer, the worse the latency becomes. Latency is minimized by requesting the 
same number of bytes than a lower-layer frame can carry. Requesting more than 
that will increase latency, as frame will need to be batched. If the request 
size exceeds the upstream buffer size, then latency will explode, as network 
round-trip becomes necessary to fill the request.

That is why the prefetch module makes relative small requests by default 
(16 KiB), but then keeps asking for data until buffers are full, to reduce the 
risk of TCP-level congestion and back-off.

> Additionally, because of network conditions, many of
> the requests return less than what was requested.

Actually that is perfect. If the plugin can return whatever it has readily 
available rather than what the requested size is, it will avoid latency caused 
by crossing frame boundaries and batching multiple consecutive frames.

> My solution uses a small (1 MiB) internal circular buffer, to allow
> requesting as much data in one call as possible from libssh2.

That will _maximize_, not minimize, latency. And it is reinventing the 
prefetch module that we already have to solve that problem globally.

> Data is read into that buffer and the amount requested by the player is
> copied from there to the buffer provided in the Read() call.

The prefetch filter already does that. You can achieve the same thing with 
zero lines of code, by tweaking the prefetch parameters.

Or then there is a bug that SFTP does not use prefetch somehow.

> In effect, the calls to Read() now request more bytes with libssh2 (as many
> as possible without blocking the socket), which are buffered and can help
> cover for the times when less than requested is returned by libssh2.
> I found in my testing that this increases peak network throuput by about 4
> times by using just the small buffer. Of course this increases with the
> buffer size.
> Here are the buffer sizes compared to the peak network utilization when
> playing and skipping through the same video file:
> 
>   1 MiB -> 40 Mbit/s
>  10 MiB -> 100 Mbit/s
> 100 MiB -> 220 Mbit/s

While bandwidth is far more important than latency, the latency potentially 
incurred by blocking reads of 1 MiB is too high to be generally acceptable, 
not to even mention 10 MiB or 100 MiB. If just wouldn't do if it took one 
minute before VLC started playing.

> Seeing this, ideally the buffer size would be made available as an option in
> the settings of the module.

It is already that way (Input -> Stream filter -> Prefetch).

> But since the 1 MiB buffer alone increases
> performance by quite a bit at practically no cost and I wanted to keep this
> simple, that's what I opted for.

The prefetch buffer is already 16 MiB by default. So adding a 1 MiB buffer in 
front of it is not going to do much good.

-- 
Rémi Denis-Courmont