[vlc-devel] [PATCH 0/1] Improve SFTP network performance with buffer

[Martin Disch]

Streaming video files over SFTP does not currently work well when the source
is not on the local area network, but somewhere on the internet where there
is some latency instead. Playback often stops to load more data and
when looking at the bandwidth usage, VLC doesn't use more than 8-10 Mbit/s,
even though my connection can offer more than 20 times that.

I've taken a look at the code and think I found the issue. The SFTP module
implements the Read() function for access to the next N bytes of a file. When
called, it uses libssh2 to get that number of bytes. Unfortunately, the amount
requested by VLC is relatively small, at least smaller than what would be
possible to fetch with libssh2_sftp_read() without blocking the socket.
Not requesting that many bytes in one call of Read() might be ok for devices
that have relatively low latency due to being attached to the local machine,
but when these requests go over a network with some latency, it's less than
ideal. Additionally, because of network conditions, many of the requests return
less than what was requested. Even with the network caching value set to about
a minute, smooth playback is not possible in some cases, because many calls to
libssh2_sftp_read() with a small number of bytes per request lead to network
utilization that is smaller than the bitrate of the file.

My solution uses a small (1 MiB) internal circular buffer, to allow requesting
as much data in one call as possible from libssh2. Data is read into that
buffer and the amount requested by the player is copied from there to the
buffer provided in the Read() call.
In effect, the calls to Read() now request more bytes with libssh2 (as many as
possible without blocking the socket), which are buffered and can help cover
for the times when less than requested is returned by libssh2.
I found in my testing that this increases peak network throuput by about 4
times by using just the small buffer. Of course this increases with the buffer
size.
Here are the buffer sizes compared to the peak network utilization when
playing and skipping through the same video file:

  1 MiB -> 40 Mbit/s
 10 MiB -> 100 Mbit/s
100 MiB -> 220 Mbit/s

Seeing this, ideally the buffer size would be made available as an option in
the settings of the module. But since the 1 MiB buffer alone increases
performance by quite a bit at practically no cost and I wanted to keep this
simple, that's what I opted for.

Since I'm not very familiar with VLC development, don't know if this solution
is a good idea at all or violates some principles, and have only been able to
test it on one platform so far (Ubuntu 17.10), I'm very interested in feedback.
Please let me know what you think.

Martin Disch (1):
  Improve SFTP network performance with buffer

 modules/access/sftp.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 101 insertions(+), 6 deletions(-)

-- 
2.14.1