[vlc-devel] [PATCH 0/1] Improve SFTP network performance with buffer

[Martin Disch]

On Sun, Feb 18, 2018 at 12:48 PM, Ilkka Ollakka <ileoo at videolan.org> wrote:
> By default vlc loads prefetch,cache_read modules for stream filtering for
> access modules that have pf_read defined, and prefetch,cache_block for the ones
> that have pf_block defined. So you don't need to define any of those to be
> enabled.

Thanks a lot for clearing that up, I don't know much about the inner
workings of VLC and only got into this because I use SFTP a lot and
noticed the issue. It all makes sense, now that I know prefetch is
loaded automatically.

> By remove to remove the libcache_read.so from plugins-directory. But you should
> be also be able to see which module is unsed from gui from
> view->messages->module tree tab. There should be either prefetch or cache_read
> in the tree.

Can confirm that I achieved the same thing by commenting out the
cache_read lines from the makefile. As I noticed before, the module
doesn't show up in the options anymore and after checking the messages
window now I can say that it's also not present in the module tree.

> So it might be that the prefetch-filter starves for some reason, meaning buffer
> is full and doesn't get into refilling when it's consumed in pace that would
> prevent stalling the playback.
>
> If it would be possible to check if tuning the prefetch-buffer-size helps
> instead of tuning the prefetch-read-size?

Unfortunately the remote machine where I have the files I access via
SFTP for testing is being moved right now, so I can't do that for the
next week or so. But if I remember correctly, I did already try that
yesterday before increasing the prefetch read size. After building VLC
without cache_read, I reset all the settings and increased the
prefetch buffer to 500 MiB. That alone did not help the slowness, the
buffer filled up, but only as quickly as playback was happening. And
there were multiple occasions where playback stopped for buffering, so
that did not help. Worked like a charm again after increasing the
prefetch read size though.
So my feeling is that only changing the prefetch buffer size does not
help, provided my recollection is accurate. I'll test that again in a
week and report back in case I get a different result then.

In the meantime, I was able to confirm that an increased prefetch read
size directly affects the read size used in the SFTP module. That's no
surprise, after all this is what should happen. With the default
prefetch read size of 16384 bytes, that's the maximum amount of bytes
that VLC asks the SFTP module to provide in the Read() function. With
the prefetch read size increased to 262144 (which I found to work well
on my system during testing), the SFTP module received requests for up
to 262144 bytes and sometimes libssh2_sftp_read() was actually able to
return that many in one call.
This means that increasing the prefetch read size to a very high value
has the same effect as my little sftp buffer hack, which makes the
SFTP module request as many bytes from libssh2 as it can provide in
one call without blocking the socket. Thus, it also explains why I get
the same result (smooth playback through good bandwidth utilization)
from both of these methods.

--
Martin Disch
martindisch at gmail.com