[vlc-devel] [PATCH] prefetch: remove read size, always request maximum

Thu Feb 6 14:03:44 CET 2020

On Thu, Feb 6, 2020, at 13:47, Rémi Denis-Courmont wrote:
> Hi,
> 
> A plugin should wait for data if it has none available, and return whatever data is available up to the requested length.

OK, I guess we should put that on streams.h documentation.

> 
> Of course if SMB2 behaves as I suspect it does, capping the read size will break streaming high rate media on high RTT networks... But it's necessary to stream media on low bandwidth networks. That's why I wrote that you've already lost if you care about the read size.

This seems to be a limitation of the protocol, not our libs (libsmb2 or libsmbclient), cf. https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/4801c3e5-dd73-4e1c-b2ba-8ebf73642227
The client has to send read requests of a fixed size. This make impossible to behave as you said.

Still, there is something I don't understand, your commits (the 3.0 one) say it fix #19988 and #21470, that are SMB bugs. Putting a read cap in smb2.c like you said will just make these bugs reappear, no ?

> 
> Le 6 février 2020 14:36:18 GMT+02:00, Thomas Guillem <thomas at gllm.fr> a écrit :
>> 
>> On Thu, Feb 6, 2020, at 13:22, Rémi Denis-Courmont wrote:
>>> Hi,
>>> 
>>> I don't think SMB streaming (as opposed to downloading) is a lost cause. I do think SMB streaming with libsmbclient implementation is a lost cause.
>>> 
>>> The prefetch plugin already gives maximum flexibility, within its own constraints, to the underlying access to choose how many bytes it returns in a given read, as does the core.
>> 
>> Then, we have to update the vlc_stream.h pf_read() documentation. Nothing says that access modules *should* (and not could) return less than the requested size in some case.
>> 
>> I still don't know how I should cap the read size for the smb2 access plugin.
>> If I cap the smb2 read size using the same old value, it will just re-open https://trac.videolan.org/vlc/ticket/19988 and https://trac.videolan.org/vlc/ticket/21470 ?
>> 
>> Or I missed something ?
>> 
>>> 
>>> It can't get any better without increasing the prefetch buffer size (and we have a setting for that already).
>> 
>> 
>> 
>>> 
>>> Le 6 février 2020 14:00:55 GMT+02:00, Thomas Guillem <thomas at gllm.fr> a écrit :
>>>> I quote you:
>>>> 
>>>>  "Badly behaving streams wait for the whole requested amount and do not
>>>>  perform any pipelining/buffering (e.g. CIFS, probably SFTP)."
>>>> 
>>>> Why SMB, NFS, SFTP are badly behaving streams ? For me it is just like local I/O. You read only what you request.
>>>> 
>>>> On Thu, Feb 6, 2020, at 12:28, Rémi Denis-Courmont wrote:
>>>>> Hi,
>>>>> 
>>>>> The access is free to return any non-zero size up to the requested read size, so the requested read size can never be too big. It can only ever be too small, and that's exactly why this patch made sense and still makes sense.
>>>> 
>>>> At the VideoLabs office with 1TB network, I don't care, but at home with a shitty 2MB wifi, reading chunks by 16MB is terrible. Specially when 99% is trashed since the demuxer will only ask few kB and seek.
>>>> 
>>>>> 
>>>>> But if you need to worry about the read size, you've already lost.
>>>> 
>>>> Why am I lost ? Are you saying SMB streaming is a lost cause ? 
>>>> 
>>>>> Small read size will cause bandwidth problems and large read size will cause latency problems.
>>>> 
>>>> Can we find a solution for both cases in the core or the cache modules ? I really don't see why I should cap the read size of all "badly behaving" streams.
>>>> If I need to do that, what size should I use ? Should I add one more option (likely the same that you removed in prefetch) ?
>>>> 
>>>>> 
>>>>> Le 6 février 2020 12:33:55 GMT+02:00, Thomas Guillem <thomas at gllm.fr> a écrit :
>>>>>> Hello,
>>>>>> 
>>>>>> This commit and this one on vlc-3.0 : https://git.videolan.org/?p=vlc/vlc-3.0.git;a=commit;h=3dfa6d4ddd0922552e6575ded416f62b8ce828f5 are making the smb2 module ultra slow, around 10 times slower.
>>>>>> 
>>>>>> The read size requested by prefetch is really big. This make probing / seek ultra slow since a lot  of small read() are executed (reproduced with mkv and mp4).
>>>>>> 
>>>>>> So, I have one big question: Is the smb2 module supposed to not listen to the size argument and always read a smaller size ?
>>>>>> How can I decide the read size then ? Should this size be decided by the core of cache modules instead (like before) ?
>>>>>> 
>>>>>> PS:  There is not the same problem with the libdsm module since the max read size is caped inside libdsm.
>>>>>> 
>>>>>> On Fri, Apr 20, 2018, at 15:45, Thomas Guillem wrote:
>>>>>> > 
>>>>>>> On Fri, Apr 20, 2018, at 15:05, Rémi Denis-Courmont wrote:
>>>>>>>> And I don't really want to hear a commit is dangerous from somebody that 
>>>>>>>> backports stuff as lightly and quickly  as you have lately.
>>>>>>> That is a bit close to false accusation IMHO.
>>>>>>> 
>>>>>>>> This was sent for review, and it was *not* backported.
>>>>>>> Yes and I was OK with it. It's just lately that I found out that this 
>>>>>>> commit might be dangerous.
>>>>>>> Dangerous as changing the behavior of some access modules and maybe 
>>>>>>> induce some new bugs (that are access related).
>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser 
>>>>>>>> ma brièveté.vlc-devel mailing list
>>>>>>>> To unsubscribe or modify your subscription options:
>>>>>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>>>>> vlc-devel mailing list
>>>>>>> To unsubscribe or modify your subscription options:
>>>>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>>>> vlc-devel mailing list
>>>>>> To unsubscribe or modify your subscription options:
>>>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>>> 
>>>>> -- 
>>>>> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté. 
>>>>> _______________________________________________
>>>>> vlc-devel mailing list
>>>>> To unsubscribe or modify your subscription options:
>>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>> 
>>> 
>>> -- 
>>> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté. 
>>> _______________________________________________
>>> vlc-devel mailing list
>>> To unsubscribe or modify your subscription options:
>>> https://mailman.videolan.org/listinfo/vlc-devel
>> 
> 
> -- 
> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté. 
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/vlc-devel/attachments/20200206/4f6fc998/attachment-0001.html>