[vlc-devel] [RFC] video filters, hw acceleration and 10bits

[David Fuhrmann]

Hi j-b,

Thanks for summarizing the problems and options again.

> Am 19.05.2017 um 12:37 schrieb Jean-Baptiste Kempf <jb at videolan.org>:
> 
> Hello everyone,
> 
> 1/ Hardware decoders
> 
> For the VLC 3.0 release, we want to enable hardware decoders with 0 CPU
> copy by default on all platforms: Linux (vaapi/vdpau), Windows
> (d3d11/9),
> macOS/iOS (videotoolbox) and Android (mediacodec).
> 
> Indeed, we should publish 3.0 with this feature since there are more and
> more
> contents (4K, HEVC, 10bits, >= 60fps) that need a hardware decoder to
> be played correctly. Notably because newer laptop and ultrabooks have
> very small CPU but good GPU decoders.

I fully agree here. Hardware decoders are a very important feature for 3.0 in
my opinion, and we should definitely enable them by default if the input format
allows that. We cannot ask the user here to do this by themselves, as they
will not understand that.

> 
> In order to activate those hardware decoders, we need to have at least:
> - hw decoding + display with scaling + subtitles blending in GPU,
> - hw deinterlacing,
> - snapshot,
> - hw adjust filter, aka gamma correction.
> 
> All of theses features are done or under work for all the platform we
> care about.
> 
> 
> 2/ The issue: CPU filters
> 
> The issues we have is CPU filters: those do not work with hw decoding,
> and we can't port all of them to GPU in a timely fashion (some might 
> be even impossible to port).
> 
> At the same time, we're seeing exactly more and more content with
> non-YUV 8bits chromas, notably with HEVC, and very few filters work
> in those cases too.
> 
> Users expect VLC filters to work and don't understand why some filters
> work and why some other (when 10bits/RGB or when using hw) don't.
> 
> 
> 3) Solutions
> 
> There are a few solutions, but not all of them have the same User
> eXperience.
> 
> a) It's very difficult to restart the decoder and the vout when
> a CPU filter is selected: glitches because a different vout will be
> used, we loose input and output frames, we'll need to wait for an I
> frame, we're not sure to be able to go back to the same state/position.
> We have already this difficulty with VT and Mediacodec when we restart
> them.
> And this also does not solve the non-8bits-chroma issue.
> 
> b) Another solution, the most seamless we can do, is to copy back the
> video buffer to the CPU, filter it, and copy to the GPU when a CPU
> filter is requested. This is quite friendly for the user, but is of
> course slow.
> 
> This approach is much slower than full hardware rendering but,
> if you have SSE4.1 (MOVNTDQA), it uses a less CPU than full CPU
> decoding,
> which is a gain compared to what we have now.
> Every CPU for the last 10 years, have SSE4.1.
> 
> We've benched that on Linux (vaapi), Windows (D3D), macOS (VT), and we
> always
> have faster results than full CPU decoding, notably in fullHD.
> However, it starts to be worse with 4K videos: only one thread (the vout
> one) can't cope with 2 GPU/CPU copy.

I would also vote for option b). We had copy back to CPU since some time
on macOS and it was still always faster than software decoding as far as
I remember.

> 
> c) We could also wait for all the filters to be in GPU/shaders, but
> that's unrealistic
> for this release, but should be a major goal of the next one.
> 
> d) Or just do nothing.

This is not an option in my opinion, if some UI visible features are not
working anymore. If the user can enable a video filter, he expects it
to work.

> 
> 
> 4/ Necessary user interface improvements
> 
> However, whatever we do, we need to mention this issue to the users.

In general I believe the user expects that the UI elements work as before.
So if there is a hardware filter available, it should be controllable by the
same UI elements ideally.

> Notably, the filters dialog must be reworked to mention when we're using
> hw decoding, and we must not save by default the VLC filters. (#6873)

Not sure if this is that easy on macOS. The whole presets system for the
video filters works in such a way that the settings are saved.

However, I agree that the user should be somehow informed if there is
a potential problem. We could of course just disable the video filter controls
which are not working in hardware decoding anymore, but this is quite
limiting.

Best regards,
David

> 
> Maybe we can open the correct settings when the user click on the
> message,
> to be able to reach easily the "hardware decoder" configuration.
> 
> 
> 5/ Opinions?
> 
> In my opinion, it's better to insert CPU filters, even if they could be
> slow,
> than to do nothing. With enough warnings, though.
> This is by far the simplest for our users and our support.
> 
> Moreover, this would solve the different chromas issue (#14037, #13066,
> #16466
> for example). We should probably use I420 as the middle-man, though.
> 
> Of course this solution is not perfect, and we need, for the future VLC
> releases, to focus on getting way more GPU filters.
> But this will not be ready for 3.0.
> 
> 
> What are your opinions on the matter?
> 
> -- 
> Jean-Baptiste Kempf -  President
> +33 672 704 734
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel