[vlc-devel] [RFC] video filters, hw acceleration and 10bits

[Marvin Scholz]

On 19 May 2017, at 12:37, Jean-Baptiste Kempf wrote:

> Hello everyone,
>
> 1/ Hardware decoders
>
> For the VLC 3.0 release, we want to enable hardware decoders with 0 
> CPU
> copy by default on all platforms: Linux (vaapi/vdpau), Windows
> (d3d11/9),
> macOS/iOS (videotoolbox) and Android (mediacodec).
>
> Indeed, we should publish 3.0 with this feature since there are more 
> and
> more
> contents (4K, HEVC, 10bits, >= 60fps) that need a hardware decoder to
> be played correctly. Notably because newer laptop and ultrabooks have
> very small CPU but good GPU decoders.
>
> In order to activate those hardware decoders, we need to have at 
> least:
>  - hw decoding + display with scaling + subtitles blending in GPU,
>  - hw deinterlacing,
>  - snapshot,
>  - hw adjust filter, aka gamma correction.
>
> All of theses features are done or under work for all the platform we
> care about.

I am in favor of this as well, most other macOS players I know do this 
too and
if VLC would not do that, users would experience Videos playing fine in 
Player X
but not in VLC.

>
>
> 2/ The issue: CPU filters
>
> The issues we have is CPU filters: those do not work with hw decoding,
> and we can't port all of them to GPU in a timely fashion (some might
> be even impossible to port).
>
> At the same time, we're seeing exactly more and more content with
> non-YUV 8bits chromas, notably with HEVC, and very few filters work
> in those cases too.
>
> Users expect VLC filters to work and don't understand why some filters
> work and why some other (when 10bits/RGB or when using hw) don't.
>
>
> 3) Solutions
>
> There are a few solutions, but not all of them have the same User
> eXperience.
>
> a) It's very difficult to restart the decoder and the vout when
> a CPU filter is selected: glitches because a different vout will be
> used, we loose input and output frames, we'll need to wait for an I
> frame, we're not sure to be able to go back to the same 
> state/position.
> We have already this difficulty with VT and Mediacodec when we restart
> them.
> And this also does not solve the non-8bits-chroma issue.
>
> b) Another solution, the most seamless we can do, is to copy back the
> video buffer to the CPU, filter it, and copy to the GPU when a CPU
> filter is requested. This is quite friendly for the user, but is of
> course slow.
>
> This approach is much slower than full hardware rendering but,
> if you have SSE4.1 (MOVNTDQA), it uses a less CPU than full CPU
> decoding,
> which is a gain compared to what we have now.
> Every CPU for the last 10 years, have SSE4.1.
>
> We've benched that on Linux (vaapi), Windows (D3D), macOS (VT), and we
> always
> have faster results than full CPU decoding, notably in fullHD.
> However, it starts to be worse with 4K videos: only one thread (the 
> vout
> one) can't cope with 2 GPU/CPU copy.
>
> c) We could also wait for all the filters to be in GPU/shaders, but
> that's unrealistic
> for this release, but should be a major goal of the next one.
>
> d) Or just do nothing.
>

I think that b is probably the best option for now, until c is done for 
most
filters. And yes, delaying 3.0 any longer for c certainly does not seem
like a good idea.

>
> 4/ Necessary user interface improvements
>
> However, whatever we do, we need to mention this issue to the users.
>
> Notably, the filters dialog must be reworked to mention when we're 
> using
> hw decoding, and we must not save by default the VLC filters. (#6873)
>
> Maybe we can open the correct settings when the user click on the
> message,
> to be able to reach easily the "hardware decoder" configuration.
>

This clearly needs to be communicated in a good way to the user,
and give them a easy way to disable hardware decoding if necessary.
I am not sure what would be the best way to do this though.

>
> 5/ Opinions?
>
> In my opinion, it's better to insert CPU filters, even if they could 
> be
> slow,
> than to do nothing. With enough warnings, though.
> This is by far the simplest for our users and our support.
>
> Moreover, this would solve the different chromas issue (#14037, 
> #13066,
> #16466
> for example). We should probably use I420 as the middle-man, though.
>
> Of course this solution is not perfect, and we need, for the future 
> VLC
> releases, to focus on getting way more GPU filters.
> But this will not be ready for 3.0.
>
>
> What are your opinions on the matter?
>
> -- 
> Jean-Baptiste Kempf -  President
> +33 672 704 734