[vlc-devel] [RFC] video filters, hw acceleration and 10bits

[Steve Lhomme]

On Fri, May 19, 2017 at 12:37 PM, Jean-Baptiste Kempf <jb at videolan.org> wrote:
> Hello everyone,
>
> 1/ Hardware decoders
>
> For the VLC 3.0 release, we want to enable hardware decoders with 0 CPU
> copy by default on all platforms: Linux (vaapi/vdpau), Windows
> (d3d11/9),
> macOS/iOS (videotoolbox) and Android (mediacodec).
>
> Indeed, we should publish 3.0 with this feature since there are more and
> more
> contents (4K, HEVC, 10bits, >= 60fps) that need a hardware decoder to
> be played correctly. Notably because newer laptop and ultrabooks have
> very small CPU but good GPU decoders.

I agree this should be on by default. Also by default most people
never use filters so won't even be bothered by the other issues. They
would not understand it has to be manually done (and up until now
restrict the number of threads to 1).

> In order to activate those hardware decoders, we need to have at least:
>  - hw decoding + display with scaling + subtitles blending in GPU,
>  - hw deinterlacing,
>  - snapshot,
>  - hw adjust filter, aka gamma correction.
>
> All of theses features are done or under work for all the platform we
> care about.
>
>
> 2/ The issue: CPU filters
>
> The issues we have is CPU filters: those do not work with hw decoding,
> and we can't port all of them to GPU in a timely fashion (some might
> be even impossible to port).
>
> At the same time, we're seeing exactly more and more content with
> non-YUV 8bits chromas, notably with HEVC, and very few filters work
> in those cases too.
>
> Users expect VLC filters to work and don't understand why some filters
> work and why some other (when 10bits/RGB or when using hw) don't.
>
>
> 3) Solutions
>
> There are a few solutions, but not all of them have the same User
> eXperience.
>
> a) It's very difficult to restart the decoder and the vout when
> a CPU filter is selected: glitches because a different vout will be
> used, we loose input and output frames, we'll need to wait for an I
> frame, we're not sure to be able to go back to the same state/position.
> We have already this difficulty with VT and Mediacodec when we restart
> them.
> And this also does not solve the non-8bits-chroma issue.
>
> b) Another solution, the most seamless we can do, is to copy back the
> video buffer to the CPU, filter it, and copy to the GPU when a CPU
> filter is requested. This is quite friendly for the user, but is of
> course slow.
>
> This approach is much slower than full hardware rendering but,
> if you have SSE4.1 (MOVNTDQA), it uses a less CPU than full CPU
> decoding,
> which is a gain compared to what we have now.
> Every CPU for the last 10 years, have SSE4.1.
>
> We've benched that on Linux (vaapi), Windows (D3D), macOS (VT), and we
> always
> have faster results than full CPU decoding, notably in fullHD.
> However, it starts to be worse with 4K videos: only one thread (the vout
> one) can't cope with 2 GPU/CPU copy.
>
> c) We could also wait for all the filters to be in GPU/shaders, but
> that's unrealistic
> for this release, but should be a major goal of the next one.

I think b+c is the way to go. It gives optimal performance and room
for improvement over time. Some deinterlacing filters like yadif2 may
not be possible in GPU (it might be). But there's always a way to
fallback to full CPU decoding+yadif2 and hope it works better this
way. Users who manage to select a specific filter should be able to
fallback to CPU decoding.

> d) Or just do nothing.
>
>
> 4/ Necessary user interface improvements
>
> However, whatever we do, we need to mention this issue to the users.
>
> Notably, the filters dialog must be reworked to mention when we're using
> hw decoding, and we must not save by default the VLC filters. (#6873)
>
> Maybe we can open the correct settings when the user click on the
> message,
> to be able to reach easily the "hardware decoder" configuration.
>
>
> 5/ Opinions?
>
> In my opinion, it's better to insert CPU filters, even if they could be
> slow,
> than to do nothing. With enough warnings, though.
> This is by far the simplest for our users and our support.
>
> Moreover, this would solve the different chromas issue (#14037, #13066,
> #16466
> for example). We should probably use I420 as the middle-man, though.
>
> Of course this solution is not perfect, and we need, for the future VLC
> releases, to focus on getting way more GPU filters.
> But this will not be ready for 3.0.
>
>
> What are your opinions on the matter?
>
> --
> Jean-Baptiste Kempf -  President
> +33 672 704 734
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel