[vlc-devel] [PATCH 05/14] filter: allow the owner not to provide a buffer callback

Steve Lhomme robux4 at ycbcr.xyz
Wed Sep 18 15:16:06 CEST 2019 

On 2019-09-18 14:33, Alexandre Janniaux wrote:
> Hi,
> 
> Just to have a common language:
> 
> + a decoder device is a hint given by the windowing system to the decoder,
> and potentially the decoder won't start without the correct decoder device
> if it can have some meaning.

Yes, although it's a bit more generic than that. It can be used without 
a window at all, like in transcode. It can also be forced with the 
--dec-dev option.

> + a video context is the common denominator between all the pictures going
> out of a decoder, or a filter, although it could change in the middle of a
> playback, and is pushed with the pictures (so it comes from before in the
> pipeline, and can therefore be changed when creating new pictures).

It's also used to initialize the playback pipeline, before pictures are 
pushed.

> Using the decoder device correctly would help zero copy in the pipeline,
> if I understood correctly.

Yes.

> As I sent privately (and will publish soon), I've been writing an OpenGL
> filter API for some time, which can be plugged to the classical filter_t
> API.
> 
> When doing so, there is a filter_t creating an OpenGL context and the
> OpenGL pipeline, and outputting native buffers. So it's built upon things
> like GBM or Android Native Buffer. GBM is probably a good test case here:
> 
> The filter must be initialized with a gbm device, which represent the
> device used for allocation. When supported by the EGL implementation, it
> also represents the rendering device that will be used.
> 
> In this case, the most likely to happen is having a VAAPI decoder device
> or NVDEC decoder device with X11 or Wayland window system and OpenGL video
> output display. So we have vaapi/nvdec video context out of the decoder,
> then our GBM opengl filter_t should produce a GBM video context, and the
> last part, the OpenGL vout display, should be able to handle a GBM video
> context.
> 
> Then, the question becomes: how to adapt the VAAPI decoder device in the
> GBM OpenGL filter_t so as to use the same decoding and rendering device
> everywhere?

It's the job of the "chain" filter to try to adapt an added filter to 
the previous output. If there's a VAAPI/GBM "chroma" converter then it 
should work (it may not copy anything and just translate surfaces from 
one format to the other).

To make sure they use the same physical device is trickier and currently 
not handled with what I did. There is the possibility to compare two 
video context but they must have the same type. Matching a physical 
device between different video context formats is undefined behaviour.

But it could still be handled, when the GBM filter is first created, 
it's given the video context of the decoder (VAAPI), there could be some 
GBM code that it can use to get a GBM context matching the same physical 
device (if zero copy or direct GPU copy is possible between the 2 APIs).

> Is it correct?
> 
> 
> Regards,
> --
> Alexandre Janniaux
> Videolabs
> 
> On Wed, Sep 18, 2019 at 02:32:01PM +0300, Rémi Denis-Courmont wrote:
>> Hmm well yeah... in other words, it's not easy or at least not obvious in the general case.
>>
>> Also how does should an OpenGL shader-based filter if the decoder device is, say, NVdec, and the video context is X11? It knows neither of them. Sure it'd solvable, but to me it's not obvious at all, and I'd rather sort it out in a workshop than over a mailing list.
>>
>> Le 18 septembre 2019 12:15:30 GMT+03:00, Steve Lhomme <robux4 at ycbcr.xyz> a écrit :
>>> On input of filters it's easy, they get a video context (or NULL for
>>> CPU
>>> based sources).
>>>
>>> The issue is when a filter needs to create a video context on its own.
>>> For CPU to GPU filters it cannot use the input video context. Just like
>>>
>>> a decoder it should use the "decoder device" to create its output video
>>>
>>> context.
>>>
>>> So just like for the decoder, the owner of the filter provides a
>>> "get_decoder_device" callback to get it.
>>>
>>> On 2019-09-18 10:39, Rémi Denis-Courmont wrote:
>>>> Hi,
>>>>
>>>> I am not sure how hardware should be exposed to filters/converters. I
>>> don't recall covering that part of the puzzle at the February workshop.
>>>>
>>>> Le 18 septembre 2019 09:00:41 GMT+03:00, Steve Lhomme
>>> <robux4 at ycbcr.xyz> a écrit :
>>>>> On 2019-09-17 17:32, Rémi Denis-Courmont wrote:
>>>>>> Le tiistaina 17. syyskuuta 2019, 17.22.33 EEST Steve Lhomme a écrit
>>> :
>>>>>>> In this case we just allocate a picture from the filter output
>>>>> format.
>>>>>>> ---
>>>>>>>     include/vlc_filter.h | 7 ++++++-
>>>>>>>     1 file changed, 6 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/include/vlc_filter.h b/include/vlc_filter.h
>>>>>>> index 0c756ce10ec..0bc10e91997 100644
>>>>>>> --- a/include/vlc_filter.h
>>>>>>> +++ b/include/vlc_filter.h
>>>>>>> @@ -26,6 +26,7 @@
>>>>>>>     #define VLC_FILTER_H 1
>>>>>>>
>>>>>>>     #include <vlc_es.h>
>>>>>>> +#include <vlc_picture.h>
>>>>>>>
>>>>>>>     /**
>>>>>>>      * \defgroup filter Filters
>>>>>>> @@ -160,7 +161,11 @@ struct filter_t
>>>>>>>      */
>>>>>>>     static inline picture_t *filter_NewPicture( filter_t *p_filter
>>> )
>>>>>>>     {
>>>>>>> -    picture_t *pic = p_filter->owner.video->buffer_new( p_filter
>>> );
>>>>>>> +    picture_t *pic;
>>>>>>> +    if ( p_filter->owner.video != NULL &&
>>>>> p_filter->owner.video->buffer_new
>>>>>>> != NULL)
>>>>>>
>>>>>> Two checks look redundant, though anyway this callback should go
>>> away
>>>>>> altogether eventually.
>>>>>
>>>>> Yes, it will go away.
>>>>>
>>>>> The owner will have the "get_device" callback so it's possible at
>>> some
>>>>> point in the code to have one callback and not the other. But in the
>>>>> end
>>>>> this code won't be there anymore.
>>>>>
>>>>>>> +        pic = p_filter->owner.video->buffer_new( p_filter );
>>>>>>> +    else
>>>>>>> +        pic = picture_NewFromFormat( &p_filter->fmt_out.video );
>>>>>>>         if( pic == NULL )
>>>>>>>             msg_Warn( p_filter, "can't get output picture" );
>>>>>>>         return pic;
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rémi Denis-Courmont
>>>>>> http://www.remlab.net/
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> vlc-devel mailing list
>>>>>> To unsubscribe or modify your subscription options:
>>>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>>>>
>>>>> _______________________________________________
>>>>> vlc-devel mailing list
>>>>> To unsubscribe or modify your subscription options:
>>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>>
>>>> --
>>>> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez
>>> excuser ma brièveté.
>>>>
>>>>
>>>> _______________________________________________
>>>> vlc-devel mailing list
>>>> To unsubscribe or modify your subscription options:
>>>> https://mailman.videolan.org/listinfo/vlc-devel
>>>>
>>> _______________________________________________
>>> vlc-devel mailing list
>>> To unsubscribe or modify your subscription options:
>>> https://mailman.videolan.org/listinfo/vlc-devel
>>
>> --
>> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.
> 
>> _______________________________________________
>> vlc-devel mailing list
>> To unsubscribe or modify your subscription options:
>> https://mailman.videolan.org/listinfo/vlc-devel
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel
>

More information about the vlc-devel mailing list