[vlc-devel] [PATCH 06/15] vdpau/display: check the source video format via vd->source

[Steve Lhomme]

On 2020-09-03 7:52, Steve Lhomme wrote:
> The output format is only needed at the end to set the chroma and apply
> the source rotation.
> ---
>   modules/hw/vdpau/display.c | 25 +++++++++++--------------
>   1 file changed, 11 insertions(+), 14 deletions(-)
> 
> diff --git a/modules/hw/vdpau/display.c b/modules/hw/vdpau/display.c
> index e3484a72988..2ca9c519362 100644
> --- a/modules/hw/vdpau/display.c
> +++ b/modules/hw/vdpau/display.c
> @@ -323,19 +323,16 @@ static int Open(vout_display_t *vd, const vout_display_cfg_t *cfg,
>           msg_Dbg(vd, "using back-end %s", info);
>   
>       /* Check source format */
> -    video_format_t fmt;
>       VdpChromaType chroma;
>       VdpYCbCrFormat format;
>       VdpStatus err;
>   
> -    video_format_ApplyRotation(&fmt, fmtp);
> -
> -    if (fmt.i_chroma == VLC_CODEC_VDPAU_VIDEO_420
> -     || fmt.i_chroma == VLC_CODEC_VDPAU_VIDEO_422
> -     || fmt.i_chroma == VLC_CODEC_VDPAU_VIDEO_444)
> +    if (vd->source->i_chroma == VLC_CODEC_VDPAU_VIDEO_420
> +     || vd->source->i_chroma == VLC_CODEC_VDPAU_VIDEO_422
> +     || vd->source->i_chroma == VLC_CODEC_VDPAU_VIDEO_444)
>           ;
>       else
> -    if (vlc_fourcc_to_vdp_ycc(fmt.i_chroma, &chroma, &format))
> +    if (vlc_fourcc_to_vdp_ycc(vd->source->i_chroma, &chroma, &format))
>       {
>           uint32_t w, h;
>           VdpBool ok;
> @@ -348,7 +345,7 @@ static int Open(vout_display_t *vd, const vout_display_cfg_t *cfg,
>                       vdp_get_error_string(sys->vdp, err));
>               goto error;
>           }
> -        if (!ok || w < fmt.i_width || h < fmt.i_height)
> +        if (!ok || w < vd->source->i_width || h < vd->source->i_height)

Actually video_format_ApplyRotation() is used. So the width/height may 
be flipped.
Not sure how the core responds to that, if it adds a rotate filter 
automatically (the source and display format have different sizes and 
orientation).

And then if you crop or change the aspect ratio, it goes through 
RESET_PICTURES. But the source orientation is not handled anymore to 
create the new display format...