[vlc-devel] [PATCH 00/10] Let the VA module set the output chroma

[Steve Lhomme]

As explained in a previous email, in D3D11VA it is possible to select the best
output DXGI format based on the source characteristics (like bitdepth and chroma
subsampling). If the best available format is not the one expected by the
generic VA module, the output i_chroma doesn't match the DXGI format it
corresponds to. (for example NV12 instead of P010 for a 10 bits source).

Now that we push the video format from the decoder we don't need to guess the
i_chroma that the VA module is going to use. The VA module can set it itself
when it knows the format it's actually using.

This patchset changes the code to behave like that. We still check early if the
pixel format is likely to be supported by the VA or not, but this can be moved
in each VA module later. In the end the generic lavc module will no nothing 
about each VA capability.

Instead of loading each module for each hwfmt tested, we may also pass the list
of supported formats to each VA module and they can see if there's the one they
support or not. In that case the lavc module will know about the VA modules at
all.

Note: The VAAPI and VDPAU patches are untested.

A branch with this code can be found at:
https://code.videolan.org/robUx4/vlc/tree/va_chroma/1

Steve Lhomme (10):
  avcodec: va: add a function to quickly check if the format might be
    decoded
  avcodec: va: use the check function to tell if we should try the VA
    flavor
  avcodec: va: pass the hardware lavc format to the module
  d3d11va: set the i_chroma corresponding to the decoded format
  dxva2: set the i_chroma corresponding to the decoded format
  avcodec: vaapi: set the i_chroma corresponding to the decoded format
  avcodec: vdpau: set the i_chroma corresponding to the decoded format
  avcodec: va: move the hw chroma setting out of lavc_GetVideoFormat()
  avcodec: va: let the VA module set the output i_chroma
  avcodec: va: remove unused PixelFormat->vlc_fourcc_t mapping function

 modules/codec/avcodec/d3d11va.c |  8 +++----
 modules/codec/avcodec/dxva2.c   | 26 ++++++++++-------------
 modules/codec/avcodec/va.c      | 37 +++++++++++++--------------------
 modules/codec/avcodec/va.h      | 14 +++++++------
 modules/codec/avcodec/vaapi.c   | 11 ++++------
 modules/codec/avcodec/video.c   | 18 ++++++++++------
 modules/hw/vdpau/avcodec.c      | 12 ++++++-----
 7 files changed, 61 insertions(+), 65 deletions(-)

-- 
2.17.1