[vlc-devel] [PATCH 00/36] add video context to filters (v5)

[Steve Lhomme]

Changes since v4:
- bring back hold_device for filter owners. In order for some filters to create
  an output video context, they need to do it from the decoder device (CPU to GPU
  filters). So we need to implement this feature before we can assume filters
  set a vctx_out when they output a GPU surface
- use a static inline for picture_GetVideoContext()
- fix videotoolbox compilation
- the vctx hold when cloning/copying a picture is done in each callback, in case
  they want to use a different vctx, it's also easier to track responsibilities
- add the video context in NVDEC pictures
- add the video context in VDPAU pictures
- set the video context in pictures coming out of D3D9, D3D11, VAAPI filters
- propagate the video context between filter chains input/output

Changes since v3:
- the video context in pictures is stored in the picture_context_t
- reorder some patches so the video context is available in more intermediate
  cases
- no more vlc_video_context_IsSimilar() for now
- some of the filter chain cleaning patches not related to push have been merged

Changes since v2:
- fix some vctx leaks
- example setting an output vctx keep a reference to it as long as they use it
- no more rename of the filter_chain fmt_out
- cleaning of the filter_chain API calls, for example once Reset is called, we
  don't need to append a filter stating the same input video format+context, in
  many cases the output video format is also implied by the Reset call.
  This cleaning makes the use of filter more readable in terms of what the
  input and output are. It also requires less patching when involving the video
  context.
- add checks on the video contexts received by GPU filters
- simplify the NVDEC GPU to CPU filter using the input video context
- add an example of VAAPI filter setting its output video context
- initialize the VAAPI pool using the video context rather than decoder device.

Changes since v1:
- added D3D9/D3D11 deinterlace use of the vctx on input and output
- use the input vctx in the NVDEC GPU to CPU filter
- add VLC_USED to the inline picture_NewFromFormat()
- reorder some patches
- add documentation to vlc_video_context_IsSimilar()
- fix crash when the vctx doesn't have "operations" callbacks (nvdec)
- less variable renaming


This patchset adds video context handling to filters.

The video context coming from the decoder is added to the filter chain
and each filter can create a video context on output if it outputs GPU surfaces.
This is not automatic.

Pictures also store a video context (via the picture pool).


The patches of this set can be found in this branch:
https://code.videolan.org/robUx4/vlc/tree/push/merge/86


Steve Lhomme (36):
  filters: provide the decoder device to filter so they can create a
    video context
  video_output: add a dummy filter_video_callbacks for the blender
  video_output: implement the hold_device for filters
  d3d11_surface: implement the hold_device callback for the CPU to GPU
    filter
  dxa9: implement the hold_device callback for the CPU to GPU filter
  vdpau: chroma: get the decoder device using filter_HoldDecoderDevice()
  picture: add the video context in the picture context
  video_output: pass the input video context with the source format
  filters: set the input/output video context on video filters
  filter_chain: set the input video context before opening a filter
  display: keep the video context to feed the converter locally
  filter_chain: keep the input video context on Reset
  d3d11_surface: create the output video context for CPU to D3D11
    converters
  dxa9: create the output video context for CPU to D3D9 converters
  d3d11: set the video context in the picture context
  d3d9: set the video context in the picture context
  vdpau: set the video context in the picture context
  nvdec: set the video context in the picture context
  vaapi: create VAAPI pools with a video context
  vaapi: set the video context in the picture context
  vaapi: filters: set the output video context
  vaapi: filters: get the decoder device from the input vctx
  vaapi: chroma: get the decoder device using filter_HoldDecoderDevice()
  vaapi: chroma: don't keep the decoder device after we put it in the
    vctx
  vaapi: use local values instead of stored in the pool instance when
    possible
  vaapi: remove unused vlc_vaapi_FilterHoldInstance()
  vdpau: chroma: create the filter output video context
  filter_chain: add function to get the current output video context
  video_output: use the intermediate output video context to init the
    next chain
  transcode: use the intermediate video context between filter chains
  chroma: chain: set the output vctx from the internal filter chain
    output vctx
  nvdec: don't keep the decoder_device_nvdec_t in the picture context
  dxva2_deinterlace: init the IDirect3DDevice9 from the video context
  d3d11_deinterlace: only create the module with a valid input video
    context
  d3d11va: don't release the d3d_dev interfaces when failing to query
    others
  dxva2_deinterlace: use the d3d9_device_t from the decoder device

 include/vlc_filter.h                          |  27 ++-
 include/vlc_picture.h                         |   6 +
 modules/codec/avcodec/d3d11va.c               |  11 +-
 modules/codec/avcodec/dxva2.c                 |   6 +-
 modules/codec/avcodec/vaapi.c                 |   2 +
 modules/codec/vt_utils.c                      |   1 +
 modules/hw/d3d11/d3d11_deinterlace.c          |  42 +++--
 modules/hw/d3d11/d3d11_surface.c              |  96 +++++++---
 modules/hw/d3d9/dxa9.c                        | 169 +++++++++++-------
 modules/hw/d3d9/dxva2_deinterlace.c           |  29 +--
 modules/hw/nvdec/chroma.c                     |  10 +-
 modules/hw/nvdec/nvdec.c                      |   5 +-
 modules/hw/nvdec/nvdec_fmt.h                  |   3 -
 modules/hw/vaapi/chroma.c                     |  34 ++--
 modules/hw/vaapi/filters.c                    |  80 +--------
 modules/hw/vaapi/filters.h                    |   8 -
 modules/hw/vaapi/vlc_vaapi.c                  |  24 +--
 modules/hw/vaapi/vlc_vaapi.h                  |   6 +-
 modules/hw/vdpau/avcodec.c                    |   2 +-
 modules/hw/vdpau/chroma.c                     |  51 +++++-
 modules/hw/vdpau/picture.c                    |   9 +-
 modules/hw/vdpau/vlc_vdpau.h                  |   4 +-
 modules/stream_out/mosaic_bridge.c            |   4 +-
 modules/stream_out/sdi/SDIStream.cpp          |   9 +-
 modules/stream_out/sdi/SDIStream.hpp          |   2 +-
 modules/stream_out/transcode/video.c          |  18 +-
 modules/video_chroma/chain.c                  |  30 +++-
 modules/video_chroma/d3d11_fmt.c              |   1 +
 modules/video_chroma/d3d9_fmt.c               |   1 +
 modules/video_filter/canvas.c                 |   4 +-
 modules/video_filter/edgedetection.c          |   4 +-
 modules/video_output/opengl/converter_vaapi.c |  11 +-
 src/libvlccore.sym                            |   1 +
 src/misc/filter_chain.c                       |  39 +++-
 src/misc/picture.c                            |   3 +
 src/video_output/display.c                    |  14 +-
 src/video_output/video_output.c               |  57 ++++--
 src/video_output/vout_internal.h              |   1 +
 38 files changed, 519 insertions(+), 305 deletions(-)

-- 
2.17.1