[vlc-devel] [PATCH 0/8] External pool cancelation

[Steve Lhomme]

When closing the decoder, it may be waiting a picture in decoder thread(s).
To be able to deblock this wait the picture pool has picture_pool_Cancel().
The idea that the decoder can by pass what's going on in the pool. But now that
we are using a push model, the pool and/or the wait may be done on a pool
unknown to the decoder.

Because of this we currently have a deadlock when using hardware decoders that
wait on their own (surface) pools. This patchset proposes a solution to this
issue by keeping the "object" to cancel in decoder.c and passing it to decoders
so they can plug it to their own pool. The input/decoder thread cancels this
"object" that will unblock whatever pool is using it.

A proof of concept is done with nvdec, I have code working for the other VA
modules using this technique, but let's first get nvdec fixed.

Decoders that will use this external "vlc_cancelable" have to declare they are
using their own pool. That will also avoid creating the decoder->out_pool for
nothing. Each decoder can better manage the size of the pool it needs.

Steve Lhomme (8):
  decoder: add a flag allowing decoders to use their own picture pool
  picture_pool: use a separate API to cancel pending picture requests
  picture_pool: use the internal condition/lock indirectly
  picture_pool: initialize with an external cancelable structure
  picture_pool: fix indentation
  decoder: use internal cancelable structure to cancel the picture pool
    on exit
  nvdec: use a surface pool for the decoder to CPU
  nvdec: use our own picture that is cancelable externally

 include/vlc_codec.h             | 10 ++++
 include/vlc_picture_pool.h      | 35 +++++++++++++-
 modules/codec/qsv.c             |  2 +-
 modules/hw/nvdec/nvdec.c        | 63 ++++++++++++++++++-------
 src/input/decoder.c             | 23 +++++----
 src/input/decoder_helpers.c     |  1 +
 src/misc/picture_pool.c         | 83 ++++++++++++++++++++++-----------
 src/test/picture_pool.c         |  4 +-
 src/video_output/display.c      |  2 +-
 src/video_output/vout_wrapper.c |  3 +-
 10 files changed, 165 insertions(+), 61 deletions(-)

-- 
2.26.2