[vlc-devel] [PATCH] decoder: fix pause+close deadlock for decoders waiting internally for buffers

[Steve Lhomme]

When paused the decoder decodes until it exhausts its available buffers then
waits until new buffers are available. When seeking we force flushing the vout
which releases the buffers the decoder is waiting for.

When closing while pause we didn't flush the vout so the decoder was still
waiting for buffers even though it's not going to use them (or display them).
So we do the same flushing to automatically release the used/pending buffers.
---
 src/input/decoder.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/src/input/decoder.c b/src/input/decoder.c
index 141120d6395..464ceff046e 100644
--- a/src/input/decoder.c
+++ b/src/input/decoder.c
@@ -2153,6 +2153,22 @@ void vlc_input_decoder_Delete( vlc_input_decoder_t *p_owner )
     {
         if (p_owner->out_pool)
             picture_pool_Cancel( p_owner->out_pool, true );
+
+        if( p_owner->paused )
+        {
+            /* The DecoderThread could be stuck in pf_decode(). This is likely the
+            * case with paused asynchronous decoder modules that have a limited
+            * input and output pool size. Indeed, with such decoders, you have to
+            * release an output buffer to get an input buffer. So, when paused and
+            * flushed, the DecoderThread could be waiting for an output buffer to
+            * be released (or rendered). In that case, the DecoderThread will
+            * never be flushed since it be never leave pf_decode(). To fix this
+            * issue, pre-flush the vout from here. The vout will have to be
+            * flushed again since the module could be outputting more buffers just
+            * after being unstuck. */
+
+            vout_FlushAll( p_owner->p_vout );
+        }
     }
     vlc_mutex_unlock( &p_owner->lock );
 
-- 
2.26.2