[vlc-devel] [PATCH] decoder: unblock broken decoders when flushing

[Thomas Guillem]

MediaCodec or iomx with direct rendering can be stuck waiting for an output
buffer when video is paused since output buffers are released by vout.

A previous hack was made in order to fix a deadlock when theses broken decoders
were stuck and when exit was requested. These decoders returned an invalid
picture after a small delay, then DecoderIsExitRequested in DecoderDecodeVideo
was checked and these decoders could be exited without blocking.

This previous hack didn't fix a deadlock when video was flushed on pause.
Indeed, when stuck, these broken decoders are feed with the same block.
Therefore they won't receive a new block containing the FLUSH flags and they
will be stuck returning an invalid picture.

One way to fix this deadlock is to also check for DecoderIsFlushing in order to
break the DecoderDecodeVideo loop. If this loop is broken, the DecoderProcess
function will terminate, and a new block containing the FLUSH flags will be
given to the decoder.

(fixes #12397)
---
 src/input/decoder.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/input/decoder.c b/src/input/decoder.c
index 972522a..03fc21b 100644
--- a/src/input/decoder.c
+++ b/src/input/decoder.c
@@ -1202,7 +1202,7 @@ static void DecoderDecodeAudio( decoder_t *p_dec, block_t *p_block )
     }
     else while( (p_aout_buf = p_dec->pf_decode_audio( p_dec, &p_block )) )
     {
-        if( DecoderIsExitRequested( p_dec ) )
+        if( DecoderIsExitRequested( p_dec ) || DecoderIsFlushing( p_dec ) )
         {
             /* It prevent freezing VLC in case of broken decoder */
             block_Release( p_aout_buf );
@@ -1371,7 +1371,7 @@ static void DecoderDecodeVideo( decoder_t *p_dec, block_t *p_block )
     while( (p_pic = p_dec->pf_decode_video( p_dec, &p_block )) )
     {
         vout_thread_t  *p_vout = p_owner->p_vout;
-        if( DecoderIsExitRequested( p_dec ) )
+        if( DecoderIsExitRequested( p_dec ) || DecoderIsFlushing( p_dec ) )
         {
             /* It prevent freezing VLC in case of broken decoder */
             vout_ReleasePicture( p_vout, p_pic );
-- 
2.1.0