[vlc-commits] decoder: fix forgotten vout error state

[Thomas Guillem]

vlc | branch: master | Thomas Guillem <thomas at gllm.fr> | Wed Dec 16 16:43:31 2020 +0100| [0abed25515a492896c89cf36ab1e4befc7dd23b6] | committer: Thomas Guillem

decoder: fix forgotten vout error state

If the pool allocation fails or if the vout fails to start (via
input_resource_RequestVout()), an error is returned from
decoder_UpdateVideoFormat(). The next call to
decoder_UpdateVideoFormat() will return a success because the vctx will
be the same since only this variable is checked to detect a vout change.

To fix this issue, clean the owner->fmt and release the vctx when
returning an error.

> http://git.videolan.org/gitweb.cgi/vlc.git/?a=commit;h=0abed25515a492896c89cf36ab1e4befc7dd23b6
---

 src/input/decoder.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/src/input/decoder.c b/src/input/decoder.c
index 94e328f040..46bb17a384 100644
--- a/src/input/decoder.c
+++ b/src/input/decoder.c
@@ -457,7 +457,7 @@ static int ModuleThread_UpdateVideoFormat( decoder_t *p_dec, vlc_video_context *
             msg_Err(p_dec, "Failed to create a pool of %d %4.4s pictures",
                            dpb_size + p_dec->i_extra_picture_buffers + 1,
                            (char*)&p_dec->fmt_out.video.i_chroma);
-            return -1;
+            goto error;
         }
 
         vlc_mutex_lock( &p_owner->lock );
@@ -481,6 +481,18 @@ static int ModuleThread_UpdateVideoFormat( decoder_t *p_dec, vlc_video_context *
         return 0;
     }
 
+error:
+    /* Clean fmt and vctx to trigger a new vout creation on the next update
+     * call */
+    vlc_mutex_lock( &p_owner->lock );
+    es_format_Clean( &p_owner->fmt );
+    vlc_mutex_unlock( &p_owner->lock );
+
+    if (p_owner->vctx != NULL)
+    {
+        vlc_video_context_Release(p_owner->vctx);
+        p_owner->vctx = NULL;
+    }
     return -1;
 }