[vlc-devel] [PATCH] nvdec: fixed output chroma selection

[Thomas Guillem]

On Wed, Feb 26, 2020, at 12:07, Quentin Chateau wrote:
> Hi,
> 
> what happens is that a failed decoder_UpdateVideoOutput ends up calling 
> vout_DisableWindow (at least in my case), which resets sys->dec_device. 
> On the second call to decoder_UpdateVideoOutput, the assert line 1576 of 
> the file src/video_output/video_output.c fails.
> 
> Was not sure what the VLC core behavior was supposed to be, so I assumed 
> the bug came from the nvdec module, which is pretty new.

I think the problem is on the decoder_UpdateVideoOutput().

Let's wait for Steve insight.

> 
> On 26/02/2020 11:03, Thomas Guillem wrote:
> > Hello,
> >
> > Are you sure it's not a bug in VLC core ? A failing decoder_UpdateVideoOutput() should not invalidate the video context.
> > Anyway, it need to be documented in vlc_codec.h
> >
> > I will wait for steve (that is AFK until Monday) before doing anything.
> >
> > On Wed, Feb 26, 2020, at 00:05, Quentin Chateau wrote:
> >> The vlc_video_context needs to be re-created after
> >> each call to decoder_UpdateVideoOutput.
> >> ---
> >>   modules/hw/nvdec/nvdec.c | 30 ++++++++++++++++--------------
> >>   1 file changed, 16 insertions(+), 14 deletions(-)
> >>
> >> diff --git a/modules/hw/nvdec/nvdec.c b/modules/hw/nvdec/nvdec.c
> >> index e5cda56ef3..7720da0201 100644
> >> --- a/modules/hw/nvdec/nvdec.c
> >> +++ b/modules/hw/nvdec/nvdec.c
> >> @@ -912,20 +912,6 @@ static int OpenDecoder(vlc_object_t *p_this)
> >>           goto error;
> >>       }
> >>   
> >> -    vlc_decoder_device *dec_device = decoder_GetDecoderDevice( p_dec );
> >> -    if (dec_device == NULL)
> >> -        goto error;
> >> -    if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
> >> -    {
> >> -        p_sys->vctx_out = vlc_video_context_Create( dec_device,
> >> VLC_VIDEO_CONTEXT_NVDEC, 0, NULL );
> >> -    }
> >> -    vlc_decoder_device_Release(dec_device);
> >> -    if (unlikely(p_sys->vctx_out == NULL))
> >> -    {
> >> -        msg_Err(p_dec, "failed to create a video context");
> >> -        goto error;
> >> -    }
> >> -
> >>       vlc_fourcc_t output_chromas[3];
> >>       size_t chroma_idx = 0;
> >>       if (cudaChroma == cudaVideoChromaFormat_420)
> >> @@ -950,6 +936,20 @@ static int OpenDecoder(vlc_object_t *p_this)
> >>   
> >>       for (chroma_idx = 0; output_chromas[chroma_idx] != 0; chroma_idx++)
> >>       {
> >> +        vlc_decoder_device *dec_device =
> >> decoder_GetDecoderDevice(p_dec);
> >> +        if (dec_device == NULL)
> >> +            goto error;
> >> +        if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
> >> +        {
> >> +            p_sys->vctx_out = vlc_video_context_Create(dec_device,
> >> VLC_VIDEO_CONTEXT_NVDEC, 0, NULL);
> >> +        }
> >> +        vlc_decoder_device_Release(dec_device);
> >> +        if (unlikely(p_sys->vctx_out == NULL))
> >> +        {
> >> +            msg_Err(p_dec, "failed to create a video context");
> >> +            goto error;
> >> +        }
> >> +
> >>           p_dec->fmt_out.i_codec = p_dec->fmt_out.video.i_chroma =
> >> output_chromas[chroma_idx];
> >>           result = decoder_UpdateVideoOutput(p_dec, p_sys->vctx_out);
> >>           if (result == VLC_SUCCESS)
> >> @@ -958,6 +958,8 @@ static int OpenDecoder(vlc_object_t *p_this)
> >>               break;
> >>           }
> >>           msg_Warn(p_dec, "Failed to use output chroma %4.4s",
> >> (char*)&p_dec->fmt_out.video.i_chroma);
> >> +        vlc_video_context_Release(p_sys->vctx_out);
> >> +        p_sys->vctx_out = NULL;
> >>       }
> >>       if (result != VLC_SUCCESS)
> >>           goto error;
> >> -- 
> >> 2.20.1
> >>
> >> _______________________________________________
> >> vlc-devel mailing list
> >> To unsubscribe or modify your subscription options:
> >> https://mailman.videolan.org/listinfo/vlc-devel
> > _______________________________________________
> > vlc-devel mailing list
> > To unsubscribe or modify your subscription options:
> > https://mailman.videolan.org/listinfo/vlc-devel
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel