[vlc-devel] [PATCH] nvdec: fixed output chroma selection

[Quentin Chateau]

After re-reading the whole video_ouput.c file, I agree with you, it seems
to me that vout_DisableWindow should simply not release the decoder device.
I can confirm it fixes the problem.

Let me know if this is the correct way of fixing it, I'll send the patch
straight away

Le mer. 26 févr. 2020 à 13:21, Alexandre Janniaux <ajanni at videolabs.io> a
écrit :

> Hi,
>
> On Wed, Feb 26, 2020 at 12:07:14PM +0100, Quentin Chateau wrote:
> > Hi,
> >
> > what happens is that a failed decoder_UpdateVideoOutput ends up calling
> > vout_DisableWindow (at least in my case), which resets sys->dec_device.
> On
> > the second call to decoder_UpdateVideoOutput, the assert line 1576 of the
> > file src/video_output/video_output.c fails.
>
> I believe the problem lies here. Disabling a window should
> not do anything to the decoder device, as we don't even need
> the window to be enabled to use it.
>
>
> >
> > Was not sure what the VLC core behavior was supposed to be, so I assumed
> the
> > bug came from the nvdec module, which is pretty new.
> >
> > On 26/02/2020 11:03, Thomas Guillem wrote:
> > > Hello,
> > >
> > > Are you sure it's not a bug in VLC core ? A failing
> decoder_UpdateVideoOutput() should not invalidate the video context.
> > > Anyway, it need to be documented in vlc_codec.h
> > >
> > > I will wait for steve (that is AFK until Monday) before doing anything.
> > >
> > > On Wed, Feb 26, 2020, at 00:05, Quentin Chateau wrote:
> > > > The vlc_video_context needs to be re-created after
> > > > each call to decoder_UpdateVideoOutput.
> > > > ---
> > > >   modules/hw/nvdec/nvdec.c | 30 ++++++++++++++++--------------
> > > >   1 file changed, 16 insertions(+), 14 deletions(-)
> > > >
> > > > diff --git a/modules/hw/nvdec/nvdec.c b/modules/hw/nvdec/nvdec.c
> > > > index e5cda56ef3..7720da0201 100644
> > > > --- a/modules/hw/nvdec/nvdec.c
> > > > +++ b/modules/hw/nvdec/nvdec.c
> > > > @@ -912,20 +912,6 @@ static int OpenDecoder(vlc_object_t *p_this)
> > > >           goto error;
> > > >       }
> > > > -    vlc_decoder_device *dec_device = decoder_GetDecoderDevice(
> p_dec );
> > > > -    if (dec_device == NULL)
> > > > -        goto error;
> > > > -    if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
> > > > -    {
> > > > -        p_sys->vctx_out = vlc_video_context_Create( dec_device,
> > > > VLC_VIDEO_CONTEXT_NVDEC, 0, NULL );
> > > > -    }
> > > > -    vlc_decoder_device_Release(dec_device);
> > > > -    if (unlikely(p_sys->vctx_out == NULL))
> > > > -    {
> > > > -        msg_Err(p_dec, "failed to create a video context");
> > > > -        goto error;
> > > > -    }
> > > > -
> > > >       vlc_fourcc_t output_chromas[3];
> > > >       size_t chroma_idx = 0;
> > > >       if (cudaChroma == cudaVideoChromaFormat_420)
> > > > @@ -950,6 +936,20 @@ static int OpenDecoder(vlc_object_t *p_this)
> > > >       for (chroma_idx = 0; output_chromas[chroma_idx] != 0;
> chroma_idx++)
> > > >       {
> > > > +        vlc_decoder_device *dec_device =
> > > > decoder_GetDecoderDevice(p_dec);
> > > > +        if (dec_device == NULL)
> > > > +            goto error;
> > > > +        if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
> > > > +        {
> > > > +            p_sys->vctx_out = vlc_video_context_Create(dec_device,
> > > > VLC_VIDEO_CONTEXT_NVDEC, 0, NULL);
> > > > +        }
> > > > +        vlc_decoder_device_Release(dec_device);
> > > > +        if (unlikely(p_sys->vctx_out == NULL))
> > > > +        {
> > > > +            msg_Err(p_dec, "failed to create a video context");
> > > > +            goto error;
> > > > +        }
> > > > +
> > > >           p_dec->fmt_out.i_codec = p_dec->fmt_out.video.i_chroma =
> > > > output_chromas[chroma_idx];
> > > >           result = decoder_UpdateVideoOutput(p_dec, p_sys->vctx_out);
> > > >           if (result == VLC_SUCCESS)
> > > > @@ -958,6 +958,8 @@ static int OpenDecoder(vlc_object_t *p_this)
> > > >               break;
> > > >           }
> > > >           msg_Warn(p_dec, "Failed to use output chroma %4.4s",
> > > > (char*)&p_dec->fmt_out.video.i_chroma);
> > > > +        vlc_video_context_Release(p_sys->vctx_out);
> > > > +        p_sys->vctx_out = NULL;
> > > >       }
> > > >       if (result != VLC_SUCCESS)
> > > >           goto error;
> > > > --
> > > > 2.20.1
> > > >
> > > > _______________________________________________
> > > > vlc-devel mailing list
> > > > To unsubscribe or modify your subscription options:
> > > > https://mailman.videolan.org/listinfo/vlc-devel
> > > _______________________________________________
> > > vlc-devel mailing list
> > > To unsubscribe or modify your subscription options:
> > > https://mailman.videolan.org/listinfo/vlc-devel
> > _______________________________________________
> > vlc-devel mailing list
> > To unsubscribe or modify your subscription options:
> > https://mailman.videolan.org/listinfo/vlc-devel
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/vlc-devel/attachments/20200226/4f6e755a/attachment.html>