[vlc-devel] [PATCH] nvdec: fixed output chroma selection

[Alexandre Janniaux]

Hi,

On Wed, Feb 26, 2020 at 12:07:14PM +0100, Quentin Chateau wrote:
> Hi,
>
> what happens is that a failed decoder_UpdateVideoOutput ends up calling
> vout_DisableWindow (at least in my case), which resets sys->dec_device. On
> the second call to decoder_UpdateVideoOutput, the assert line 1576 of the
> file src/video_output/video_output.c fails.

I believe the problem lies here. Disabling a window should
not do anything to the decoder device, as we don't even need
the window to be enabled to use it.


>
> Was not sure what the VLC core behavior was supposed to be, so I assumed the
> bug came from the nvdec module, which is pretty new.
>
> On 26/02/2020 11:03, Thomas Guillem wrote:
> > Hello,
> >
> > Are you sure it's not a bug in VLC core ? A failing decoder_UpdateVideoOutput() should not invalidate the video context.
> > Anyway, it need to be documented in vlc_codec.h
> >
> > I will wait for steve (that is AFK until Monday) before doing anything.
> >
> > On Wed, Feb 26, 2020, at 00:05, Quentin Chateau wrote:
> > > The vlc_video_context needs to be re-created after
> > > each call to decoder_UpdateVideoOutput.
> > > ---
> > >   modules/hw/nvdec/nvdec.c | 30 ++++++++++++++++--------------
> > >   1 file changed, 16 insertions(+), 14 deletions(-)
> > >
> > > diff --git a/modules/hw/nvdec/nvdec.c b/modules/hw/nvdec/nvdec.c
> > > index e5cda56ef3..7720da0201 100644
> > > --- a/modules/hw/nvdec/nvdec.c
> > > +++ b/modules/hw/nvdec/nvdec.c
> > > @@ -912,20 +912,6 @@ static int OpenDecoder(vlc_object_t *p_this)
> > >           goto error;
> > >       }
> > > -    vlc_decoder_device *dec_device = decoder_GetDecoderDevice( p_dec );
> > > -    if (dec_device == NULL)
> > > -        goto error;
> > > -    if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
> > > -    {
> > > -        p_sys->vctx_out = vlc_video_context_Create( dec_device,
> > > VLC_VIDEO_CONTEXT_NVDEC, 0, NULL );
> > > -    }
> > > -    vlc_decoder_device_Release(dec_device);
> > > -    if (unlikely(p_sys->vctx_out == NULL))
> > > -    {
> > > -        msg_Err(p_dec, "failed to create a video context");
> > > -        goto error;
> > > -    }
> > > -
> > >       vlc_fourcc_t output_chromas[3];
> > >       size_t chroma_idx = 0;
> > >       if (cudaChroma == cudaVideoChromaFormat_420)
> > > @@ -950,6 +936,20 @@ static int OpenDecoder(vlc_object_t *p_this)
> > >       for (chroma_idx = 0; output_chromas[chroma_idx] != 0; chroma_idx++)
> > >       {
> > > +        vlc_decoder_device *dec_device =
> > > decoder_GetDecoderDevice(p_dec);
> > > +        if (dec_device == NULL)
> > > +            goto error;
> > > +        if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
> > > +        {
> > > +            p_sys->vctx_out = vlc_video_context_Create(dec_device,
> > > VLC_VIDEO_CONTEXT_NVDEC, 0, NULL);
> > > +        }
> > > +        vlc_decoder_device_Release(dec_device);
> > > +        if (unlikely(p_sys->vctx_out == NULL))
> > > +        {
> > > +            msg_Err(p_dec, "failed to create a video context");
> > > +            goto error;
> > > +        }
> > > +
> > >           p_dec->fmt_out.i_codec = p_dec->fmt_out.video.i_chroma =
> > > output_chromas[chroma_idx];
> > >           result = decoder_UpdateVideoOutput(p_dec, p_sys->vctx_out);
> > >           if (result == VLC_SUCCESS)
> > > @@ -958,6 +958,8 @@ static int OpenDecoder(vlc_object_t *p_this)
> > >               break;
> > >           }
> > >           msg_Warn(p_dec, "Failed to use output chroma %4.4s",
> > > (char*)&p_dec->fmt_out.video.i_chroma);
> > > +        vlc_video_context_Release(p_sys->vctx_out);
> > > +        p_sys->vctx_out = NULL;
> > >       }
> > >       if (result != VLC_SUCCESS)
> > >           goto error;
> > > --
> > > 2.20.1
> > >
> > > _______________________________________________
> > > vlc-devel mailing list
> > > To unsubscribe or modify your subscription options:
> > > https://mailman.videolan.org/listinfo/vlc-devel
> > _______________________________________________
> > vlc-devel mailing list
> > To unsubscribe or modify your subscription options:
> > https://mailman.videolan.org/listinfo/vlc-devel
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel