[vlc-devel] [PATCH] nvdec: fixed output chroma selection

[Quentin Chateau]

Hi,

what happens is that a failed decoder_UpdateVideoOutput ends up calling 
vout_DisableWindow (at least in my case), which resets sys->dec_device. 
On the second call to decoder_UpdateVideoOutput, the assert line 1576 of 
the file src/video_output/video_output.c fails.

Was not sure what the VLC core behavior was supposed to be, so I assumed 
the bug came from the nvdec module, which is pretty new.

On 26/02/2020 11:03, Thomas Guillem wrote:
> Hello,
>
> Are you sure it's not a bug in VLC core ? A failing decoder_UpdateVideoOutput() should not invalidate the video context.
> Anyway, it need to be documented in vlc_codec.h
>
> I will wait for steve (that is AFK until Monday) before doing anything.
>
> On Wed, Feb 26, 2020, at 00:05, Quentin Chateau wrote:
>> The vlc_video_context needs to be re-created after
>> each call to decoder_UpdateVideoOutput.
>> ---
>>   modules/hw/nvdec/nvdec.c | 30 ++++++++++++++++--------------
>>   1 file changed, 16 insertions(+), 14 deletions(-)
>>
>> diff --git a/modules/hw/nvdec/nvdec.c b/modules/hw/nvdec/nvdec.c
>> index e5cda56ef3..7720da0201 100644
>> --- a/modules/hw/nvdec/nvdec.c
>> +++ b/modules/hw/nvdec/nvdec.c
>> @@ -912,20 +912,6 @@ static int OpenDecoder(vlc_object_t *p_this)
>>           goto error;
>>       }
>>   
>> -    vlc_decoder_device *dec_device = decoder_GetDecoderDevice( p_dec );
>> -    if (dec_device == NULL)
>> -        goto error;
>> -    if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
>> -    {
>> -        p_sys->vctx_out = vlc_video_context_Create( dec_device,
>> VLC_VIDEO_CONTEXT_NVDEC, 0, NULL );
>> -    }
>> -    vlc_decoder_device_Release(dec_device);
>> -    if (unlikely(p_sys->vctx_out == NULL))
>> -    {
>> -        msg_Err(p_dec, "failed to create a video context");
>> -        goto error;
>> -    }
>> -
>>       vlc_fourcc_t output_chromas[3];
>>       size_t chroma_idx = 0;
>>       if (cudaChroma == cudaVideoChromaFormat_420)
>> @@ -950,6 +936,20 @@ static int OpenDecoder(vlc_object_t *p_this)
>>   
>>       for (chroma_idx = 0; output_chromas[chroma_idx] != 0; chroma_idx++)
>>       {
>> +        vlc_decoder_device *dec_device =
>> decoder_GetDecoderDevice(p_dec);
>> +        if (dec_device == NULL)
>> +            goto error;
>> +        if (dec_device->type == VLC_DECODER_DEVICE_NVDEC)
>> +        {
>> +            p_sys->vctx_out = vlc_video_context_Create(dec_device,
>> VLC_VIDEO_CONTEXT_NVDEC, 0, NULL);
>> +        }
>> +        vlc_decoder_device_Release(dec_device);
>> +        if (unlikely(p_sys->vctx_out == NULL))
>> +        {
>> +            msg_Err(p_dec, "failed to create a video context");
>> +            goto error;
>> +        }
>> +
>>           p_dec->fmt_out.i_codec = p_dec->fmt_out.video.i_chroma =
>> output_chromas[chroma_idx];
>>           result = decoder_UpdateVideoOutput(p_dec, p_sys->vctx_out);
>>           if (result == VLC_SUCCESS)
>> @@ -958,6 +958,8 @@ static int OpenDecoder(vlc_object_t *p_this)
>>               break;
>>           }
>>           msg_Warn(p_dec, "Failed to use output chroma %4.4s",
>> (char*)&p_dec->fmt_out.video.i_chroma);
>> +        vlc_video_context_Release(p_sys->vctx_out);
>> +        p_sys->vctx_out = NULL;
>>       }
>>       if (result != VLC_SUCCESS)
>>           goto error;
>> -- 
>> 2.20.1
>>
>> _______________________________________________
>> vlc-devel mailing list
>> To unsubscribe or modify your subscription options:
>> https://mailman.videolan.org/listinfo/vlc-devel
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel