[vlc-devel] [BUG] nvdec: use after free

[Quentin Chateau]

Thank, I'll take a look at va_surface

On 24/03/2020 09:10, Steve Lhomme wrote:
> Hi,
>
> On 2020-03-23 19:02, Quentin Chateau wrote:
>> Hi,
>>
>> I've identified a bug in the nvdec decoder but I have absolutely no 
>> idea as to what is the correct way of fixing it. I've already sent an 
>> email about this but as nobody answered it, I'm trying again.
>>
>> The decoder contains a pool of CUDA memory:
>>
>> CUdeviceptr                 outputDevicePtr[MAX_POOL_SIZE];
>>
>> This memory is allocated and freed by the decoder. Pictured decoded 
>> by the decoder contain a pointer to this resource:
>>
>> picctx->devicePtr = p_sys->outputDevicePtr[pool_idx];
>>
>> The issue is that when a video ends, the decoder can be released 
>> before all decoded pictures have been consumed by the rest of the 
>> video pipeline. This results in other plugins using already freed 
>> CUDA memory. A visible cause of this is: when using the GPU/CPU 
>> conversion filter, the last frame of the video often logs:
>>
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: 
>> invalid argument (CUDA_ERROR_INVALID_VALUE)
>>
>> My problem is that I can't figure out how to resolve this. It feels 
>> to me that the CUDA memory pool lifetime management does no belong to 
>> the decoder, and yet it needs to allocate/reallocate it depending on 
>> the frame format.
>>
>> Any help would be welcome.
>
> This was designed before we had the whole push system done. The 
> problem you describe indeed exists and was fixed in other decoders. 
> The general solution is not to pre-allocate pictures in the decoders, 
> only the surfaces. Then when you output a picture you allocate it 
> there and set the picture->context which will have a release callback. 
> The last picture displayed should release the last reference to the 
> surface(s) and release everything. There is a va_surface helper/API 
> that helps handle the refcounting.
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel