[vlc-devel] [BUG] nvdec: use after free

[Steve Lhomme]

Hi,

On 2020-03-23 19:02, Quentin Chateau wrote:
> Hi,
> 
> I've identified a bug in the nvdec decoder but I have absolutely no idea 
> as to what is the correct way of fixing it. I've already sent an email 
> about this but as nobody answered it, I'm trying again.
> 
> The decoder contains a pool of CUDA memory:
> 
> CUdeviceptr                 outputDevicePtr[MAX_POOL_SIZE];
> 
> This memory is allocated and freed by the decoder. Pictured decoded by 
> the decoder contain a pointer to this resource:
> 
> picctx->devicePtr = p_sys->outputDevicePtr[pool_idx];
> 
> The issue is that when a video ends, the decoder can be released before 
> all decoded pictures have been consumed by the rest of the video 
> pipeline. This results in other plugins using already freed CUDA memory. 
> A visible cause of this is: when using the GPU/CPU conversion filter, 
> the last frame of the video often logs:
> 
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> [00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
> 
> My problem is that I can't figure out how to resolve this. It feels to 
> me that the CUDA memory pool lifetime management does no belong to the 
> decoder, and yet it needs to allocate/reallocate it depending on the 
> frame format.
> 
> Any help would be welcome.

This was designed before we had the whole push system done. The problem 
you describe indeed exists and was fixed in other decoders. The general 
solution is not to pre-allocate pictures in the decoders, only the 
surfaces. Then when you output a picture you allocate it there and set 
the picture->context which will have a release callback. The last 
picture displayed should release the last reference to the surface(s) 
and release everything. There is a va_surface helper/API that helps 
handle the refcounting.