[vlc-devel] [BUG] nvdec: use after free

Quentin Chateau quentin.chateau at deepskycorp.com
Mon Mar 23 19:02:57 CET 2020


Hi,

I've identified a bug in the nvdec decoder but I have absolutely no idea 
as to what is the correct way of fixing it. I've already sent an email 
about this but as nobody answered it, I'm trying again.

The decoder contains a pool of CUDA memory:

CUdeviceptr                 outputDevicePtr[MAX_POOL_SIZE];

This memory is allocated and freed by the decoder. Pictured decoded by 
the decoder contain a pointer to this resource:

picctx->devicePtr = p_sys->outputDevicePtr[pool_idx];

The issue is that when a video ends, the decoder can be released before 
all decoded pictures have been consumed by the rest of the video 
pipeline. This results in other plugins using already freed CUDA memory. 
A visible cause of this is: when using the GPU/CPU conversion filter, 
the last frame of the video often logs:

[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)
[00007f0280a3c570] chroma filter error: cuMemcpy2DAsync failed: invalid argument (CUDA_ERROR_INVALID_VALUE)

My problem is that I can't figure out how to resolve this. It feels to 
me that the CUDA memory pool lifetime management does no belong to the 
decoder, and yet it needs to allocate/reallocate it depending on the 
frame format.

Any help would be welcome.


Quentin CHATEAU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/vlc-devel/attachments/20200323/eb98a1a6/attachment.html>


More information about the vlc-devel mailing list