[vlc-devel] DXVA no CPU access design

[Steve Lhomme]

It seems that we're turning around discussing solutions that never
work. So let's summarize the constraints we have to try to find a
working solution. Unless we code a DXVA2 decoder from scratch (that
may be a possibility since we have packtizers), we have to deal with
how avcodec works

1/ the avcodec decoder is opened, found to be capable of handling the
codec, and so is used.

2/ It then starts decoding the input stream.

3/ When it has enough data to write to an output frame, it first tries
to define the output format using the decoder->get_format() calback,
by providing some possible ones, including the hardware accelerated
variants. This is where we open our DXVA2 module.

4/ If we finds a va decoder, we set it up using vlc_va_Setup()

At this point the decoder is provided with a `hwaccel_context` that
contains the list of `surface` it can use to read the decoded frames
coming out of the va.

5/ The decoder decodes the input stream by using the provided hardware
accelerated surfaces provided in the Setup, via the vlc_va_Get() which
picks which surface is free to use. It gets a surface and look for the
index in its internal buffer in ff_dxva2_get_surface_index()
http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/dxva2.c;h=c1c7681402ab6615252e8e7205c1d19ea9e055d0;hb=HEAD#l38

6/ an AVFrame eventually comes out of the video decoder

The AVFrame planes/surfaces is turned into a picture_t:

7/ decoder_NewPicture() is called with the decoder_t

8/ which then calls decoder_UpdateVideoFormat() which creates the vout
if there's not one already usable.

9/ the vout is created in a different thread and the decoder thread

10/ the VOUT_CONTROL_INIT control signal is pushed and then the
decoder thread wait for the control message to be finished handling,
at which point the vout is created (or won't be). The vout is created
though a vout_configuration_t structure.

11/ a picture_t is created in decoder_NewPicture().

Because of #5 we cannot go easily around the design of avcodec. It
needs to know the table of surfaces that will be used by the decoded.
The vout is only involved after the first AVFrame with a DXVA surface
comes out of the decoder.


For these surfaces to be used to render, the D3D9 output needs to use
the D3D9 handle and the same D3D9 device as the DXVA decoder. So at
least these 2 objects need to be passed along the way. They are D3D
objects so the receiving vout can add/release references to these
objects so there won't be dereferencing issues of these pointers
during the lifetime of the module(s).

The main problem we have is how to share these objects in a structure
that is safe from dereferencing between #8 and #11 (and also when the
vout is reinited). IMO vout_configuration_t and the pointers inside is
safe during the duration of the vout create/reinit call. The vout
should not keep a pointer passed through there though.

Right now I pass a picture_pool_setup_t* through there. It could as
well be a decoder_t* or some other objects. A decoder_t* might be
better as it's a VLC object so a reference can be kept for safety. But
in my case I don't even need a reference, I just need to keep the D3D9
handle and the D3D9 device used by the decoder.

I think this is safe, but we can define the proper nature of data
passed through there.