[vlc-devel] Video processing APIs

[Rémi Denis-Courmont]

Le lundi 18 mai 2009 21:51:51 Andre-John Mas, vous avez écrit :
> I am just curious as to the choice of VA-API and how this could
> eventually be migrated to use OpenCL. I ask the question, as OpenCL
> promises to cover more architectures than those originating from Intel.
> Of course VA-API is probably more mature at this point in time.

At this point, every one can see an API/standards war coming to the 
multimedia, video and GPU processing. That seems much like we had with 3D 
rendering a long time ago. Basically only OpenGL and Direct3D are left - does 
anyone remember Glide and PowerVR?


At the lower level, we have at least:
 - CUDA (NVidia)
 - AMD FireStream (AMD/ATI, obviously...)
 - DirectX 11 (Microsoft)
 - OpenCL (mostly Apple, but also AMD, Intel...)
 - OpenMAX development layer (about everyone and more, except Microsoft)
 - whatever x86 extensions Intel will doubtless come up with

I would actually hope we (the VideoLAN project) can bypass CUDA and Firestream 
as writing to a single GPU vendor seems like a huge waste of effort. On the 
other hand, with differing hardware capabilities, it is not obvious that a 
unifying API can really be written.

I think I cannot be blamed for being biased against Microsoft technologies. 
Afterall, I wrote the /official/ if very incomplete CIL bindings for VLC. And 
as one of the early user of Miredo on OSX, you know well I maintain the Teredo 
stack for Unices. *But*, DxVA will unavoidably be Windows-specific, which is 
unacceptable from a VideoLAN perspective. Then again, it is not unlikely we 
will have to support it anyway, for lack of a credible alternative on Windows, 
i.e. for the same reason as VLC implements GDI, DirectDraw and Direct3D.

OpenMAX is perhaps more focused? At least, it's only implementing some clearly 
defined algorithms for video and audio decoding. It's not really programmable 
in the way CUDA or OpenCL are. But it is targetted at the embedded world 
mostly, as can be inferred from the presence of ARM, TI or Nokia (my current 
employer). I very much doubt it will ever hit the desktop, save for reference 
implementations. Similarly, if Intel x86 extension would never work on 
anything x86. But, if AMD implements them, we could have a nice 
complementarity with OpenMAX DL.
OpenCL is also a potentially nice option - if it is ever adopted by others 
than Apple, and that might be too big an if. At least, it's not chipset-
dependent. Time will tell.

There is another factor to consider: VLC, with its "clean cut" architecture is 
not friendly to intertwined decoding and rendering. Being able to decode in 
normal memory would be a big win. On the other hand, it collides with the very 
idea of GPU-based acceleration. And in fact, things like OpenMAX DL can be 
implemented fully within FFMPEG/avcodec and other coding libraries without 
touching VLC at all, since it does not have any non-trivial codec built-in. 
That's very much not like the hack that XvMC support is. But that brings us to 
the second category...

Slightly up the stack, it is just as messy with those challengers:
 - VA-API (Intel)
 - VDPAU (NVidia)
 - Xvideo BA (AMD/ATI)
 - DirectX VA (Microsoft)
 - OpenMAX integration layer
 - I wonder what Apple is doing?
And of course, we still have Xvideo Motion Compensation on older hardware.

So we basically have three X11-dependent *and* vendor-dependent solution. 
Hopefully VA-API will conquer the other two, much like DxVA rules on Windows. 
Otherwise it will become a terrible mess. I guess I'm missing what Apple is 
doing... it just cannot be doing nothing.

The bad news is that it very much seems like we will end up with one more 
triplet of OS-specific plugins. And then we may want to add OpenMAX. Also, I 
guess we will need to hack the decoder/filter/output split to support any of 
those.

But we can still go higher in the stack... except we end up on top of VLC 
rather than underneath it. There are (meta-)frameworks like DirectShow, 
QuickTime(or is it?), Phonon, OpenMAX application layer, which VLC could 
provide a back-end for. And there is LibVLC and its immediate competition: 
GStreamer, xine-lib and almost FFMPEG... and the tentative "replacement" 
technologies for "legacy" media players like Flash/AIR, Silverlight and 
HTML5+ogg although that's completely off-topic.


Conclusion: It is a mess. Dust will probably settle after a while, but there 
is a risk that we miss the train if we just sit and wait. But as usual, people 
should express themselves with code, not with wishlist and whining. If you 
want VLC to support foobar, then patches for foobar are typically welcome.

-- 
Rémi Denis-Courmont
http://www.remlab.net/