[vlc-devel] Video processing APIs
remi at remlab.net
Mon May 18 22:27:02 CEST 2009
Le lundi 18 mai 2009 21:51:51 Andre-John Mas, vous avez écrit :
> I am just curious as to the choice of VA-API and how this could
> eventually be migrated to use OpenCL. I ask the question, as OpenCL
> promises to cover more architectures than those originating from Intel.
> Of course VA-API is probably more mature at this point in time.
At this point, every one can see an API/standards war coming to the
multimedia, video and GPU processing. That seems much like we had with 3D
rendering a long time ago. Basically only OpenGL and Direct3D are left - does
anyone remember Glide and PowerVR?
At the lower level, we have at least:
- CUDA (NVidia)
- AMD FireStream (AMD/ATI, obviously...)
- DirectX 11 (Microsoft)
- OpenCL (mostly Apple, but also AMD, Intel...)
- OpenMAX development layer (about everyone and more, except Microsoft)
- whatever x86 extensions Intel will doubtless come up with
I would actually hope we (the VideoLAN project) can bypass CUDA and Firestream
as writing to a single GPU vendor seems like a huge waste of effort. On the
other hand, with differing hardware capabilities, it is not obvious that a
unifying API can really be written.
I think I cannot be blamed for being biased against Microsoft technologies.
Afterall, I wrote the /official/ if very incomplete CIL bindings for VLC. And
as one of the early user of Miredo on OSX, you know well I maintain the Teredo
stack for Unices. *But*, DxVA will unavoidably be Windows-specific, which is
unacceptable from a VideoLAN perspective. Then again, it is not unlikely we
will have to support it anyway, for lack of a credible alternative on Windows,
i.e. for the same reason as VLC implements GDI, DirectDraw and Direct3D.
OpenMAX is perhaps more focused? At least, it's only implementing some clearly
defined algorithms for video and audio decoding. It's not really programmable
in the way CUDA or OpenCL are. But it is targetted at the embedded world
mostly, as can be inferred from the presence of ARM, TI or Nokia (my current
employer). I very much doubt it will ever hit the desktop, save for reference
implementations. Similarly, if Intel x86 extension would never work on
anything x86. But, if AMD implements them, we could have a nice
complementarity with OpenMAX DL.
OpenCL is also a potentially nice option - if it is ever adopted by others
than Apple, and that might be too big an if. At least, it's not chipset-
dependent. Time will tell.
There is another factor to consider: VLC, with its "clean cut" architecture is
not friendly to intertwined decoding and rendering. Being able to decode in
normal memory would be a big win. On the other hand, it collides with the very
idea of GPU-based acceleration. And in fact, things like OpenMAX DL can be
implemented fully within FFMPEG/avcodec and other coding libraries without
touching VLC at all, since it does not have any non-trivial codec built-in.
That's very much not like the hack that XvMC support is. But that brings us to
the second category...
Slightly up the stack, it is just as messy with those challengers:
- VA-API (Intel)
- VDPAU (NVidia)
- Xvideo BA (AMD/ATI)
- DirectX VA (Microsoft)
- OpenMAX integration layer
- I wonder what Apple is doing?
And of course, we still have Xvideo Motion Compensation on older hardware.
So we basically have three X11-dependent *and* vendor-dependent solution.
Hopefully VA-API will conquer the other two, much like DxVA rules on Windows.
Otherwise it will become a terrible mess. I guess I'm missing what Apple is
doing... it just cannot be doing nothing.
The bad news is that it very much seems like we will end up with one more
triplet of OS-specific plugins. And then we may want to add OpenMAX. Also, I
guess we will need to hack the decoder/filter/output split to support any of
But we can still go higher in the stack... except we end up on top of VLC
rather than underneath it. There are (meta-)frameworks like DirectShow,
QuickTime(or is it?), Phonon, OpenMAX application layer, which VLC could
provide a back-end for. And there is LibVLC and its immediate competition:
GStreamer, xine-lib and almost FFMPEG... and the tentative "replacement"
technologies for "legacy" media players like Flash/AIR, Silverlight and
HTML5+ogg although that's completely off-topic.
Conclusion: It is a mess. Dust will probably settle after a while, but there
is a risk that we miss the train if we just sit and wait. But as usual, people
should express themselves with code, not with wishlist and whining. If you
want VLC to support foobar, then patches for foobar are typically welcome.
More information about the vlc-devel