[Android] [vlc-android] H264 1080p perf issue

[Martin Storsjö]

On Wed, 20 Feb 2013, Hugues FRUCHET wrote:

> Gallery app uses OMX AndroidNativeBuffer extensions, aiming to achieve 0-copy by giving to decoder the graphic buffer details where to decode in.
> Then buffers are sent to composition stage which get access to graphic buffer details in hwcomposer HAL, so vendors can achieve 0-copy by matching their OMX decoder buffers implementation with their hw compositor buffer implementation (gralloc).
> So I was wondering if those extensions are exposed or not through IOMX, and it seems they are.
> I have found a player, MXPlayer which uses those IOMX extensions to address the hardware OMX decoder, and results are very close to what I obtain with Gallery (no more performance problem).
> I have attached the OMX logs with an h264 1080p 30 fps file, fps that I obtain on FillBufferDone is about 32ms (30fps ok).

Sure, I don't need logs to realize that it is faster with that approach.

> I have not checked the code and I don't know how they got the graphic 
> buffer handle they give in IOMX->UseBuffer, I suppose we can get it from 
> Android Surface jni stuff.
>
> Is this an option you have explored ? or are there some reasons to keep 
> buffer output (smarter YUV->RGB conversion for example, ...) ?

The big reason is that it's not totally trivial to convert the codebase to 
handle this. Yes, for the "normal" case with no subtitles or anything else 
extra on top, no filtering etc, it can be done, but adapting the whole of 
VLC to work in that case is a huge task.

So to summarize: Yes, I'm aware this can be done. Adapting VLC to it is 
not easy, that's why it hasn't been done yet.

And secondly, IOMX itself is not a public API, while there in 4.1 and 
newer is a proper public API for all this, named MediaCodec. Direct 
rendering like this is supported in that API (so it's better used effort 
to make that work than to extend the unofficial IOMX stuff), and as a 
matter of fact, I've got an insanely hacky and immature patchset (which is 
not even on a RFC level but more kind of a proof of concept) that uses 
direct rendering from MediaCodec.

// Martin