[Android] How to debug A/V sync issues when dealing with OMX hardware decoders ?

[Rafaël Carré]

Hello,

Le 04/02/2013 11:24, Hugues FRUCHET a écrit :
> Hi all,
> 
> First of all, I have searched on VLC bug tracker (https://trac.videolan.org/vlc) &  VLC forum (http://forum.videolan.org/) but couldn't get much infos on how to debug such issues (mainly found some reference to audio output opensles bug fixes).
> I'm using VLC apk 0.0.8 with Android ICS and OMX hardware codec interface.
> 
> When adding support of some audio & video OMX hardware decoders in VLC, I faced 2 majors problems:
> 
> 1)      Audio drops
> Example of warning message:
> W/VLC     ( 1173): [0x14c72cc]: main audio output PTS is out of range (479247), dropping buffer
> From src/audio_output/input.c
> 
> I wanted to know what are your advices to debug this type of problem ?
> For example, I have observed with MP3, that if decoder bufferisation is too big, decoded samples are claimed as too late, but it's only a problem of latency, seems that reference time taken by VLC is not the first sample received but the first sample sent.
> Reading code, I cannot really understand how it works.
> By reducing latency of decoder, I have fixed this exact problem.
> Is there a patch or an option to set in order  to disable the checks on lateness and let audio be rendered without any check ?

No there's no such patch.

I think you should increase the audio packets PTS if you're sending them
to the output a while after decoding.

Why do you need high latency in your decoder?

> 2)      A/V sync issues with video late
> With some files, video is rendered completely out of sync regarding audio, video is late by some seconds and some video drops are observed.
> I don't have fresh log to give on this exact issue I have faced some weeks ago, but I'll restart this activity this week, and I want to restart with a better basis
> regarding VLC debug tips.
> *Have you some patch or options to set in order to disable audio and let video be rendered as fast as it can ? This would allow to test the pure performance of video chain, including OMX decoder, YUV=>RGB conversion & display. With Android Gallery apk, I don't face perf issue and A/V is in sync

I do have a patch to test decoder performace, but that completely
bypasses display/rendering. It's not very convenient as it disables
audio/video output completely (it's not configurable at runtime so you
have to use a custom patch).

I might be able to adapt it to add YUV->RGB conversion, I need to check.

BTW did you try the YV12 rendering option in advanced preferences? (It
is broken on several devices but might work for you)

> *For YUV=>RGB conversion, how can I know the performance (duration) ? is the neon optim enabled or not (trace ?)

Performance is not checked atm but we could add some counters.
NEON should be enabled, you can see it in logcat : look for "yuv_rgb_neon".

> *When reading code, I cannot found the A/V sync mechanism I usually encounter in Stagefright/AwesomePlayer.cpp (or other players I have encountered in the past) which consist to take the audio clock (system clock if no audio) as reference and render the decoded video frames when the audio clock has elapsed based on the video frames timestamp. Have you some doc which explain this ?

No there are no docs, the video output code was written by fenrir and
he's not available to explain it.

Video logic is all in src/video_output, and audio rendering
synchronization logic is performed by audio_output_t->time_get