[Android] A word on release and audio

[Rafaël Carré]

Hello,

I think we should make a new release as soon as possible.

Since the opensles is the only module handling audio/video
synchronization, I think it should be the default.

It is certainly not perfect as noticed by j-b and etix last month, since
*some* samples show lipsync issue.


Unfortunately I have not enough material to make a fix and I think
releasing with that module as default would give precious user feedback.


That last patch I tried seems to change the behaviour of opensl module
but unfortunately I have no idea why (see the end of the mail for an
explanation).


IMO releasing now would have 2 benefits:
- testing 2 months of android.git commits that you probably all expect.
- give me useful feedback about opensl:

- if users say it's worse, i'll accept the statistical evidence and dig
the code to find what i did wrong
- if users say it's better, i'll feel confident that i did something
useful and will look to further improve it
- if users say it's the same, i'll feel good about not breaking things
too much.

-----------------------------------------------------------------------

Now j-b and etix will think I distrust them for asking user feedback
instead of relying on their own feedback, but the truth is more nuanced.

There is a problem, I acknowledged it already as soon as you made that
report. I certainly trust what you say when you told me you had a
lipsync issue.

Which one?
I don't know and I am sorry to tell you your testing didn't make the
issue more clear.


I will be honest here, I also had some concerns with your report:

- I had to ask for a sample, while I expected the sample to be provided
with the report (we need the exact same sample to reproduce the exact
same bug)

http://www.catb.org/esr/faqs/smart-questions.html#beprecise
"If at all possible, provide a way to reproduce the problem in a
controlled environment."

IMO, using the exact same file sample is key to "controlled environment"
and I expected you had the same interpretation than me on this one.

- After complaining that I needed more samples I was promised other ones.
I'm still waiting for them today despite claims that "it's broken for a
lot of files".

- The issue that I was described is not the one that I could finally
reproduce with the help of etix at FOSDEM (sync appearing only at the
end of movie versus appearing immediately at the start).
If we don't have the same symptoms I can not be sure we are experiencing
the same bug.
Now they *could* be the same bug but before I confirm it.

As the only developer working on the audio code I feel entitled to
labeling the bug reports.
If you ever closed a bug report on trac as incomplete or can't
reproduce, I guess you will sympathize with me.

Also I don't have a problem being alone working on this, I know manpower
isn't an infinite ressource.

-----------------------------------------------------------------------

Maybe it is my fault for not asking precisely what I wanted to be tested
(like different sampling rates on the same device, same video codec,
same container).


I am also very aware that you are very busy at the moment, but that does
not make this report more useful to me.

A lot of work is obviously needed to identify and fix this problem since
all of us spent hours already on it and the bug is still here.


Now I still want this to be fixed, and I still can't do it without your
testing.


Here is what I suggest:
- documenting tests on a trac ticket so we can find them later
- uploading each and every sample tested to a common location so other
people can reproduce the problem in the same "controlled environment".
- expliciting the devices tested. I don't remember the exact model or
android version running on your devices (I think now you give me this
info but I complained about it when you first reported it though).
- complete mediainfo output on these files or selected essential
information (sampling rate, codec, channels, frame size)
- logcat | grep VLC - output, with the msg_Dbg uncommented in
opensles_android.c (although we should rather uncomment it in git since
we still have issues).

-----------------------------------------------------------------------

Also I was not explicit about why I think the last change helps, let me
fix that.

OPENSLES_BUFLEN is currently 4, I changed it to 10.

This define is used in 2 places:

    /* XXX: rounding shouldn't affect us at normal sampling rate */
    sys->samples_per_buf = OPENSLES_BUFLEN * fmt->i_rate / 1000;

We calculate the number of samples to be put in each opensl es buffer
(of which we have 255 available).

For 48kHz we have:
samples_per_buf = 4 * 48000 / 1000 = 192 samples
192 samples = 192 * 1000 / 48000 = 4 milliseconds

For 44.1kHz we have:
samples_per_buf = 4 * 44100 / 1000 = 176 samples
176 samples = 176 * 1000 / 44100 = 3.9909297052 milliseconds

If we add up our 255 buffers, that makes 1017.6870748260 milliseconds,
instead of our expected 4 * 255 = 1020 milliseconds

Thus it makes a difference of 2.3129251740 milliseconds in drift
calculation, way below the recommanded 60 or 80 milliseconds advised by
the DVB consortium (under these values we consider there is no lipsync
issue), thus the comment "rounding shouldn't affect us at normal
sampling rate" (where I meant 'normal' as in 44.1 or 48kHz).

It is even below the 4ms accuracy that we have.


The second place is the drift calculation itself:

    *drift = (CLOCK_FREQ * OPENSLES_BUFLEN * st.count / 1000)
        + sys->samples * CLOCK_FREQ / sys->rate;

(st.count is the number of busy buffers, between 0 and 255).


Bumping the value to 10 effectively makes 44.1kHz round nicely to 10ms =
441 samples exactly.

But it doesn't explain why a 2.3ms difference makes the lipsync go away
while our precision is limited to 4ms.

It also doesn't explain why it makes lipsync go away on the big bang
theory sample (the only one we have in common except maybe billy elliott
that I uploaded and that I think only etix tested), because this one has
48kHz audio which has no rounding error with 4ms or 10ms BUFLEN.

-----------------------------------------------------------------------

I hope this will clear any misunderstanding and remove any feeling that
I'm responsible for the lack of interest in the android work (I do not
see this lack of interest myself anyway).