[vlc-devel] Deadlock in freetype module

[Sergey E. Galanov]

Hello everybody.

After upgrading to the latest version of vlc in my gentoo distribution 
(0.9.0-beta3)
there has appeared a problem: while playing any video file the window 
with video
output remains black, while sound goes normally.

When, compiled in debug mode, vlc simply crashes because of mutex 
acquisition
assertion failure. This problem is caused by a deadlock situation in the 
freetype module.
Here is a code excerpt from file 'modules/misc/freetype.c':

<...> *FontBuilderAttach(...)
{
  p_lock = var_acquire_mutex("fontbuilder");
  <...>

  vlc_thread_create(FontBuilderThread);
  <...>
  if (p_fontbuilder)
  {
    var_AddCallback(p_fontbuilder, "build-done", FontBuilderDone, p_filter);
    var_TriggerCallback(p_fontbuilder, "build-done");
  }
  vlc_mutex_unlock(p_lock);
  <...>
}

<...> FontBuilderThread(...)
{
  <...>
  var_SetBool(p_this, "build-done", true);
  <...>
}

<...> FontBuilderDone(...)
{
  <...>
  if (newval.b_bool)
  {
    var_AcquireMutex("fontbuilder");
    <...>
  }
  <...>
}

Here we have an obvious deadlock: FontBuilderAttach acquires mutex 
"fontbuilder", and FontBuilderDone
which is being called from that function (via var_TriggerCallback) 
acquires the same mutex.
Perhaps, it is assumed that in this case 'newval.b_bool' is always 
false, but since it deadlocks, this assumption is wrong :)

Of course I don't have any insight into this code and contexts from 
which it may be called, but it seems to me that the
call to var_TriggerCallback is erroneous, since callback should be 
called when builder thread has finished its work. So I just
commented this call. It worked fine for me.

With this bug vlc is absolutely unusable.

Best regards, Sergey Galanov