[vlc-devel] libswscale logo overlay crash on Win32

[Tom Maneiro]

Hello there...

I've just updated to the latest VLC 0.9.9, just to find that it
crashes when i try to use a logo overlay over the video. The picture
used was a transparent, alpha-channel PNG, and at first i thinked in
that old libpng-related bug on early 0.9.x releases... so i tried
switching libpng_plugin.dll with the one from 0.9.8a. No luck! The
crash was still there.

Then, i've used OllyDbg to break into VLC at the crash time, and found this:

Call stack of thread 00000938
Address    Stack      Procedure / arguments                 Called
from                   Frame
0477F704   6546390D   <JMP.&msvcrt.memset>                  libswsca.65463908
0477F708   00000090     s = 00000090
0477F70C   000000FF     c = FF
0477F710   00000082     n = 82 (130.)
0477F744   6545978F   Includes libswsca.6546390D            libswsca.6545978C
0477F7C4   654419C7   libswsca.654591F0
libswsca.654419C2             0477F7F4
0477F834   6544264A   libswsca.65441950
libswsca.65442645             0477F830
0477F8A4   6A5BAA59   Includes libswsca.6544264A
libvlcco.6A5BAA53             0477F8A0
0477F8D4   6A5BAD26   libvlcco.6A5BA7C0                     libvlcco.6A5BAD21
0477F924   66141C6E   Includes libvlcco.6A5BAD26
liblogo_.66141C6B             0477F9C8
0477FA34   661434BE   liblogo_.66141AE0                     liblogo_.661434B9
0477FAA4   6A5AB450   Includes liblogo_.661434BE            libvlcco.6A5AB44E
0477FB44   6A5C8C65   libvlcco.__module_Need                libvlcco.6A5C8C60
0477FB84   6A5C8EF8   libvlcco.6A5C8B90                     libvlcco.6A5C8EF3
0477FBC4   6A5C92E6   libvlcco.6A5C8E80                     libvlcco.6A5C92E1
0477FBF4   6A589AA9   libvlcco.filter_chain_AppendFromStri  libvlcco.6A589AA4
0477FC24   6A580D2A   libvlcco.spu_Init                     libvlcco.6A580D25
0477FCA4   6A581476   libvlcco.__vout_Create                libvlcco.6A581471
0477FCF4   6A561C4F   libvlcco.__vout_Request               libvlcco.6A561C4A
0477FD44   03DB4063   Includes libvlcco.6A561C4F            libavcod.03DB405D
0477FE04   6A563215   Includes libavcod.03DB4063            libvlcco.6A56320F
0477FEA4   6A5645D5   libvlcco.6A5631D0                     libvlcco.6A5645D0
0477FF54   6A564A11   libvlcco.6A563D30                     libvlcco.6A564A0C
0477FF64   6A5B1A54   Includes libvlcco.6A564A11
libvlcco.6A5B1A52             0477FFB4
0477FF84   77C0A3B0   Includes libvlcco.6A5B1A54
msvcrt.77C0A3AD               0477FFB4

The problem seems to be located in libswscale, calling memset to
location 00000090, causing an access violation exception, that crashes
VLC. Now i've tried the ol' DLL swap trick with a few
libswscale_plugin.dll versions:

- using libswscale_plugin.dll from previous VLC release (0.9.8a) WORKS
(no crashes, logo overlay works as expected)
- using libswscale_plugin.dll from latest VLC nightly (03/04/2009)
FAILS (same memcpy crash)

I tried to file this report on Trac, but for some weird reason, i
can't login there (it dies with a Firefox "redirect loop" error after
a while)..

-- 
Tom Maneiro
- tomman at gmail.com -
- http://t38.webhop.biz -