[vlc-devel] [PATCH] Fixed a crash caused by yadif deinterlacer on Windows XP

[Naohiro KORIYAMA]

Hello,

2012/1/9 Rafaël Carré <funman at videolan.org>:
> Hello,
>
> Le 12-01-08 00:52, Naohiro KORIYAMA a écrit :
>> The old patch, I sent to vlc-devel ML yesterday, didn't help the crash
>> on Windows XP 32bit.
>> I was wrong that I tested with the module that was compiled with a
>> different patch.
>>
>> So I made a new patch and tested again
>>  (before applying old patch, after applying old patch, after applying
>> new patch):
>> - Windows XP 32bit using XP mode on Windows 7 (win32)
>>   NG->NG->OK
>> - Windows 7 64bit (win32/win64)
>>   OK->OK->OK
>> - Mac OS X Lion (intel64)
>>   OK->OK->OK
>> - Ubuntu 11.10 32bit using VirtualBox (32)
>>   OK->OK->OK
>> - Ubuntu 11.10 64bit using VirtualBox (64)
>>   OK->OK->OK
>>
>> It's all OK with the new patch.
>> But as jb mentioned on the ML few days ago, compiler warnings appear again
>> on 32bit builds both Windows and Linux.
>
> These warnings might be fixed by using uintptr_t instead of uint64_t.
>
> Perhaps you noticed but this patch broke compile on OSX 32 bits and
> FreeBSD 32 bits : http://buildbot.videolan.org/builders/
>
> OSX says:
>
> ../../../modules/video_filter/deinterlace/yadif_template.h:245: error:
> can't find a register in class 'GENERAL_REGS' while reloading 'asm'
> ../../../modules/video_filter/deinterlace/yadif_template.h:251: error:
> can't find a register in class 'GENERAL_REGS' while reloading 'asm'
> ../../../modules/video_filter/deinterlace/yadif_template.h:245: error:
> 'asm' operand has impossible constraints
> ../../../modules/video_filter/deinterlace/yadif_template.h:251: error:
> 'asm' operand has impossible constraints
>
> and FreeBSD:
> cc1: error in backend: Ran out of registers during register allocation!
> and other stuff.

I heard OS X 32 bit build has a problem, and I'm investigating this issue.

>
> I suggest reverting the patch and doing something like:
>
> uint8_t tmp[5*16];
> uint8_t *tmpA= (uint8_t*)(((uintptr_t)(tmp+15)) & ~15);
> uint8_t *tmp0 = &tmpA[0]; uint8_t *tmp1 = &tmpA[16] ...
>
> what do you think?
> It would use the old version of the code and ensure that tmp* are aligned.

Modifying the yadif code with your suggestion, on OS X 64bit, Windows
32/64bit, Linux 32/64bit environment, it worked.
Of course the issue, yafdif crash win32 build on Windows XP, is fixed, too.

If it works on OS X 32bit and FreeBSD 32bit, I think it's ok. I don't
have environments to check it on OS X 32bit/FreeBSD immediately.

> Btw any idea why alignement doesn't work? Is it only for static data and
> not for stack variables?
>
> Other functions use (and rely on?) alignement in VLC so it should be
> important to know what's going on there


-- 
KORIYAMA, Naohiro
nkoriyama at gmail.com