[vlc-devel] [PATCH] Fixed a crash caused by yadif deinterlacer on Windows XP

[Naohiro KORIYAMA]

I'm sorry, please igore this mail, ATM.
I didn't want to send it but save it...

2012/1/9 Naohiro KORIYAMA <nkoriyama at gmail.com>:
> Hello,
>
> 2012/1/9 Rafaël Carré <funman at videolan.org>:
>> Hello,
>>
>> Le 12-01-08 00:52, Naohiro KORIYAMA a écrit :
>>> The old patch, I sent to vlc-devel ML yesterday, didn't help the crash
>>> on Windows XP 32bit.
>>> I was wrong that I tested with the module that was compiled with a
>>> different patch.
>>>
>>> So I made a new patch and tested again
>>>  (before applying old patch, after applying old patch, after applying
>>> new patch):
>>> - Windows XP 32bit using XP mode on Windows 7 (win32)
>>>   NG->NG->OK
>>> - Windows 7 64bit (win32/win64)
>>>   OK->OK->OK
>>> - Mac OS X Lion (intel64)
>>>   OK->OK->OK
>>> - Ubuntu 11.10 32bit using VirtualBox (32)
>>>   OK->OK->OK
>>> - Ubuntu 11.10 64bit using VirtualBox (64)
>>>   OK->OK->OK
>>>
>>> It's all OK with the new patch.
>>> But as jb mentioned on the ML few days ago, compiler warnings appear again
>>> on 32bit builds both Windows and Linux.
>>
>> These warnings might be fixed by using uintptr_t instead of uint64_t.
>>
>> Perhaps you noticed but this patch broke compile on OSX 32 bits and
>> FreeBSD 32 bits : http://buildbot.videolan.org/builders/
>>
>> OSX says:
>>
>> ../../../modules/video_filter/deinterlace/yadif_template.h:245: error:
>> can't find a register in class 'GENERAL_REGS' while reloading 'asm'
>> ../../../modules/video_filter/deinterlace/yadif_template.h:251: error:
>> can't find a register in class 'GENERAL_REGS' while reloading 'asm'
>> ../../../modules/video_filter/deinterlace/yadif_template.h:245: error:
>> 'asm' operand has impossible constraints
>> ../../../modules/video_filter/deinterlace/yadif_template.h:251: error:
>> 'asm' operand has impossible constraints
>>
>> and FreeBSD:
>> cc1: error in backend: Ran out of registers during register allocation!
>> and other stuff.
>
> I heard OS X 32 bit build has a problem, and I'm investigating this issue.
>
>>
>> I suggest reverting the patch and doing something like:
>>
>> uint8_t tmp[5*16];
>> uint8_t *tmpA= (uint8_t*)(((uintptr_t)(tmp+15)) & ~15);
>> uint8_t *tmp0 = &tmpA[0]; uint8_t *tmp1 = &tmpA[16] ...
>>
>> what do you think?
>> It would use the old version of the code and ensure that tmp* are aligned.
>
> Modifying the yadif code with your suggestion, on OS X 64bit, Windows
> 32/64bit, Linux 32/64bit environment, it worked.
> Of course the issue, yafdif crash win32 build on Windows XP, is fixed, too.
>
> If it works on OS X 32bit and FreeBSD 32bit, I think it's ok. I don't
> have environments to check it on OS X 32bit/FreeBSD immediately.
>
>> Btw any idea why alignement doesn't work? Is it only for static data and
>> not for stack variables?
>>
>> Other functions use (and rely on?) alignement in VLC so it should be
>> important to know what's going on there
>
>
> --
> KORIYAMA, Naohiro
> nkoriyama at gmail.com



-- 
KORIYAMA, Naohiro
nkoriyama at gmail.com