[x264-devel] [bug] asm operand has impossible constraints (x86, -fstack-check)

[Johannes Dewender]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


After checking and testing several things in my build environment
(like starting a git bisect for gcc) I found the actual problem:
- -fstack-check was recently added in my build environment wrappers.
This has nothing to do with gcc or kernel updates. Details below.

On 12/08/15 04:20, Loren Merritt wrote:
> On Mon, 10 Aug 2015, Johannes Dewender wrote:
>> yasm -I. -I. -DARCH_X86_64=0 -I./common/x86/ -f elf32 
>> -Worphan-labels -DSTACK_ALIGNMENT=32 -DPIC -DHIGH_BIT_DEPTH=0 
>> -DBIT_DEPTH=8 -o common/x86/pixel-32.o common/x86/pixel-32.asm
> 
> Btw, this isn't the command that produced the error. You're 
> probably using multithreaded make, which means that compilation 
> messages aren't necessarily displayed in order.

Yes, that was only a quick idea to also give that line.
Disabling multithreaded make and removing additional
optimization/debug CFLAGS on my system this is the problematic line:

gcc -Wno-maybe-uninitialized -Wshadow -O3 -ffast-math -m32 -Wall -I.
- -I. -march=i686 -mfpmath=sse -msse -std=gnu99
- -mpreferred-stack-boundary=5     -fPIC -fomit-frame-pointer
- -fno-tree-vectorize  -c -o encoder/me.o encoder/me.c

However, on Arch Linux I also have
https://github.com/thestinger/hardening-wrapper
running, effectively adding more CFLAGS "hidden".
There the flag "-fstack-check" was recently added in version 10.

Effectively the failing (and minimized) line is:

gcc -Wno-maybe-uninitialized -Wshadow -O3 -ffast-math -m32
- -fstack-check -Wall -I.
- -I. -march=i686 -mfpmath=sse -msse -std=gnu99
- -mpreferred-stack-boundary=5     -fPIC -fomit-frame-pointer
- -fno-tree-vectorize -c -o encoder/me.o encoder/me.c


>> In file included from ./common/common.h:1014:0, from 
>> encoder/me.c:28: encoder/me.c: In function 'x264_me_search_ref': 
>> ./common/x86/util.h:130:5: error: 'asm' operand has impossible 
>> constraints asm( ^ ./common/x86/util.h:193:5: error: 'asm' 
>> operand has impossible constraints asm( ^ <builtin>: recipe for 
>> target 'encoder/me.o' failed
>> 
>> The problem appears in building the Arch Linux 32 bit package 
>> "lib32-libx264": 
>> https://aur.archlinux.org/packages/lib32-libx264/
>> 
>> Built on a 64 bit machine with: ./configure --prefix=/usr 
>> --libdir=/usr/lib32 --host=i686-linux-gnu \ --enable-shared \ 
>> --enable-pic make
>> 
>> The 64 bit build runs fine on the same machine, but that isn't 
>> using files out of common/x86/, I guess.
> 
> The 64bit build does use the same code, it's just that it has more 
> registers available, so the same constraint is easier to solve.
>> 
>> I can't tell if that is a bug in the build tools or in x264 and I
>> don't have pratical experience in assembler.
> 
> It's a gcc bug. More precisely: Allocating registers to inline-asm 
> blocks is an NP Hard optimization problem. N is sufficiently small 
> that "NP Hard" doesn't imply "intractable", but it still requires
> a certain amount of intelligence on the compiler's part. That's
> true of ordinary C code too. The difference is that in C code, a 
> suboptimal compilation is just a bit slower than it could have 
> been, whereas in inline-asm it can result in the compiler 
> incorrectly throwing an "impossible constraints" error.
> 
> I have verified that the constraints are not in fact impossible. 
> But if you want to make it easier on the compiler, you could 
> disable pic.

Removing "--enable-pic" helps in general.

However, the full list of flags added by hardening-wrapper is:
- -fPIE -D_FORTIFY_SOURCE=2 -pie -fstack-check -fstack-protector-strong

- -fPIE is added when -fPIC is not.
To keep hardening-wrapper from adding -fPIE I added -fno-PIC in the
Arch Linux packaging CFLAGS now.

Thanks for the help.

- -- 
JonnyJD
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlXTZJYACgkQaDBIOMxoZgeT2ACgp2N0vMerS7MzwZOm1FfbCgqz
WQgAn3IZnlhp3QLUxRWA/UNFbwQz6sHR
=1Df1
-----END PGP SIGNATURE-----