[vlc-devel] [PATCH] configure: Fix illegal instructions on PPC when --enable-altivec used

[Lennart Sorensen]

From: Len Sorensen <lsorense at csclub.uwaterloo.ca>

When --enable-altivec is used on powerpc, -maltivec is passed to gcc
along with -O4.  When -O3 or higher is used along with -maltivec, gcc
enabled -ftree-vectorize, which generates vector instructions from C
code.  Since the code intended to run with altivec is wrapped in a CPU
feature check for altivec support, it doesn't work when gcc generates
vector instructions by itself.  Adding -fno-tree-vectorize along with
-maltivec fixes the problem and then only the desired locations end up
with vector instructions.  With this fix, VLC can run on G3 and e5500
machines while still supporting altivec when available.

Also fix the leak of -maltivec into the CPPFLAGS due to a missing
SAVE/RESTORE in configure.
---
 configure.ac                     | 7 +++++--
 modules/video_filter/Makefile.am | 1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/configure.ac b/configure.ac
index c8900d2..af9a1d9 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1504,16 +1504,18 @@ AS_IF([test "${enable_altivec}" = "yes"], [
   VLC_SAVE_FLAGS
   AC_CACHE_CHECK([if \$CC groks AltiVec C extensions],
   [ac_cv_c_altivec], [
-    CFLAGS="${CFLAGS} -maltivec"
+    CFLAGS="${CFLAGS} -maltivec -fno-tree-vectorize"
     AC_COMPILE_IFELSE([AC_LANG_PROGRAM([
 [#include <altivec.h>]], [
 [vec_ld(0, (unsigned char *)0);]])], [
-      ac_cv_c_altivec="-maltivec"
+      ac_cv_c_altivec="-maltivec -fno-tree-vectorize"
     ], [
       ac_cv_c_altivec="no"
     ])
   ])
   VLC_RESTORE_FLAGS
+
+  VLC_SAVE_FLAGS
   AS_IF([test "${ac_cv_c_altivec}" != "no"], [
     CPPFLAGS="${CPPFLAGS} ${ac_cv_c_altivec}"
     AC_DEFINE(CAN_COMPILE_C_ALTIVEC, 1, [Define to 1 if C AltiVec extensions are available.])
@@ -1523,6 +1525,7 @@ AS_IF([test "${enable_altivec}" = "yes"], [
     have_altivec="yes"
   ])
   AC_CHECK_HEADERS(altivec.h)
+  VLC_RESTORE_FLAGS
 
   VLC_SAVE_FLAGS
   LDFLAGS="${LDFLAGS} -Wl,-framework,vecLib"
diff --git a/modules/video_filter/Makefile.am b/modules/video_filter/Makefile.am
index 5d5fdaf..73d80a3 100644
--- a/modules/video_filter/Makefile.am
+++ b/modules/video_filter/Makefile.am
@@ -128,6 +128,7 @@ if HAVE_ARM64
 libdeinterlace_plugin_la_SOURCES += video_filter/deinterlace/merge_arm64.S
 libdeinterlace_plugin_la_CFLAGS += -DCAN_COMPILE_ARM64
 endif
+libdeinterlace_plugin_la_CFLAGS += $(ALTIVEC_CFLAGS)
 video_filter_LTLIBRARIES += libdeinterlace_plugin.la
 
 libdynamicoverlay_plugin_la_SOURCES = \
-- 
2.10.2