[x264-devel] x86inc: Avoid using eax/rax for storing the stack pointer

[Henrik Gramner]

x264 | branch: master | Henrik Gramner <henrik at gramner.com> | Sat Oct  8 17:20:18 2016 +0200| [0706ddb1df88d716cf73decba4d82b953011760c] | committer: Henrik Gramner

x86inc: Avoid using eax/rax for storing the stack pointer

When allocating stack space with an alignment requirement that is larger
than the current stack alignment we need to store a copy of the original
stack pointer in order to be able to restore it later.

If we chose to use another register for this purpose we should not pick
eax/rax since it can be overwritten as a return value.

> http://git.videolan.org/gitweb.cgi/x264.git/?a=commit;h=0706ddb1df88d716cf73decba4d82b953011760c
---

 common/x86/x86inc.asm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/common/x86/x86inc.asm b/common/x86/x86inc.asm
index ff150f1..9543d89 100644
--- a/common/x86/x86inc.asm
+++ b/common/x86/x86inc.asm
@@ -373,10 +373,18 @@ DECLARE_REG_TMP_SIZE 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
     %ifnum %1
         %if %1 != 0 && required_stack_alignment > STACK_ALIGNMENT
             %if %1 > 0
+                ; Reserve an additional register for storing the original stack pointer, but avoid using
+                ; eax/rax for this purpose since it can potentially get overwritten as a return value.
                 %assign regs_used (regs_used + 1)
+                %if ARCH_X86_64 && regs_used == 7
+                    %assign regs_used 8
+                %elif ARCH_X86_64 == 0 && regs_used == 1
+                    %assign regs_used 2
+                %endif
             %endif
             %if ARCH_X86_64 && regs_used < 5 + UNIX64 * 3
-                ; Ensure that we don't clobber any registers containing arguments
+                ; Ensure that we don't clobber any registers containing arguments. For UNIX64 we also preserve r6 (rax)
+                ; since it's used as a hidden argument in vararg functions to specify the number of vector registers used.
                 %assign regs_used 5 + UNIX64 * 3
             %endif
         %endif