[x264-devel] Fix incorrect zero-extension assumptions in x86_64 asm

[Henrik Gramner]

x264 | branch: master | Henrik Gramner <hengar-6 at student.ltu.se> | Wed Feb  1 23:52:48 2012 +0100| [02c3d5ec58d6bcbc5e22715ae80d53d8556f3c8f] | committer: Jason Garrett-Glaser

Fix incorrect zero-extension assumptions in x86_64 asm
Some x264 asm assumed that the high 32 bits of registers containing "int" values would be zero.
This is almost always the case, and it seems to work with gcc, but it is *not* guaranteed by the ABI.
As a result, it breaks with some other compilers, like Clang, that take advantage of this in optimizations.
Accordingly, fix all x86 code by using intptr_t instead of int or using movsxd where neccessary.
Also add checkasm hack to detect when assembly functions incorrectly assumes that 32-bit integers are zero-extended to 64-bit.

> http://git.videolan.org/gitweb.cgi/x264.git/?a=commit;h=02c3d5ec58d6bcbc5e22715ae80d53d8556f3c8f
---

 common/arm/mc-a.S        |   26 +++---
 common/arm/mc-c.c        |   82 ++++++++++----------
 common/arm/pixel.h       |   32 ++++----
 common/deblock.c         |  144 +++++++++++++++++-----------------
 common/frame.c           |    4 +-
 common/frame.h           |    6 +-
 common/macroblock.c      |    2 +-
 common/mc.c              |   94 ++++++++++++-----------
 common/mc.h              |   69 +++++++---------
 common/pixel.c           |   61 ++++++++------
 common/pixel.h           |   33 ++++----
 common/ppc/deblock.c     |    4 +-
 common/ppc/mc.c          |   92 ++++++++++-----------
 common/ppc/pixel.c       |   97 ++++++++++++-----------
 common/sparc/pixel.h     |    8 +-
 common/x86/deblock-a.asm |   82 ++++++++++----------
 common/x86/mc-a.asm      |   44 +++++------
 common/x86/mc-a2.asm     |  108 ++++++++++---------------
 common/x86/mc-c.c        |  197 +++++++++++++++++++++++-----------------------
 common/x86/pixel-32.asm  |    6 +-
 common/x86/pixel-a.asm   |   38 +++++----
 common/x86/pixel.h       |   76 +++++++++---------
 common/x86/quant-a.asm   |    2 +
 common/x86/sad-a.asm     |   23 ++----
 common/x86/sad16-a.asm   |    9 +--
 common/x86/x86inc.asm    |    5 -
 encoder/analyse.c        |   10 +-
 encoder/me.c             |   10 +-
 encoder/slicetype.c      |    2 +-
 tools/checkasm-a.asm     |  120 +++++++++++++++++++---------
 tools/checkasm.c         |  170 ++++++++++++++++++++++------------------
 31 files changed, 843 insertions(+), 813 deletions(-)

Diff:   http://git.videolan.org/gitweb.cgi/x264.git/?a=commitdiff;h=02c3d5ec58d6bcbc5e22715ae80d53d8556f3c8f