[x265] [ANN] automated test harness and PGO

[Steve Borho]

All,

I've been reworking our regression test infrastructure so it could be
openly published and community maintained. The system is implemented in
pure Python code (2.7 recommended, but I expect it should work with
Python 2.6 or 2.5).

The repo can be found at https://bitbucket.org/sborho/test-harness. It
has a readme.md that describes how to get it configured. It is primarily
designed to catch unintended output changes and/or non-determinism. It
requires the x265 source to be in a Mercurial clone, since it relies on
knowledge of the revision history to make decisions about which changes
were deliberate.

The test cases are tracked in the x265 repo itself under source/test, so
tests can be modifed together with encoder feature changes, much as we
do with the documentation.

Over the weekend I have added a script that uses the test harness
infrastructure to drive profile-guided optimizations for GCC (including
for MinGW on Windows).

If you are regularly submitting patches, you will probably find it
helpful to setup the test harness to run smoke tests, which makes sure
you haven't introduced any MSVC or GCC warnings, and ensures the test
bench still runs clean, and you haven't accidently changed encoder
outputs. I smoke test with these builds:

*  VC12-win64
*  VC12-win32
*  VC12-win64-16bpp
*  VC12-win64-debug (with Visual Leak Detector installed)
*  VC11
*  VC9
*  mingw
*  mingw-16bpp

This takes about half an hour on my oldish Windows desktop. I'd like to
cover mingw -m32 but I get link errors from this configuration.

On Linux I smoke-test gcc and gcc-16bpp and on Mac I smoke-test gcc,
gcc-16bpp, and gcc-m32.

The longer regression tests and rate-control tests get run at least once
a day on various machines (a mix of desktops and servers)

Feedback (and/or patches) welcome

-- 
Steve Borho