[x264-devel] implementing Cluster farming?

[Shawn Nock]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I actually worked on this for a bit as a weekend hack / POC. I
implemented a *very basic* [read: not looking for criticism, I know it
is terrible and borderline worthless] x264-mpi workflow last year.

If anyone cares I put it on my gitweb: http://git.nocko.net/?p=x264-mpi
The primary commitdiff of interest is here:
http://git.nocko.net/?p=x264-mpi;a=commitdiff;h=2200ac1a260a20085b2df588936911338f486f95
although there are useful patches (multipass support) later on (look for
commits by 'Shawn Nock').

There is no frame server (so it relies on shared storage). I am sure
that it breaks a lot of optimizations (specifically scene detection,
which is right out).

In a nutshell, It counts the frames and splits the frames into groups
equal to the number of requested MPI processes. Then each process
encodes a frame group separately, outputting to a file with a sequential
extension. Concatenating these files (I don't think I implemented
concatenation in the primary process... memory fades) produces coherent
output, but nothing resembling the baseline output of a normal x264 run.

Caveats aside, It compiles on x86_64 (mpich2) and ia64 (sgi propriatary
mpi for it's numalink technology). If you don't care that it butchers
encoding efficiency (and ultimately the output file), the raw fps
numbers are encouraging.

I'd love to see a proper MPI support and I could provide several testing
platforms if someone was seriously interested in doing this.

Peace,
Shawn

Steven Walters wrote:
> On Sun, Mar 15, 2009 at 5:10 AM, Guillaume POIRIER
> <gpoirier at mplayerhq.hu> wrote:
>> The cleanest solution to do what you want is to use MPI
>> (http://en.wikipedia.org/wiki/Message_Passing_Interface).
>>
>> Guillaume
>> --
>> Only a very small fraction of our DNA does anything; the rest is all
>> comments and ifdefs.
>>
>> Douglas Adams  - "I love deadlines. I like the whooshing sound they
>> make as they fly by."
> 
> OK, I found the MPICH2 implementation of the MPI API to be one that
> actually works for both windows and linux.
> seems to be plenty of others that are linux only though.
> 
> This is going to be a fairly slow process of transformation to have
> x264 become MPI capable
> progress is probably going to be sparse in between working, classes,
> and job hunting with graduation coming up.
> _______________________________________________
> x264-devel mailing list
> x264-devel at videolan.org
> http://mailman.videolan.org/listinfo/x264-devel

- --
Shawn Nock (OpenPGP: 0x4744488F)
nock at nocko.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAknAj3IACgkQEnjZZ0dESI8wzgCeN0O9tpf9a15x5nlTc3J8orxV
ih0An1ZE5/Zo8PQAifFJh4FAQNTRR4au
=T2/0
-----END PGP SIGNATURE-----