[x264-devel] implementing Cluster farming?

[Steven Walters]

Since x264 can use up a 100% of a computer's CPU,
I was thinking if it would be at all feasible to implement beowulf clustering,
so it can use up nearby (or not so nearby) computers as well...

i was thinking of implementing it over sockets, which after doing some
sample coding with
it, it was fairly simple to implement with gnu's (linux) and winsock2
(win) libraries.
(dunno what mac does currently for socket implementation)

implementing the actual socket programming outside of x264 is all fine
and decently easy
but inserting it into x264 and being useful is becoming a much different story.

at the original thought, i was thinking of making the insertion more
transparent to x264 by having the master node
have more threads and just specify which ones were sent to other nodes
for processing.
but then remembering that x264 spawns and kills the processing threads
at an incredibly fast rate,
this would seemingly only proceed to slow down x264 overall, rather
than speed it up.

so then the thought transferred to split by GOPs.
but then how would x264 know where GOPs lie to split them to different
clusters when it hasn't even decided the slice type?

then another problem crops up:
certain structs will need to be sent across the sockets
(currently looking at
- (partial) x264_param_t: master -> slaves (for encoder initialization)
- x264_picture_t: master -> slaves
- x264_frame_t: slaves -> master )
but how to go about actually deep copying them into/from the socket buffer?
(would rather not have to add sizes for the pointers that need copying)

this is a lot of rambling but like many people,
i have a number of computers...
just trying to think of ways to utilize them better.

would helpful for feedback on overall feasibility or ideas of how to go about
some of the implementation hurdles.