[x265] pool threads limitted

[steve at borho.org]

On 05/18, Zhuo Li wrote:
> Hi Steve,
> 
> 
> My name is Scotty,  a research master student of Institute of
> technology Tallaght.
> 
> 
> Recently I am working on video compression, the world-wide known open
> source HEVC project X265 is taken as my studying material.

please send x265 questions to x265-devel at videolan.org (cc:'d)

> My  research area is focused on parallel encoding,  and normally run
> x265 on  the station machine  with 120 threads   (CPU 4890 V2).
> 
> But when I read your code as shown below:
> 
> threadpool.h   ---   enum { MAX_POOL_THREADS = sizeof(sleepbitmap_t) *
> 8 };              //  MAX_POOL_THREADS is limited to 64.

this is the maximum number of threads *per thread pool* (a hard limit
from the maximum bitmap size type). The encoder will gladly create more
than one pool if it thinks there are more than 64 cores on a single
socket.

> I have tried to change the parameter MAX_POOL_THREADS to 128, but only
> the value changes. Seems it has no effect on the encoding process,
> even if I set the reference number 8, frame threads 10.
> 
>  x265 --input-res 1920x1080 --input tennis_500frames_1080p.yuv
>  --bitrate 1200 --vbv-maxrate 1380 --vbv-bufsize 1000 --psnr --tune
>  psnr  --ctu 32 --fps 20 --pmode   --pme    --preset slow    --F 10
>  --ref 8   -o out.hevc
> 
> Could you please give me some suggestions that I can fully utilize?
> the 120 threads?

Install libnuma, when x265 links with libnuma it is aware of the
individual CPU sockets/packages and makes more efficient use of each.

But.. an HD encode will not use very many threads. You'll be lucky to
get 10% utilization on a machine that large. With 4k you can get 3-4
times the occupancy. Exposing parallelism in an encode (without reducing
compression efficiency) is difficult.

To fully utilize a machine that large, you'll need multiple encodes.

-- 
Steve Borho