[x264-devel] [PATCH 0/2] Fix input-frame recycling and threadpool shutdown races

[汪秀军]

Hello,

This two-patch series fixes two races reproduced on x264 master
0480cb05.

Patch 1 serializes input-frame release and unused-list push/pop with a
shared mutex. The lookahead thread can release last_nonb concurrently
with the caller releasing fenc or obtaining another input frame.

Patch 2 moves the threadpool exit check under its existing run mutex.
This independent race was exposed by closing without flushing.

Validation on macOS arm64, Apple clang 21.0.0:
- Unmodified master reproduces both races under ThreadSanitizer.
- Final series: six 1,200-frame TSan runs pass (8/1 frame threads,
  sync-lookahead 0/32, 10-bit and MB-tree), without suppressions.
- A separate 128x288, four-slice-thread TSan run passes for 240 frames.
- Closing without flushing after each input count 0..40 passes under TSan.
- Native-assembly checkasm8 and checkasm10 pass.
- Six release-mode comparisons have identical non-SEI NAL data and
  decoded frame MD5s. Version SEI differs due to the Git revision.
- A --disable-thread build and a 200-frame encode pass.

CLI reproducer build:

./configure --disable-asm --disable-opencl \
  --extra-cflags="-O1 -g -fsanitize=thread" \
  --extra-ldflags="-fsanitize=thread"
make -j8

Run against deterministic 128x72 I420 input:

TSAN_OPTIONS=halt_on_error=1 ./x264 --no-progress \
  --input-res 128x72 --input-csp i420 --frames 1200 \
  --preset ultrafast --threads 8 --sync-lookahead 32 \
  --bframes 8 --b-adapt 2 --rc-lookahead 60 \
  -o out.264 synthetic.yuv

Before patch 1, TSan reports frame_push_unused in the lookahead thread
racing with frame_push_unused in encoder_frame_end in the caller.
Atomic decrements alone also leave the shared unused-list push/pop race.

These tests cover the configurations above; OpenCL and other platforms
were not tested.

Best regards,
Xiujun Wang