[vlc-devel] Variations in test coverage while running test suite

[John Smith]

Hey everyone,

short addendum, very sorry, a little scatterbrained today:

We also, as far as we know not connected to the test coverage, detected 
two flaky tests while running the test suite, test_src_player and 
test_src_misc_keystore, both failed *very very rarely* for reasons we 
were unable to debug.

test_src_misc_keystore failed with no apparent error in its log file 
(i.e. test_src_misc_keystore.log was empty apart from:

 > FAIL test_src_misc_keystore (exit status: 134)

test_src_player logged some errors, but nothing not also logged for 
successful test runs - test log is attached. It also failed with error 
134 (sigabort?)

Are flaky tests like this in vlc something you are aware of in general? 
If so, do you have an idea what, in the broadest of strokes, the reason 
could be? Maybe race conditions like in issue #27582? 
(https://code.videolan.org/videolan/vlc/-/issues/27582)

As we used a version from more than 7 months ago, its quite possible 
that the issue behind this is already fixed - so please do not use too 
much mental capacity to investigate this! :)

Sorry for the additional mail, again thank you so much for your time and 
kind regards,
John

On 23/07/2023 14:45, John Smith via vlc-devel wrote:
> Hey everyone,
>
> during a study in which vlc was part of the sample (commit 49a0bccff70aabc7b2a06ca20388c8c7f01df31b, from the beginning of the year - that should be some version of 4.0.0-dev), we detected many variations in test coverage while running the test suite, i.e. the lines of code executed during the tests were not static. We build vlc using automake on Debian stable, executed the test suite using make check and measured the coverage using gcov.
>
> Main sources of these fluctuations were 'src/input/decoder.c', 'src/playlist/test.c', 'src/posix/timer.c', 'modules/stream_out/transcode/transcode.c'.
> (I attached one such diff between two such test runs at the end of this mail)
>
> We're assuming that those non-determinisms are caused by thread synchronization, e.g. one fluctuation in 'src/input/decoder.c' at line 831 is the else-path of the condition 'p_owner->p_vout != p_vout' in function 'ModuleThread_NewSpuBuffer', and timings during decoding / transcoding, and thus are normal, to be expected and harmless.
>
> However, vlc is very complex and we don't feel confident to make such claims based on a quick glance at the source code. It would be very helpful if a vlc developer could give a more authoritative answer.
>
> Thus, my question is: Is this assumption most likely correct and / or can you confirm that vlc (especially the decoder) can be internally non-deterministic?
>
> Thank you for your time and have a nice week! :)
>
> Kind regards
> John
>
> ==============================================
>
> Delta of coverage in lines of code executed between two runs of the test suite:
>
> File | Total LoC | LoC executed | % executed | All lines executed
> < modules/stream_out/transcode/transcode.c 393 250 63% 240,243,251-254,268-270,273,275,277,333-334,336,343-346,358,360-363,392-393,408,432,438,440-441,443,449,458,501,503-507,509,511-512,514,516-523,526,528-535,538,540-541,544,546,564,566,570,572,594,604-607,609-610,617-625,634-638,648-650,653,655-656,658,662,669,675,680-683,696-704,717-720,728,741,745-746,748,764,771-773,779-781,783-784,799-800,805,816-818,846,848-849,851,853
> ---
>> modules/stream_out/transcode/transcode.c 393 254 64% 240,243,251-254,268-270,273,275,277,333-334,336,343-346,358,360-363,392-393,408,432,438,440-441,443,449,458,501,503-507,509,511-512,514,516-523,526,528-535,538,540-541,544,546,564,566,570,572,594,604-607,609-610,617-625,634-638,648-650,653,655-656,658,662,669,675,680-683,696-704,717-720,728,745-746,748,764,771-773,779-781,783-784,799-800,805,846,848-849,851,853
> 907c907
> < src/input/decoder.c 1268 865 68% 291-293,307-308,318-320,322-324,326-328,334-336,356-359,372,387,402-403,406-410,412,419-420,422-423,436,438-439,441,460-461,512,527-530,535-536,546,549-550,606-607,645-646,650-651,655-656,689-690,713-715,718-719,721-722,726-727,730,733-735,774-775,777-778,780-783,785-787,789,795,802-803,805-806,818-820,844,848-849,852,854,856-857,860,863,865-869,871,874,877,879,881,883-886,913,932,935-936,940,951-952,956,959,962-964,967,969-975,978-979,981,1005-1007,1016,1018,1020-1021,1040,1043,1047-1048,1085,1087,1091-1092,1103,1105-1106,1109-1110,1112,1114-1115,1117,1131,1142-1144,1160-1161,1168,1170-1171,1186,1191-1192,1198,1213,1229,1236,1238,1242,1284-1286,1292-1294,1306-1307,1314,1316,1325,1332,1345,1360,1375-1377,1387-1388,1402,1413,1415-1416,1433,1442-1444,1458,1460,1473,1486,1493,1523,1526,1528,1537,1556,1562-1564,1576,1591,1608,1617,1638-1639,1641-1642,1647,1652,1654,1656,1658-1659,1661-1663,1665-1667,1669-1670,1672-1673,1678,1680,1700-1701,1758-1760,1922
>   -1923,1940-1941,1965-1967,2028,2039,2089,2097,2105,2107-2112,2115-2117,2120,2132-2133,2135,2141,2144-2145,2155-2156,2158,2160,2170-2172,2197,2200,2209,2301,2303-2304,2362-2363,2395-2396,2414,2416,2418-2419,2448,2450-2451,2474-2475,2477-2479,2481,2483-2486,2491-2492,2494-2495,2498-2499,2501,2503-2506,2508,2510,2520,2532-2533,2622,2625-2626,2628-2631,2633-2634,2636-2637,2639-2640,2663-2665,2689-2690,2693,2695-2696,2698,2701,2703-2704,2706,2709,2713,2715,2717-2718,2720-2721,2723,2726-2727,2729,2731,2733-2734,2736-2738,2740-2741,2743-2747,2749-2750,2753,2755,2757,2759,2761-2762,2764,2766-2767,2770,2773,2776-2777,2781-2782,2784-2785,2788,2790-2791
> ---
>> src/input/decoder.c 1268 861 67% 291-293,307-308,318-320,322-324,326-328,334-336,356-359,372,387,402-403,406-410,412,419-420,422-423,436,438-439,441,460-461,512,527-530,535-536,546,549-550,606-607,645-646,650-651,655-656,689-690,713-715,718-719,721-722,726-727,730,733-735,774-775,777-778,780-783,785-787,789,795,802-803,805-806,818-820,831,844,848-849,852,854,856-857,860,863,865-869,871,874,877,879,881,883-886,913,932,935-936,940,951-952,956,959,962-964,967,969-975,978-979,981,1005-1007,1016,1018,1020-1021,1040,1043,1047-1048,1066-1068,1085,1087,1091-1092,1103,1105-1106,1109-1110,1112,1114-1115,1117,1131,1142-1144,1160-1161,1168,1170-1171,1186,1191-1192,1198,1213,1229,1236,1238,1242,1284-1286,1292-1294,1306-1307,1314,1316,1325,1332,1345,1360,1375-1377,1387-1388,1402,1413,1415-1416,1433,1442-1444,1458,1460,1473,1486,1493,1523,1526,1528,1537,1556,1562-1564,1576,1591,1608,1617,1638-1639,1641-1642,1647,1652,1654,1656,1658-1659,1661-1663,1665-1667,1669-1670,1672-1673,1678,1680,1700-1701,
>   1758-1760,1922-1923,1940-1941,1965-1967,2028,2039,2089,2097,2105,2107-2112,2115-2117,2120,2132-2133,2135,2141,2144-2145,2155-2156,2158,2160,2170-2172,2197,2200,2209,2301,2303-2304,2362-2363,2395-2396,2414,2416,2418-2419,2448,2450-2451,2474-2475,2477-2479,2481,2483-2486,2491-2492,2494-2495,2498-2499,2501,2503-2506,2508,2510,2520,2532-2533,2622,2625-2626,2628-2631,2633-2634,2636-2637,2639-2640,2663-2665,2689-2690,2693,2695-2696,2698,2701,2703-2704,2706,2709,2713,2715,2717-2718,2720-2721,2723,2726-2727,2729,2731,2733-2734,2736-2738,2740-2741,2743-2747,2749-2750,2753,2755,2757,2759,2761-2762,2764,2766-2767,2770,2773,2776-2777,2781-2782,2784-2785,2788,2790-2791
> 1033c1033
> < src/playlist/test.c 1437 1430 99% 2211,2220-2222,2232-2234
> ---
>> src/playlist/test.c 1437 1429 99% 2214-2215,2220-2222,2232-2234
> 1046c1046
> < src/posix/timer.c 65 62 95% 95,125-126
> ---
>> src/posix/timer.c 65 59 90% 65-67,95,125-126
> 1147c1147
> < TOTAL 172117 50475 29%
> ---
>> TOTAL 172117 50471 29%
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel
-------------- next part --------------
[000055d4ab33b440] main audio output error: no suitable audio output module
[00007f6d98000b60] main audio output error: no suitable audio output module
[00007f6da00063c0] main decoder error: buffer deadlock prevented
[00007f6d9c000b60] main video output error: video output display creation failed
[00007f6da00076c0] main decoder error: buffer deadlock prevented
[00007f6d9c04b4b0] main decoder error: buffer deadlock prevented
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d9c03a1c0] main decoder error: buffer deadlock prevented
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d9c001ec0] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.020001
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d9c003380] main decoder error: buffer deadlock prevented
[00007f6d9c04b6c0] main decoder error: buffer deadlock prevented
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[000055d4ab31b3b0] main input error: EsOutTimeshift does not work with streams that have pace control
[000055d4ab31b3b0] main input warning: cannot set pause state at es_out level
[000055d4ab31b3b0] main input: cannot set rate faster
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d9c003c70] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.599980
[00007f6d9c003c70] [clock] main generic warning: resetting master clock: coefficient too unstable: 1.200040
[00007f6d98003690] main subpicture warning: original picture size is undefined
[000055d4ab31b3b0] main input warning: INPUT_CONTROL_SET_TIME @0 failed or not possible
[000055d4ab31b3b0] main input error: Your input can't be opened
[000055d4ab31b3b0] main input error: VLC is unable to open the MRL 'unknownuri://foo'. Check the log for details.
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d9c037e40] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.001001
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d9c04c610] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.001001
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6da00052f0] main subpicture warning: original picture size is undefined
[00007f6d78003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
[00007f6d98003690] main subpicture warning: original picture size is undefined
FAIL test_src_player (exit status: 134)