[vlc-devel] Variations in test coverage while running test suite

Sun Jul 30 18:25:09 UTC 2023

Hey Alexandre,

thank you so much for your answer!

Yes, I think that's it. Taking a closer look again, the SIGABRT signal 
which causes the tests to fail is also preceeded by a SIGALRM signal, 
which is caused by a call to alarm(), which sets the test timeout. So 
both failing tests run into the test timeout. Mystery solved, thank you! :)

 > Do you have tools to check such non-deterministic behaviour?

We wrote some tools to detect them for our study - as it's not published 
/ accepted yet I'm unfortunatly not able to publish them yet without 
jeopardizing the acceptance of the study, either. Sorry. :(

I'll make sure to write you another mail as soon as I'm able to publish 
/ open source the tools, though, and...

 > Don't hesitate to improve the testsuite and make reports like those. ;)

... will try to do that! :D

Kind regards and have a great week,
John

On 24/07/2023 11:15, Alexandre Janniaux wrote:
> Hi,
>
> Thanks for investigating the testsuite! Answers inline:
>
> On Sun, Jul 23, 2023 at 06:34:23PM +0200, John Smith via vlc-devel wrote:
>> Hey everyone,
>>
>> short addendum, very sorry, a little scatterbrained today:
>>
>> We also, as far as we know not connected to the test coverage, detected two
>> flaky tests while running the test suite, test_src_player and
>> test_src_misc_keystore, both failed *very very rarely* for reasons we were
>> unable to debug.
>>
>> test_src_misc_keystore failed with no apparent error in its log file (i.e.
>> test_src_misc_keystore.log was empty apart from:
>>
>>> FAIL test_src_misc_keystore (exit status: 134)
> This might be a timeout because the keystore could not be
> opened for some reason, maybe linked to the creation of the
> tmpfile.
>
>> test_src_player logged some errors, but nothing not also logged for
>> successful test runs - test log is attached. It also failed with error 134
>> (sigabort?)
> test_src_player is the longest test in the testsuite, with a
> worrying runtime of 9s on my machine:
>
> https://code.videolan.org/videolan/vlc/-/issues/27105
>
> Most of the test is waiting but we would probably need to
> improve the speed of the test to avoid those random errors.
> We use VLC_TEST_TIMEOUT=60 in the CI to workaround that.
>
> I haven't checked whether a new issue exists on this test
> though.
>
>> Are flaky tests like this in vlc something you are aware of in general? If
>> so, do you have an idea what, in the broadest of strokes, the reason could
>> be? Maybe race conditions like in issue #27582?
>> (https://code.videolan.org/videolan/vlc/-/issues/27582)
> Regarding the input_decoder test, this is a location where
> testing is quite complex and it's mostly doing non-regression
> testing as it is currently. Since it was written when fixing
> the input_decoder behaviour, it's possible that some
> behaviour is not deterministic (because there is no way to
> sync some locations) but still check the wanted path in the
> end.
>
> Some parts still require some changes, though the branch has
> been partially merged. I'm still working on this.
>
> https://code.videolan.org/videolan/vlc/-/merge_requests/2917
>
>> As we used a version from more than 7 months ago, its quite possible that
>> the issue behind this is already fixed - so please do not use too much
>> mental capacity to investigate this! :)
> Don't hesitate to improve the testsuite and make reports
> like those. ;)
>
> Do you have tools to check such non-deterministic behaviour?
>
> Regards,
> --
> Alexandre Janniaux
> Videolabs
>
>> Sorry for the additional mail, again thank you so much for your time and
>> kind regards,
>> John
>>
>> On 23/07/2023 14:45, John Smith via vlc-devel wrote:
>>> Hey everyone,
>>>
>>> during a study in which vlc was part of the sample (commit 49a0bccff70aabc7b2a06ca20388c8c7f01df31b, from the beginning of the year - that should be some version of 4.0.0-dev), we detected many variations in test coverage while running the test suite, i.e. the lines of code executed during the tests were not static. We build vlc using automake on Debian stable, executed the test suite using make check and measured the coverage using gcov.
>>>
>>> Main sources of these fluctuations were 'src/input/decoder.c', 'src/playlist/test.c', 'src/posix/timer.c', 'modules/stream_out/transcode/transcode.c'.
>>> (I attached one such diff between two such test runs at the end of this mail)
>>>
>>> We're assuming that those non-determinisms are caused by thread synchronization, e.g. one fluctuation in 'src/input/decoder.c' at line 831 is the else-path of the condition 'p_owner->p_vout != p_vout' in function 'ModuleThread_NewSpuBuffer', and timings during decoding / transcoding, and thus are normal, to be expected and harmless.
>>>
>>> However, vlc is very complex and we don't feel confident to make such claims based on a quick glance at the source code. It would be very helpful if a vlc developer could give a more authoritative answer.
>>>
>>> Thus, my question is: Is this assumption most likely correct and / or can you confirm that vlc (especially the decoder) can be internally non-deterministic?
>>>
>>> Thank you for your time and have a nice week! :)
>>>
>>> Kind regards
>>> John
>>>
>>> ==============================================
>>>
>>> Delta of coverage in lines of code executed between two runs of the test suite:
>>>
>>> File | Total LoC | LoC executed | % executed | All lines executed
>>> < modules/stream_out/transcode/transcode.c 393 250 63% 240,243,251-254,268-270,273,275,277,333-334,336,343-346,358,360-363,392-393,408,432,438,440-441,443,449,458,501,503-507,509,511-512,514,516-523,526,528-535,538,540-541,544,546,564,566,570,572,594,604-607,609-610,617-625,634-638,648-650,653,655-656,658,662,669,675,680-683,696-704,717-720,728,741,745-746,748,764,771-773,779-781,783-784,799-800,805,816-818,846,848-849,851,853
>>> ---
>>>> modules/stream_out/transcode/transcode.c 393 254 64% 240,243,251-254,268-270,273,275,277,333-334,336,343-346,358,360-363,392-393,408,432,438,440-441,443,449,458,501,503-507,509,511-512,514,516-523,526,528-535,538,540-541,544,546,564,566,570,572,594,604-607,609-610,617-625,634-638,648-650,653,655-656,658,662,669,675,680-683,696-704,717-720,728,745-746,748,764,771-773,779-781,783-784,799-800,805,846,848-849,851,853
>>> 907c907
>>> < src/input/decoder.c 1268 865 68% 291-293,307-308,318-320,322-324,326-328,334-336,356-359,372,387,402-403,406-410,412,419-420,422-423,436,438-439,441,460-461,512,527-530,535-536,546,549-550,606-607,645-646,650-651,655-656,689-690,713-715,718-719,721-722,726-727,730,733-735,774-775,777-778,780-783,785-787,789,795,802-803,805-806,818-820,844,848-849,852,854,856-857,860,863,865-869,871,874,877,879,881,883-886,913,932,935-936,940,951-952,956,959,962-964,967,969-975,978-979,981,1005-1007,1016,1018,1020-1021,1040,1043,1047-1048,1085,1087,1091-1092,1103,1105-1106,1109-1110,1112,1114-1115,1117,1131,1142-1144,1160-1161,1168,1170-1171,1186,1191-1192,1198,1213,1229,1236,1238,1242,1284-1286,1292-1294,1306-1307,1314,1316,1325,1332,1345,1360,1375-1377,1387-1388,1402,1413,1415-1416,1433,1442-1444,1458,1460,1473,1486,1493,1523,1526,1528,1537,1556,1562-1564,1576,1591,1608,1617,1638-1639,1641-1642,1647,1652,1654,1656,1658-1659,1661-1663,1665-1667,1669-1670,1672-1673,1678,1680,1700-1701,1758-1760,1922
>>>    -1923,1940-1941,1965-1967,2028,2039,2089,2097,2105,2107-2112,2115-2117,2120,2132-2133,2135,2141,2144-2145,2155-2156,2158,2160,2170-2172,2197,2200,2209,2301,2303-2304,2362-2363,2395-2396,2414,2416,2418-2419,2448,2450-2451,2474-2475,2477-2479,2481,2483-2486,2491-2492,2494-2495,2498-2499,2501,2503-2506,2508,2510,2520,2532-2533,2622,2625-2626,2628-2631,2633-2634,2636-2637,2639-2640,2663-2665,2689-2690,2693,2695-2696,2698,2701,2703-2704,2706,2709,2713,2715,2717-2718,2720-2721,2723,2726-2727,2729,2731,2733-2734,2736-2738,2740-2741,2743-2747,2749-2750,2753,2755,2757,2759,2761-2762,2764,2766-2767,2770,2773,2776-2777,2781-2782,2784-2785,2788,2790-2791
>>> ---
>>>> src/input/decoder.c 1268 861 67% 291-293,307-308,318-320,322-324,326-328,334-336,356-359,372,387,402-403,406-410,412,419-420,422-423,436,438-439,441,460-461,512,527-530,535-536,546,549-550,606-607,645-646,650-651,655-656,689-690,713-715,718-719,721-722,726-727,730,733-735,774-775,777-778,780-783,785-787,789,795,802-803,805-806,818-820,831,844,848-849,852,854,856-857,860,863,865-869,871,874,877,879,881,883-886,913,932,935-936,940,951-952,956,959,962-964,967,969-975,978-979,981,1005-1007,1016,1018,1020-1021,1040,1043,1047-1048,1066-1068,1085,1087,1091-1092,1103,1105-1106,1109-1110,1112,1114-1115,1117,1131,1142-1144,1160-1161,1168,1170-1171,1186,1191-1192,1198,1213,1229,1236,1238,1242,1284-1286,1292-1294,1306-1307,1314,1316,1325,1332,1345,1360,1375-1377,1387-1388,1402,1413,1415-1416,1433,1442-1444,1458,1460,1473,1486,1493,1523,1526,1528,1537,1556,1562-1564,1576,1591,1608,1617,1638-1639,1641-1642,1647,1652,1654,1656,1658-1659,1661-1663,1665-1667,1669-1670,1672-1673,1678,1680,1700-1701,
>>>    1758-1760,1922-1923,1940-1941,1965-1967,2028,2039,2089,2097,2105,2107-2112,2115-2117,2120,2132-2133,2135,2141,2144-2145,2155-2156,2158,2160,2170-2172,2197,2200,2209,2301,2303-2304,2362-2363,2395-2396,2414,2416,2418-2419,2448,2450-2451,2474-2475,2477-2479,2481,2483-2486,2491-2492,2494-2495,2498-2499,2501,2503-2506,2508,2510,2520,2532-2533,2622,2625-2626,2628-2631,2633-2634,2636-2637,2639-2640,2663-2665,2689-2690,2693,2695-2696,2698,2701,2703-2704,2706,2709,2713,2715,2717-2718,2720-2721,2723,2726-2727,2729,2731,2733-2734,2736-2738,2740-2741,2743-2747,2749-2750,2753,2755,2757,2759,2761-2762,2764,2766-2767,2770,2773,2776-2777,2781-2782,2784-2785,2788,2790-2791
>>> 1033c1033
>>> < src/playlist/test.c 1437 1430 99% 2211,2220-2222,2232-2234
>>> ---
>>>> src/playlist/test.c 1437 1429 99% 2214-2215,2220-2222,2232-2234
>>> 1046c1046
>>> < src/posix/timer.c 65 62 95% 95,125-126
>>> ---
>>>> src/posix/timer.c 65 59 90% 65-67,95,125-126
>>> 1147c1147
>>> < TOTAL 172117 50475 29%
>>> ---
>>>> TOTAL 172117 50471 29%
>>> _______________________________________________
>>> vlc-devel mailing list
>>> To unsubscribe or modify your subscription options:
>>> https://mailman.videolan.org/listinfo/vlc-devel
>> [000055d4ab33b440] main audio output error: no suitable audio output module
>> [00007f6d98000b60] main audio output error: no suitable audio output module
>> [00007f6da00063c0] main decoder error: buffer deadlock prevented
>> [00007f6d9c000b60] main video output error: video output display creation failed
>> [00007f6da00076c0] main decoder error: buffer deadlock prevented
>> [00007f6d9c04b4b0] main decoder error: buffer deadlock prevented
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d9c03a1c0] main decoder error: buffer deadlock prevented
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d9c001ec0] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.020001
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d9c003380] main decoder error: buffer deadlock prevented
>> [00007f6d9c04b6c0] main decoder error: buffer deadlock prevented
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [000055d4ab31b3b0] main input error: EsOutTimeshift does not work with streams that have pace control
>> [000055d4ab31b3b0] main input warning: cannot set pause state at es_out level
>> [000055d4ab31b3b0] main input: cannot set rate faster
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d9c003c70] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.599980
>> [00007f6d9c003c70] [clock] main generic warning: resetting master clock: coefficient too unstable: 1.200040
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [000055d4ab31b3b0] main input warning: INPUT_CONTROL_SET_TIME @0 failed or not possible
>> [000055d4ab31b3b0] main input error: Your input can't be opened
>> [000055d4ab31b3b0] main input error: VLC is unable to open the MRL 'unknownuri://foo'. Check the log for details.
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d9c037e40] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.001001
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d9c04c610] [clock] main generic warning: resetting master clock: coefficient too unstable: 0.001001
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [000055d4ab31b3b0] main input warning: ES 0x8 is already unselected
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6da00052f0] main subpicture warning: original picture size is undefined
>> [00007f6d78003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> [00007f6d98003690] main subpicture warning: original picture size is undefined
>> FAIL test_src_player (exit status: 134)
>> _______________________________________________
>> vlc-devel mailing list
>> To unsubscribe or modify your subscription options:
>> https://mailman.videolan.org/listinfo/vlc-devel