[vlc-devel] Variations in test coverage while running test suite

[John Smith]

Hey everyone,

during a study in which vlc was part of the sample (commit 49a0bccff70aabc7b2a06ca20388c8c7f01df31b, from the beginning of the year - that should be some version of 4.0.0-dev), we detected many variations in test coverage while running the test suite, i.e. the lines of code executed during the tests were not static. We build vlc using automake on Debian stable, executed the test suite using make check and measured the coverage using gcov.

Main sources of these fluctuations were 'src/input/decoder.c', 'src/playlist/test.c', 'src/posix/timer.c', 'modules/stream_out/transcode/transcode.c'.
(I attached one such diff between two such test runs at the end of this mail)

We're assuming that those non-determinisms are caused by thread synchronization, e.g. one fluctuation in 'src/input/decoder.c' at line 831 is the else-path of the condition 'p_owner->p_vout != p_vout' in function 'ModuleThread_NewSpuBuffer', and timings during decoding / transcoding, and thus are normal, to be expected and harmless.

However, vlc is very complex and we don't feel confident to make such claims based on a quick glance at the source code. It would be very helpful if a vlc developer could give a more authoritative answer.

Thus, my question is: Is this assumption most likely correct and / or can you confirm that vlc (especially the decoder) can be internally non-deterministic?

Thank you for your time and have a nice week! :)

Kind regards
John

==============================================

Delta of coverage in lines of code executed between two runs of the test suite:

File | Total LoC | LoC executed | % executed | All lines executed
< modules/stream_out/transcode/transcode.c 393 250 63% 240,243,251-254,268-270,273,275,277,333-334,336,343-346,358,360-363,392-393,408,432,438,440-441,443,449,458,501,503-507,509,511-512,514,516-523,526,528-535,538,540-541,544,546,564,566,570,572,594,604-607,609-610,617-625,634-638,648-650,653,655-656,658,662,669,675,680-683,696-704,717-720,728,741,745-746,748,764,771-773,779-781,783-784,799-800,805,816-818,846,848-849,851,853
---
> modules/stream_out/transcode/transcode.c 393 254 64% 240,243,251-254,268-270,273,275,277,333-334,336,343-346,358,360-363,392-393,408,432,438,440-441,443,449,458,501,503-507,509,511-512,514,516-523,526,528-535,538,540-541,544,546,564,566,570,572,594,604-607,609-610,617-625,634-638,648-650,653,655-656,658,662,669,675,680-683,696-704,717-720,728,745-746,748,764,771-773,779-781,783-784,799-800,805,846,848-849,851,853
907c907
< src/input/decoder.c 1268 865 68% 291-293,307-308,318-320,322-324,326-328,334-336,356-359,372,387,402-403,406-410,412,419-420,422-423,436,438-439,441,460-461,512,527-530,535-536,546,549-550,606-607,645-646,650-651,655-656,689-690,713-715,718-719,721-722,726-727,730,733-735,774-775,777-778,780-783,785-787,789,795,802-803,805-806,818-820,844,848-849,852,854,856-857,860,863,865-869,871,874,877,879,881,883-886,913,932,935-936,940,951-952,956,959,962-964,967,969-975,978-979,981,1005-1007,1016,1018,1020-1021,1040,1043,1047-1048,1085,1087,1091-1092,1103,1105-1106,1109-1110,1112,1114-1115,1117,1131,1142-1144,1160-1161,1168,1170-1171,1186,1191-1192,1198,1213,1229,1236,1238,1242,1284-1286,1292-1294,1306-1307,1314,1316,1325,1332,1345,1360,1375-1377,1387-1388,1402,1413,1415-1416,1433,1442-1444,1458,1460,1473,1486,1493,1523,1526,1528,1537,1556,1562-1564,1576,1591,1608,1617,1638-1639,1641-1642,1647,1652,1654,1656,1658-1659,1661-1663,1665-1667,1669-1670,1672-1673,1678,1680,1700-1701,1758-1760,1922
 -1923,1940-1941,1965-1967,2028,2039,2089,2097,2105,2107-2112,2115-2117,2120,2132-2133,2135,2141,2144-2145,2155-2156,2158,2160,2170-2172,2197,2200,2209,2301,2303-2304,2362-2363,2395-2396,2414,2416,2418-2419,2448,2450-2451,2474-2475,2477-2479,2481,2483-2486,2491-2492,2494-2495,2498-2499,2501,2503-2506,2508,2510,2520,2532-2533,2622,2625-2626,2628-2631,2633-2634,2636-2637,2639-2640,2663-2665,2689-2690,2693,2695-2696,2698,2701,2703-2704,2706,2709,2713,2715,2717-2718,2720-2721,2723,2726-2727,2729,2731,2733-2734,2736-2738,2740-2741,2743-2747,2749-2750,2753,2755,2757,2759,2761-2762,2764,2766-2767,2770,2773,2776-2777,2781-2782,2784-2785,2788,2790-2791
---
> src/input/decoder.c 1268 861 67% 291-293,307-308,318-320,322-324,326-328,334-336,356-359,372,387,402-403,406-410,412,419-420,422-423,436,438-439,441,460-461,512,527-530,535-536,546,549-550,606-607,645-646,650-651,655-656,689-690,713-715,718-719,721-722,726-727,730,733-735,774-775,777-778,780-783,785-787,789,795,802-803,805-806,818-820,831,844,848-849,852,854,856-857,860,863,865-869,871,874,877,879,881,883-886,913,932,935-936,940,951-952,956,959,962-964,967,969-975,978-979,981,1005-1007,1016,1018,1020-1021,1040,1043,1047-1048,1066-1068,1085,1087,1091-1092,1103,1105-1106,1109-1110,1112,1114-1115,1117,1131,1142-1144,1160-1161,1168,1170-1171,1186,1191-1192,1198,1213,1229,1236,1238,1242,1284-1286,1292-1294,1306-1307,1314,1316,1325,1332,1345,1360,1375-1377,1387-1388,1402,1413,1415-1416,1433,1442-1444,1458,1460,1473,1486,1493,1523,1526,1528,1537,1556,1562-1564,1576,1591,1608,1617,1638-1639,1641-1642,1647,1652,1654,1656,1658-1659,1661-1663,1665-1667,1669-1670,1672-1673,1678,1680,1700-1701,
 1758-1760,1922-1923,1940-1941,1965-1967,2028,2039,2089,2097,2105,2107-2112,2115-2117,2120,2132-2133,2135,2141,2144-2145,2155-2156,2158,2160,2170-2172,2197,2200,2209,2301,2303-2304,2362-2363,2395-2396,2414,2416,2418-2419,2448,2450-2451,2474-2475,2477-2479,2481,2483-2486,2491-2492,2494-2495,2498-2499,2501,2503-2506,2508,2510,2520,2532-2533,2622,2625-2626,2628-2631,2633-2634,2636-2637,2639-2640,2663-2665,2689-2690,2693,2695-2696,2698,2701,2703-2704,2706,2709,2713,2715,2717-2718,2720-2721,2723,2726-2727,2729,2731,2733-2734,2736-2738,2740-2741,2743-2747,2749-2750,2753,2755,2757,2759,2761-2762,2764,2766-2767,2770,2773,2776-2777,2781-2782,2784-2785,2788,2790-2791
1033c1033
< src/playlist/test.c 1437 1430 99% 2211,2220-2222,2232-2234
---
> src/playlist/test.c 1437 1429 99% 2214-2215,2220-2222,2232-2234
1046c1046
< src/posix/timer.c 65 62 95% 95,125-126
---
> src/posix/timer.c 65 59 90% 65-67,95,125-126
1147c1147
< TOTAL 172117 50475 29%
---
> TOTAL 172117 50471 29%