[vlc-devel] OpenHMD branch?

Sat May 18 03:19:42 CEST 2019

Hi Alexandre,

I'm now up & running on a shiny new Intel NUC 8 "Hades Canyon" with a 
Core i7-8705G @ 3.10 GHz and a Radeon RX Vega M GL GPU, which totally 
spanks the previous Celeron/UHD 600 system - and then some! To 
absolutely no surprise, video playback is now butter smooth, even at the 
highest resolution I have to test with (3840x2178, h264 part 10 4:2:0 
mp4, ~75mbit!), and head movements are much less jerky. So a worthwhile 
upgrade, for sure, though I'm still surprised by how poorly the previous 
set-up performed - I would have much preferred embedding a tiny fanless 
into the exhibition, but the NUC 8 is a reasonable size.

So I started with a clean slate, still using Debian Sid (to ensure mesa 
18.x, libdrm-amdgpu1 2.4 and kernel 4.19 amongst other things). Building 
vlc was almost as painful as last time - I logged over 40(!) steps in my 
notes, most of them dealing with unexpected build errors, incorrect 
library versions, missing dependencies, some of which were more painful 
than others (assimp, irrXML, protobuf - I'm looking at you!). At least 
the i7 allowed me to race through most of them this time. Some of my 
notes may be of interest:

Packages needed (from blank netinstall of Debian Sid):
------------------------------------------------------
````
firmware-linux firmware-linux-nonfree build-essential gettext git bison 
pkg-config flex autotools-dev autoconf libtool cmake libxml2-dev 
libjson-glib-dev lua5.2 liblua5.2-dev ffmpeg libavcodec-dev 
libavformat-dev libswresample-dev libswscale-dev libgdk-pixbuf2.0-dev 
libgl1-mesa-dev libx264-dev libx265-dev libxcb-composite0 
libxcb-composite0-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-glx0-dev 
libxcb-keysyms1-dev libxcb-present-dev libxcb-randr0-dev 
libxcb-render0-dev libxcb-shape0-dev libxcb-shm0-dev libxcb-sync-dev 
libxcb-xfixes0-dev libxcb-xv0-dev libxcb1-dev libxcomposite-dev 
libxcursor-dev libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev 
libxft-dev libxi-dev libxinerama-dev libxkbcommon-dev libxrandr-dev 
libxrender-dev mesa-common-dev libxkbcommon-x11-0 libxcb-image0 
libxcb-render-util0 libxcb-xinerama0 libxcb-xkb1 libxcb-icccm4 
libxcb-xkb-dev mesa-utils libgl1-mesa-glx libva-dev vainfo 
libasound2-dev qt5-default qt5-qmake qtbase5-dev qtbase5-dev-tools 
qtbase5-private-dev libqt5svg5-dev libqt5x11extras5 libqt5x11extras5-dev 
libhidapi-libusb0 libhidapi-dev kdelibs-bin libusb-1.0.0-dev 
libspatialaudio-dev libpostproc-dev libsdl-image1.2-dev glslang-dev 
xfce4 lightdm
````
I'd be interested to hear if any of these look suspicious - or 
suspiciously absent - in this list. I do get full GPU (2D/3D) support 
indicated by vainfo and glxinfo:

````
vainfo: VA-API version: 1.4 (libva 2.4.0)
vainfo: Driver version: Mesa Gallium driver 18.3.6 for AMD VEGAM (DRM 
3.27.0, 4.19.0-5-amd64, LLVM 7.0.1)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
````
````
$ glxinfo
Extended renderer info (GLX_MESA_query_renderer):
Vendor: X.Org (0x1002)
Device: AMD VEGAM (DRM 3.27.0, 4.19.0-5-amd64, LLVM 7.0.1) (0x694e)
Version: 18.3.6
Accelerated: yes
Video memory: 4096MB
Unified memory: no
Preferred profile: core (0x1)
Max core profile version: 4.5
Max compat profile version: 4.5
Max GLES1 profile version: 1.1
Max GLES[23] profile version: 3.2
````
I'm a bit disappointed not to see h265 among the supported profiles, 
which I had with the Intel UHD 600 - that said, I have no experience of 
what actual benefits it would offer for our application, I only know 
it's supposedly somewhat more effective at compressing >=4k content.

Error when using shipped OpenHMD lib (v0.2.0):
----------------------------------------------
````
video_output/hmd/openhmd.c: In function 'HMDThread':
video_output/hmd/openhmd.c:284:36: error: 'OHMD_UNIVERSAL_DISTORTION_K' 
undeclared (first use in this function); did you mean 
'OHMD_DISTORTION_K'?
ohmd_device_getf(sys->hmd, OHMD_UNIVERSAL_DISTORTION_K,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
OHMD_DISTORTION_K
````
This was resolved by cloning OpenHMD at v0.3.0-rc1 and building & 
installing it. VLC still couldn't find it though:
````
vlc: error while loading shared libraries:
libopenhmd.so.0: cannot open shared object file: No such file or 
directory
````
This was resolved by editing /etc/ld.so.conf and adding `include 
/usr/local/lib`, and then doing `# ldconfig`

Build fails with missing -lIrrXML error:
----------------------------------------
I haven't saved the exact error message, but VLC build choked on this, 
despite the shipped assimp(-dev) lib being the same version (4.1) as 
that in vlc/contrib - apparently the Debian package doesn't include the 
obscure (and very old) irrXML library, nor is it included in the repos. 
I tried installing libirrlicht(-dev) but this too did not include 
irrXML. Many hours spent on this one. Eventually resolved by cloning 
assimp at 4.1.0, modifying the included contrib/irrXML/CMakeLists.txt, 
changing ARCHIVE DESTINATION ${ASSIMP_LIB_INSTALL_DIR} to ARCHIVE 
DESTINATION lib, and building.

Build fails with missing protoc:
--------------------------------
Despite ./configure being happy, build fails with:
````
../../contrib/src/protobuf/rules.mak:13: *** protoc not found in PATH 
/home/ola/Sources/vlc/extras/tools/build/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games 
- - . Stop.
````
Pah, easy-peasy, just install protobuf-compiler. Actually, no:
````
../../contrib/src/protobuf/rules.mak:19: *** /usr/bin/protoc version 
3.6.1 doesn't match the protobuf 3.1.0 we're building. Stop.
````
*sigh* That's the price you pay for being on Sid, I guess. Resolved by 
cloning & building protobuf @ 3.1.0.

Virtual theatre errors:
-----------------------
For some reason VLC tries to load the virtual theatre, even though the 
video is shown as a full 360 equirrectangular stereoscopic projection - 
I though the VT was only to be used for "normal" video viewing, and even 
then perhaps it is a bit of a distraction? This load falls over with a 
bunch of missing file errors, e.g.
````
[00007fa4da936210] filesystem access error: cannot open file 
/home/ola/Sources/vlc/share/VirtualTheater/vlc_theater_model_embed.fbm/sol_Sol_mtl_Metalness.png 
(No such file or directory)
````
I would much rather VLC didn't even try - we have no need at all for 
this functionality.

Other observations:
-------------------
It's fantastic to finally be able to see what these videos really look 
like; I was quite blown away by the experience of suddenly being below 
the sea and had to genuinely resist the urge to try to swim for the 
surface! Though it's also clear that the technology has its limits. 
Despite the massive panels the resolution doesn't look that great in 
practice; if you look for them, individual pixels are obvious, and I can 
clearly see where the panels end, at the far edges of the field of 
vision, with a sharp black edge. This is all to be expected of course.

I'm also seeing slightly double, though that may just be my terrible 
eyesight. Or is it possible that the 3D transformations need adjusting 
somehow?

Although playback is near perfect, and a far cry from the slideshow I 
was getting before, CPU usage remains suprisingly high, with VLC jumping 
between 100 and 300% in top while playing the high resolution test 
video. I'd be interested to hear if this seems normal?

And there *is* still a slight strobing effect when moving your head, 
like the left and right images aren't updated quite simultaneously? It's 
not so bad as to make it unusable, and again a far cry from my earlier 
tests, but still a little disappointing considering the massive upgrade 
in OpenGL performance.

What's worse is that VLC crashes frequently with errors like:
````
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)
[00007f7058fbb2a0] avcodec decoder error: more than 25 frames of late 
video -> dropping frame (computer too slow ?)

... repeats a couple dozen times, then crashes with ...

[h264 @ 0x7f705906e8c0] co located POCs unavailable
[h264 @ 0x7f705908afc0] co located POCs unavailable
[h264 @ 0x7f7058fed540] co located POCs unavailable
[h264 @ 0x7f7059035a80] mmco: unref short failure
````
And I'm also getting some weird errors on launch:
````
$ ./vlc
VLC media player 4.0.0-dev Otto Chriek (revision 7f8c788)
[000055ba518a3950] main libvlc: Running vlc with the default interface. 
Use 'cvlc' to use vlc without interface.
[000055ba5193e420] main playlist: playlist is empty
[000055ba519554f0] qt interface: trying to load an HMD device
[00007fa4d80038b0] gl gl: Enabling HMD mode
[00007fa4d80038b0] gl gl: Enabling HMD mode
[00007fa4d82697f0] chain filter error: Too high level of recursion (3)
[00007fa4d825dd10] main filter error: Failed to create video converter
[00007fa4d82697f0] chain filter error: Too high level of recursion (3)
[00007fa4d825dd10] main filter error: Failed to create video converter
[00007fa4d82697f0] chain filter error: Too high level of recursion (3)
[00007fa4d825dd10] main filter error: Failed to create video converter

... repeats lots of times (100+) ...

[00007fa4d82697f0] chain filter error: Too high level of recursion (3)
[00007fa4d825dd10] main filter error: Failed to create video converter
[00007fa4cc0012c0] main vout display error: Failed to create video 
converter
[00007fa4cc0012c0] main vout display error: Failed to adapt decoder 
format to display
[00007fa4e41a6c50] main video output error: video output creation failed
[00007fa4f40c1680] main decoder error: failed to create video output
[000055ba519554f0] qt interface: HMD device already loaded, reusing it
[00007fa4d80038b0] gl gl: Enabling HMD mode
````
So definitely not stable enough for deployment, but at least I'll and 
have something to show for the week I've spent buried under a mountain 
of code - I'm sure it will blow my co-workers' minds, as it did mine! :D

Cracking on,

Lomax

P.S. Apologies for the MarkDown - I have no idea how to best format 
messages for this mailinglist!

On 2019-05-17 15:35 Alexandre Janniaux wrote:
> Hi,
> 
> On 2019-05-17 13:55, Lomax wrote:
> 
>> Hi Alexandre,
>> 
>> Many thanks for this, very helpful! I did manage to build Adrien
>> Maglo's
>> HMD-instancing branch before I saw your message, but your branch seems
>> a
>> little more up to date; in particular it includes the Vulkan
>> video_output module. I am able to run an example 360 stereo video
>> (after
>> injecting the required meta data tags[1]) using the OpenGL output and
>> it
>> looks stunning on the Rift, but performance is absolutely terrible - 
>> at
>> full resolution VLC is unable to progress beyond the first frame and
>> head movements are extremely jerky. Using a greatly scaled down video
>> (1280x720) it does play, but at something like 0.2fps, and head
>> movements are still far too jerky.
> 
> 0.2fps on 360° videos ? Is it the theater or just 360° projection ?
> If it's the theater, it's a known issue, lights currently have a huge
> toll
> on performances and the pending refactoring will help making it much
> faster.
> However, O.2fps on 360° projection looks really weird to me, what is
> your
> material ?
> 
>> I was curious to try the new "Vulkan" renderer so checked out your
>> branch and managed to get this to build with Vulkan support (after yet
>> more dependency wrangling) but when selecting this as video output the
>> 3D transformations are no longer applied; I just get the two panoramic
>> views one above the other, as I would when playing a video without the
>> spatialmedia meta data. Switching back to OpenGL makes it work again,
>> but at slideshow speeds. Clearly, I have underestimated the GPU grunt
>> required to handle this and will need to get a much more powerful
>> machine! Any pointers as to how high I should set the bar? I'm a bit
>> rusty on the GPU charts...
> 
> The vulkan renderer doesn't implement the required transformation and
> rendering method for this yet, even on master branch. It's mostly a
> matter of refactoring stuff and integrating this in libplacebo.
> 
> I'm running 360° with integrated intel CPU or old NVidia card in
> laptop. However I couldn't run the theater without a high-end graphic
> card for now.
> 
> --
> Alexandre Janniaux,
> VideoLabs