[vlc-devel] parsing Spherical Media RFC metadata from AVC, HEVC bitstreams

[Chad Fogg]

Hello VLC developers:

Observations:

There is an implication in the following sentence that user_data_unregistered  SEI could carry this metadata in the elementary bitstream itself:

	https://github.com/google/spatial-media/blob/master/docs/spherical-video-rfc.md

	"User data unregistered SEI message syntax from the ISO-14496-10:2005 (see D.1.6). This message is an SEI message of Payload Type 5.”

The latest VLC Player source code blob appears to only parse the spherical indicator, projection mode (equirectangular)  from unregistered user data in the file container (MP4, MKV), but not from the user_data_unregistered SEI in the elementary bitstream. 

Also, the Spherical Media RFC v1 initial view elements (yaw, pitch, roll) are packed into a separate “prhd" MP4 metadata box :
 
	https://github.com/videolan/vlc/blob/master/modules/demux/mp4/libmp4.c

Questions:

(1) is it the intent for VLC player to also extract the Spherical Media RFC metadata from an SEI ?  Would it be split into two separate metadata blocks with the current practice 128-bit unregistered user ID { 0xff, 0xcc, 0x82, 0x63, 0xf8, 0x55, 0x4a, 0x93, 0x88, 0x14, 0x58, 0x7a, 0x02, 0x52, 0x1f, 0xdd } to convey spherical indicator, projection type (equi-rectangular), stereo frame packing arrangement (may be redundant with the frame_packing_arrangement SEI) …

…and additional separate messages for initial view port offsets (pitch, yaw, roll..) and other metadata introduced in Spherical Media RFC version 2 ?

Also:

(2) is it the intent to apply initial yaw, pitch roll offsets to rotate the output of the decoded video frame from a decoder, or is that metadata just passed onto the player and HMD  ? 
 
From what I can understand of the source code, VLC seems to apply the parsed Spherical Media RFC metadata as an initial viewpoint in the player that the user can override with keyboard buttons and mouse movements. I haven’t found the VLC player rotation source code yet..

In the HEVC standards development group, we are trying to start an SEI amendment for carriage of essential,  universally practiced equi-rectangular indicators in AVC and HEVC elementary bitstreams.

Regards,

--Chad
(MovieLabs)