[vlc-devel] Smooth Streaming manifest, track IDs

[Martin Storsjö]

On Fri, 17 Feb 2012, Frederic YHUEL wrote:

> Hi Martin, thanks for your help!
>
> On Fri, Feb 17, 2012 at 8:30 AM, Martin Storsjö <martin at martin.st> wrote:
>> Hi Frederic,
>>
>>
>> On Wed, 15 Feb 2012, Frederic YHUEL wrote:
>>
>>> Why does the (client) manifest *not* contain the track IDs?
>>>
>>> I guess that what Microsoft had in mind, when they wrote the specs, is
>>> that the client can associate an arbitrary ID to each track. Since the
>>> client does know from which track belongs each chunk, all should go
>>> well.
>>
>>
>> I guess the reason is that it normally isn't required - each fragment only
>> contains data from one single track anyway.
>
> Ok
>
>> (And within one smooth streaming
>> presentation, the tracks might either reside in separate ismv files and thus
>> have overlapping track ids, or they might all be within the same and have
>> unique ids.)
>>
>
> Overlapping track IDs, really? Oh that sucks, at least that means that
> my "pre-fetching" solution won't always work... :-/ Though I have the
> feeling that in the vast majority of cases, we have one and only one
> ismv file per quality level, and thus unique IDs.

Hmm, I'm not sure if we mean the same or different things here. What I 
meant was that since you have one ismv file per quality level, all the 
quality levels might have trackID=1 - there's nothing stopping that from 
happening. And if the audio stream is in a different file, that one might 
just as well have trackID=1 too.

>>
>>> But the way VLC is done makes things tricky. Retrieving the "real"
>>> track IDs is compulsory, if we use avformat demux, and is very helpful
>>> otherwise (I'm working on VLC's MP4 demux).
>>
>>
>> Why do you need knowing the real track id if using the avformat demuxer?
>
> I don't really know how the avformat demuxer works, but it doesn't
> know, a priori, what data it gets from the stream_filter module. In
> particular, for a given fragment, it cannot guess which track it
> belongs to, without looking at the ID in the tfhd box.
>
> When I was using it, I used to generate first a moov box from the
> manifest, in which I used to put the "real" track IDs in each tkhd
> box. It used to work well, if and even if track IDs of the moov box
> were matching those in fragment's tfhd boxes.

Ah, yes, if you use the demuxer that way (no, I don't know of any better 
way of tricking the lavf mov demuxer to parse plain fragments), then you 
will need to know the track ID beforehand I guess, and some kind of 
prefetch probably is the only way.

Another alternative would be to change the demuxer to be able to handle 
plain fragments without any moov header, but that then requires a way of 
initializiting the demuxer.

Yet another alternative would be to make up your own track IDs, and 
rewrite the fragments that you pass to the demuxer to match the track IDs 
you've chosen. Not pretty of course, but might not be insanely complex 
either.

>> If
>> you use one separate demuxer for each fragment, there shouldn't be any
>> ambiguity, but does that have other drawbacks?
>>
>
> s/fragment/track ?

Yes, sorry.

> Maybe I could have done that, but I don't know how.
>
>>
>>> Do you think my guess is correct? Or maybe there is a simple way to
>>> retrieve the track IDs after all? The only way I have in mind, is to
>>> pre-fetch the first (lowest quality) chunk of each track, and to parse
>>> the tfhd atom to finally get the track ID.
>>
>>
>> Also, a small note about prefetching (which might be informative but
>> probably not helpful): I guess you know that you can request
>> QualityLevels(...)/FragmentInfo(...) instead of Fragments. This gives you
>> only the moof atom, not the mdat part. This is used by the silverlight
>> player to fetch info about upcoming fragments (that are advertised within
>> the tfrf uuid tags) in live streams, without downloading the full fragments.
>>
>
> Cool! No I didn't know that... A few months ago, I didn't find
> [MS-SSTR], all what I found was the old [MS-SMTH] paper... the SS
> documentation is quite scatterd, or maybe I'm just stupid.

It's really scattered, and the documentation doesn't really explain stuff, 
it just states things without any explanation of how it all fits together.

>> This is possible, since even if the client manifest explicitly contains
>> declarations like
>>
>> Url="QualityLevels({bitrate})/Fragments(video={start time})",
>>
>> the spec actually mandates this to be almost literally this string, there's
>> not much variability. In particular, the "Fragments" string must be exactly
>> that, since apparently the silverlight players can swap it out against
>> FragmentInfo. The spec also mentions a few other similar things that can be
>> requested.
>>
>> I'm not sure if all smooth streaming content providers actually answer to
>> requests for FragmentInfo though, it might have been omitted in some
>> implementations of static content I think, since the official silverlight
>> player only requests it for live streams.
>>
>
> I got a MPD with that:
> Url="video_bitrate_{bitrate}_PTS_{start time}.ismv"
>
> But at the same time, this Smooth Streaming server is weird, since
> there is one unique track ID per track *and per quality level*

Hmm, that url doesn't seem valid according to how I read the specs at 
least. From [MS-SSTR], 2.2.2.4:

UrlPattern = QualityLevelsPattern "/" FragmentsPattern 
QualityLevelsPattern = QualityLevelsNoun "(" QualityLevelsPredicatePattern ")"
QualityLevelsNoun = "QualityLevels"
QualityLevelsPredicate = BitrateSubstitution ["," CustomAttributesSubstitution ]
Bitrate = "{bitrate}" / "{Bitrate}"
CustomAttributesSubstitution = "{CustomAttributes}"
FragmentsPattern = FragmentsNoun "(" FragmentsPatternPredicate ")";
FragmentsNoun = "Fragments"
FragmentsPatternPredicate = TrackName "=" StartTimeSubstitution;
TrackName = URISAFE_IDENTIFIER_NONNUMERIC
StartTimeSubstitution = "{start time}" / "{start_time}"

As far as I interpret this, there's not much else you can vary freely than 
TrackName and CustomAttributesSubstitution.

// Martin