[vlc-devel] VLC timing issues

[basos g]

I think there is a big timing issue with the vout.
Pictures to be displayed are set by the decoder with a fixed time.

**Q1: Who sets the time and how does he know that vout rendering is
not too heavy to keep up ?


In my short analysys i realize that the main threads for data flow
that have buffers are decoder (block fifo buffer) and vout (picture
heap buffer).

The key factor for proper video output here is the *rate* at which
pictures flow.
Let Rv be the rate at witch vout displays picts (that would ideally be fps)
And Rd the rate at which the decoder sends picts (ideally fps)

When Rv > Rd the vout will just make idle loops sometimes, the picture
heap will be almost empty and the fifo will be of low usage.
When Rv<Rd big problem. Pictures *will* be displayed late and both
buffers will be filled. But this (should) depend on the hardware
capabilities.

The BIG problem with the current implementation is the static timing
from the decoder.
Imagine that you have a video filter that does some processing. Also
Rv is > Rd : i.e. vout can keep up with our frame rate. But if we see
this part

else if(...    p_pic->date < current_date + p_vout->p->render_time &&
 b_drop_late )
            {
                /* Picture is late: it will be destroyed and the thread
                 * will directly choose the next picture */
                vout_UsePictureLocked( p_vout, p_pic );
                p_vout->p->i_picture_lost++;

                msg_Warn( p_vout, "late picture skipped (%"PRId64" > %d)",
                                  current_date - p_pic->date, -
p_vout->p->render_time );
            }

if render time is above a threshold not because of an asynchronous
event but because the rendering and filtering time ARE kindof big (
above that threshold) ALL pictures will be late -> picture WILL freeze
for good -> bye.

So here comes buffering and timing.
We have a nice buffer (the heap). We have an absolute clock though
that imposes an  'a priory' arbitrary strict timing on the vout. I
think that the codec module arbitrarily decides the ABSOLUTE display
date of a picture. That makes the buffer useless.
 If we introduce a clock shift on the vout (that will have an upper
bound, the buffer size) picture will be at the original frame rate and
bingo.
Late pictures might occur but they should be invalidated by the
already existing check

            if( p_last && p_pic != p_last && p_pic->date <= p_last->date )
            {
                /* Drop old picture */
                vout_UsePictureLocked( p_vout, p_pic );
            }
This is a real late picture. The others might don't be late. They just
need a time shift on their dates and the can keep up (e.g. vout does
not have a slow rate Rv. It just has a somewhat fixed frame overhead
that could be hidden with the picture buffer

I think of some solutions for this.

SOL1:
compute a clock_shift value with a low pass filter ( to keep it as
stable and close to reality as possible) and just make a clock
tranformation to each place on the vout loop that does time
calculations . The goal is to schedule pictures for display taking
into account the fixed clock_shift that is caused by render and filter
times. this clock shift might be the already present render time ( but
i have not checked it for stability). The vout sleep time on non idle
picture should be the remaining picture diff time .



SOL2:
Use another clock mechanism that has a differential picture timing. I
don't know till what depth should modifications be done for this to
work (e.g. modify clock methods? modify all the codec modules? modify
the decoder?). But it should have these specifications :

Pictures should be indexed.
      proposed field p_pic->i_seq. that will keep the incremental
sequence number
Picture should have ralative, differential timing
     proposed field p_pic->i_diffdate that will keep the difference
date of the previous picture

E.g. of render loop
  curtime = mtime()

  drop picts with i_seq < last_pic->i_seq
  select picture that fullfills the constraint
    pic->i_seq = minimum
    if  (sel_pic->i_seq > last_pic->i_seq+1) /* some pictures skepped
by the decoder */
      printf ( "Pictures skipped")


  perform rendering - filtering
  sleepfor ( mtime() - ( curtime + cur_pic->i_difftime )

**Q2 :Is it possible to keep sequence numbers on all input and codec
combinations ??



We could head for SOL1 for now and maybe SOL2 on later relezes.

basos