[vlc-devel] [PATCH] modules/demux/mkv/matroska_segment_parse.cpp: fix hvcC detection with mkvmerge

[Alexandre Janniaux]

mkvmerge had an issue with the first reserved bits and fixed it in v16.0.0
in the commit 4bb8ad6f5565e87ad6d6a8e7e9d453e64985344e. Some files done
with anterior versions were not played by VLC with mediacodec.
See the changelog of mkvmerge for version 16.0.0, especially the
following:
* mkvmerge: HEVC/h.265: the generation of the HEVCC structure stored in
  `CodecPrivate` was wrong in two places: 1. the position of the number of
  sub-layers was swapped with reserved bits and 2. the VPS/SPS/PPS/SEI lists
  did not start with a reserved 1 bit.
See also https://code.videolan.org/videolan/vlc-android/issues/466 for issue
and sample.
---
 modules/demux/mkv/matroska_segment_parse.cpp | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/modules/demux/mkv/matroska_segment_parse.cpp b/modules/demux/mkv/matroska_segment_parse.cpp
index 0b82866bb7..af55f4d7b8 100644
--- a/modules/demux/mkv/matroska_segment_parse.cpp
+++ b/modules/demux/mkv/matroska_segment_parse.cpp
@@ -1587,6 +1587,25 @@ bool matroska_segment_c::TrackInit( mkv_track_t * p_tk )
         }
         S_CASE("V_MPEGH/ISO/HEVC") {
             vars.p_tk->fmt.i_codec = VLC_CODEC_HEVC;
+
+            uint8_t* p_extra = (uint8_t*) vars.p_tk->p_extra_data;
+
+            /* HACK: if we found invalid format, made by mkvmerge < 16.0.0,
+             *       we try to fix it. They fixed it in 16.0.0. */
+            const char* app = vars.obj->psz_writing_application;
+            if( app != NULL && strncmp(app, "mkvmerge", strlen("mkvmerge")) == 0
+                    && p_extra && p_extra[0] == 0 )
+            {
+                int major_version;
+                if( sscanf(app, "mkvmerge v%d.", &major_version) && major_version < 16 )
+                {
+                    msg_Dbg(vars.p_demuxer,
+                            "Invalid HEVC reserved bits in mkv file"
+                            "made by mkvmerge < v16.0.0 detected, fixing it");
+                    p_extra[0] = 0x01;
+                }
+            }
+
             fill_extra_data( vars.p_tk, 0 );
         }
         S_CASE("V_QUICKTIME") {
-- 
2.16.1