[vlc-commits] demux: ts: fix MPGA and ADTS probing

[Michael Kaufmann]

vlc/vlc-3.0 | branch: master | Michael Kaufmann <mail at michael-kaufmann.ch> | Mon Mar  1 21:59:35 2021 +0100| [48afd4296d86830d4efa99ffa2cda5504cdc9ce1] | committer: Francois Cartegnie

demux: ts: fix MPGA and ADTS probing

MPEG-1 Audio Layer II (MP2) was erroneously detected as ADTS.

MPGA: The previous check used wrong bitmasks (1 bit too far to the right).
The layer check was also wrong, the layer must not be 0.

ADTS: Previously only one bit of the layer was checked. Check both bits.
The check is now identical to the one in HasADTSHeader() in mpeg4audio.c.

References:
- http://mpgedit.org/mpgedit/mpeg_format/mpeghdr.htm
- https://wiki.multimedia.cx/index.php/ADTS

Signed-off-by: Francois Cartegnie <fcvlcdev at free.fr>
(cherry picked from commit 90a54ccb4c57a61523279e410cd957b1671a29df)

> http://git.videolan.org/gitweb.cgi/vlc/vlc-3.0.git/?a=commit;h=48afd4296d86830d4efa99ffa2cda5504cdc9ce1
---

 modules/demux/mpeg/ts_hotfixes.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/modules/demux/mpeg/ts_hotfixes.c b/modules/demux/mpeg/ts_hotfixes.c
index 94aac62b2e..69a47932b5 100644
--- a/modules/demux/mpeg/ts_hotfixes.c
+++ b/modules/demux/mpeg/ts_hotfixes.c
@@ -167,11 +167,11 @@ void ProbePES( demux_t *p_demux, ts_pid_t *pid, const uint8_t *p_pesstart, size_
     {
         pid->probed.i_cat = AUDIO_ES;
         if( p_data[0] == 0xFF && (p_data[1] & 0xE0) == 0xE0 &&
-           (p_data[1] & 0x0C) != 0x04 && (p_data[1] & 0x03) == 0x00 )
+           (p_data[1] & 0x18) != 0x08 && (p_data[1] & 0x06) != 0x00 )
         {
             pid->probed.i_fourcc = VLC_CODEC_MPGA;
         }
-        else if( p_data[0] == 0xFF && (p_data[1] & 0xF2) == 0xF0 )
+        else if( p_data[0] == 0xFF && (p_data[1] & 0xF6) == 0xF0 )
         {
             pid->probed.i_fourcc = VLC_CODEC_MP4A; /* ADTS */
             pid->probed.i_original_fourcc = VLC_FOURCC('A','D','T','S');