[dvblast-devel] TS discontinuity on multiple dvblast instances
Davor Spasoski
davor.spasoski at one.mk
Wed Jun 24 14:30:46 CEST 2015
Hi Laszlo,
Thank you for your valuable information. I checked all the points that you propose and the situation is such that:
- we have e1000 NICs
- there are no rx errors on the source NIC
- irq_balancer was automatically installed by Yast and running
- I increased socket buffer sizes as suggested.
The results are the same with or without irq_balancer and with or without increase socket buffer sizes I still notice the TS discontonuity errors and
#cat /proc/interrupts still shows that most of the eth IRQs end up at CPU0
CPU0 CPU1 CPU2 CPU3
0: 68 0 0 0 IO-APIC-edge timer
1: 4 6 0 0 IO-APIC-edge i8042
3: 1 0 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 7 2 0 0 IO-APIC-edge floppy
7: 0 0 0 0 IO-APIC-edge parport0
8: 35 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 136 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge ata_piix
15: 2248 4244 949 1053 IO-APIC-edge ata_piix
17: 8587714 2597 1924 54809 IO-APIC-fasteoi ioc0, eth3
18: 3395767 0 0 1835 IO-APIC-fasteoi eth0
19: 3013 0 0 0 IO-APIC-fasteoi eth1
I stress that SLES11 is running as a VM on top of VMWare ESX, but I have experienced the same problems on a dual core PC with SLES11 on bare metal.
Most strange is that when three instances are running with completely different source MUXes, stopping the third instance makes the first two complain about discontinuity on SIDs that belonged to the third instance, not theirs. Like the instances are using the same input.
I've seen a similar problem with more than one instance of ffmpeg with different source addresses, they end-up mixing the inputs.
It looks as the OS is the culprit. I'll try installing Debian as suggested. I'll try it as a VM first, it's easier for us.
BR,
Davor
-----Original Message-----
From: dvblast-devel [mailto:dvblast-devel-bounces at videolan.org] On Behalf Of BENEDEK László
Sent: 23 June 2015 14:13
To: Mailing list for DVBlast developers
Subject: Re: [dvblast-devel] TS discontinuity on multiple dvblast instances
Hi,
Here we use ~20 instances, ~40Mbps each, without any problem, but we use
Debian-amd64 on Intel based servers. Some tips:
1) I suggest to use Intel gigabit ethernet cards (e1000/e1000e), cheap ones can not handle such traffic reliably.
2) You can try to increase socket buffer sizes:
# echo 4096000 > /proc/sys/net/core/rmem_max # echo 4096000 > /proc/sys/net/core/wmem_max # echo 4096000 > /proc/sys/net/core/rmem_default # echo 4096000 > /proc/sys/net/core/wmem_default
3) You can check if the drops are in the NIC itself:
# ethtool -S eth0 | grep rx
If so, try to increase hw ring buffer size:
# ethtool -G eth0 rx {xxx}
4) On AMD platform the irqbalance daemon may also help, try to install it.
Hope this helps!
BENEDEK Laszlo
On 2015-06-23 12:46, Davor Spasoski wrote:
> No one from the developers interested in this problem?
>
> I have direct access to DVB-T multiplexes and can do any test required.
> Anyone?
>
>
>
> BR
>
>
>
> *From:*Davor Spasoski
> *Sent:* 10 June 2015 14:03
> *To:* 'Mailing list for DVBlast developers'
> *Subject:* RE: [dvblast-devel] TS discontinuity on multiple dvblast
> instances
>
>
>
> Unfortunately, omitting –b didn’t help. After a few pages of
> continuity errors, CPU waiting for IO jumps and so does the load.
>
>
>
> Thank you for helping!
>
>
>
> *From:*dvblast-devel [mailto:dvblast-devel-bounces at videolan.org] *On
> Behalf Of *Dan Lita
> *Sent:* 10 June 2015 13:38
> *To:* dvblast-devel at videolan.org <mailto:dvblast-devel at videolan.org>
> *Subject:* Re: [dvblast-devel] TS discontinuity on multiple dvblast
> instances
>
>
>
> Try without -b switch.
>
>
>
> On 6/10/2015 1:56 PM, Davor Spasoski wrote:
>
> Hi,
>
>
>
> I’m having a problem with dvblast with more than one instance running.
>
>
>
> My environment is:
>
> - SLES11 running on SUN AMD x86 server with 4 cores, 1Gb
> NICs, plenty of RAM. alternatively I test with a dual core Intel PC
> with same results
>
> - I use IP (UDP) MPTS sources from DVB-T headend, directly
> from production switches. The same sources feed the transmitters
> country-wide. The bitrate of the source is 27 mbit/s per MUX. At the
> moment I’m using clear only (unencrypted) muxes.
>
> - I filter up to 4 SIDs from the input MUX using the
> following configuration files
>
> mux1.conf
>
> 224.10.3.1:5004 at 10.146.16.86 <mailto:224.10.3.1:5004 at 10.146.16.86>
> 1 611
>
> 224.10.3.2:5004 at 10.146.16.86 <mailto:224.10.3.2:5004 at 10.146.16.86>
> 1 612
>
>
>
> mux8.conf
>
> 224.10.3.19:5004 at 10.146.16.86 <mailto:224.10.3.19:5004 at 10.146.16.86>
> 1 1312
>
> 224.10.3.16:5004 at 10.146.16.86 <mailto:224.10.3.16:5004 at 10.146.16.86>
> 1 1314
>
> 224.10.3.17:5004 at 10.146.16.86 <mailto:224.10.3.17:5004 at 10.146.16.86>
> 1 1311
>
> 224.10.3.18:5004 at 10.146.16.86 <mailto:224.10.3.18:5004 at 10.146.16.86>
> 1 1313
>
>
>
> Problem description:
>
>
>
> I start with a single instance of dvblast like:
>
> dvblast -D @239.3.1.13:5500/udp/ifaddr=10.0.33.5 -b 8 -l -g
> ip2ip_dvb_gw-D8 -c /home/user/mux8.conf
>
>
>
> CPU reading is:
>
> Cpu(s): 0.4%us, 0.7%sy, 0.0%ni, 98.0%id, 0.1%wa, 0.0%hi,
> 0.8%si, 0.0%st
>
> load average:
>
> 0.12, 0.56, 1.29
>
>
>
> This is usually working pretty stable. But if I run a second or n^th
> instance od dvblast, for example:
>
> dvblast -D @239.3.1.6:5500/udp/ifaddr=10.0.33.5 -b 8 -l -g
> ip2ip_dvb_gw-D1 -c /home/use/mux1.conf
>
>
>
> after a short while I get a flood of discontinuity errors in the
> log, CPU utilization of dvblast remains same, but overal the wa
> counter (waiting for io) jumps to around 25% and consequently the
> load increases dramatically (enough to saturate the PC)
>
>
>
> Cpu(s): 0.0%us, 0.5%sy, 0.0%ni, 74.7%id, 24.7%wa, 0.0%hi,
> 0.1%si, 0.0%st
>
> load average: 1.75, 1.17, 1.21
>
>
>
> We tested the sources with TS analyser on the same connection and
> there wasn’t a single continuity error in 24 hourse. Otherwise it
> would have been visible on STBs country-wide.
>
>
>
> It is worth noting that sometimes the problem appears with a single
> instance of dvblast.
>
>
>
> I tried with or without –u switch but nothing helps.
>
>
>
> Any ideas?
>
>
>
>
>
>
>
>
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> discontinuity on pid 5931 expected_cc 15 got 11 (H.264/14496-10
> video (MPEG-4/AVC), sid 1313)
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D1[4549]: warning: TS
> discontinuity on pid 5052 expected_cc 5 got 7 (13818-3 audio
> (MPEG-2), sid 605)
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> discontinuity on pid 5031 expected_cc 6 got 14 (H.264/14496-10
> video (MPEG-4/AVC), sid 1303)
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> discontinuity on pid 5012 expected_cc 5 got 13 (13818-3 audio
> (MPEG-2), sid 1301)
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> discontinuity on pid 5042 expected_cc 9 got 7 (13818-3 audio
> (MPEG-2), sid 1304)
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D1[4549]: warning: TS
> discontinuity on pid 5521 expected_cc 10 got 5 (H.264/14496-10
> video (MPEG-4/AVC), sid 612)
>
> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D1[4549]: warning: TS
> discontinuity on pid 5211 expected_cc 7 got 14 (H.264/14496-10
> video (MPEG-4/AVC), sid 611)
>
>
>
>
> ----------------------------------------------------------------------
> --
>
>
> ONE Telecommunications Services Limited Liability Company Skopje
>
> This e-mail (including any attachments) is confidential and may be
> protected by legal privilege. If you are not the intended recipient,
> you should not copy it, re-transmit it, use it or disclose its
> contents, but should return it to the sender immediately and delete
> your copy from your system. Any unauthorized use or dissemination of
> this message in whole or in part is strictly prohibited. Please note
> that e-mails are susceptible to change. ONE Limited Liability
> Company Skopje shall not be liable for the improper or incomplete
> transmission of the information contained in this communication nor
> for any delay in its receipt or damage to your system.
>
>
> _______________________________________________
>
> dvblast-devel mailing list
>
> dvblast-devel at videolan.org <mailto:dvblast-devel at videolan.org>
>
> https://mailman.videolan.org/listinfo/dvblast-devel
>
>
>
>
> ----------------------------------------------------------------------
> --
>
> ONE Telecommunications Services Limited Liability Company Skopje
>
> This e-mail (including any attachments) is confidential and may be
> protected by legal privilege. If you are not the intended recipient,
> you should not copy it, re-transmit it, use it or disclose its
> contents, but should return it to the sender immediately and delete
> your copy from your system. Any unauthorized use or dissemination of
> this message in whole or in part is strictly prohibited. Please note
> that e-mails are susceptible to change. ONE Limited Liability Company
> Skopje shall not be liable for the improper or incomplete transmission
> of the information contained in this communication nor for any delay
> in its receipt or damage to your system.
>
>
> _______________________________________________
> dvblast-devel mailing list
> dvblast-devel at videolan.org
> https://mailman.videolan.org/listinfo/dvblast-devel
>
_______________________________________________
dvblast-devel mailing list
dvblast-devel at videolan.org
https://mailman.videolan.org/listinfo/dvblast-devel
________________________________
ONE Telecommunications Services Limited Liability Company Skopje
This e-mail (including any attachments) is confidential and may be protected by legal privilege. If you are not the intended recipient, you should not copy it, re-transmit it, use it or disclose its contents, but should return it to the sender immediately and delete your copy from your system. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. Please note that e-mails are susceptible to change. ONE Limited Liability Company Skopje shall not be liable for the improper or incomplete transmission of the information contained in this communication nor for any delay in its receipt or damage to your system.
More information about the dvblast-devel
mailing list