[dvblast-devel] TS discontinuity on multiple dvblast instances
Nuno Mota
mundumelga at gmail.com
Fri Jun 26 21:05:11 CEST 2015
I've been having the same issue with an AMD based server (bare metal).
Top shows cpu with a lot of idle percentage, but load is very high due to a
lot of discontinuities and transport errors. Stopping those problematic
dvblast instances everything runs smoothly.
here is my /proc/interrupts. Streaming goes out through eth1
CPU0 CPU1 CPU2
0: 175051 1607629 41033537 IO-APIC-edge timer
1: 0 0 2 IO-APIC-edge i8042
7: 1 0 0 IO-APIC-edge
8: 0 0 1 IO-APIC-edge rtc0
9: 0 0 0 IO-APIC-fasteoi acpi
14: 0 0 0 IO-APIC-edge pata_atiixp
15: 0 0 0 IO-APIC-edge pata_atiixp
16: 0 0 1 IO-APIC-fasteoi nouveau
17: 0 20 2861965 IO-APIC-fasteoi ehci_hcd:usb1,
ehci_hcd:usb2, ehci_hcd:usb3, SAA716x Core
18: 6692094 2262 93 IO-APIC-fasteoi ohci_hcd:usb4,
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, SAA716x Core
19: 632960 1404818 4529 IO-APIC-fasteoi ahci, SAA716x Core
20: 2444017 39184 6519 IO-APIC-fasteoi saa7146 (0)
22: 0 827 19 IO-APIC-fasteoi firewire_ohci
40: 826056 61 4138 PCI-MSI-edge eth0
41: 0 0 0 PCI-MSI-edge xhci_hcd
42: 0 0 0 PCI-MSI-edge xhci_hcd
43: 324044398 308 42633 PCI-MSI-edge eth1
NMI: 0 0 0 Non-maskable interrupts
LOC: 25094127 21226295 6066079 Local timer interrupts
SPU: 0 0 0 Spurious interrupts
PMI: 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 IRQ work interrupts
RES: 23337415 20225007 10270328 Rescheduling interrupts
CAL: 46577 130484 180586 Function call interrupts
TLB: 699 1543 1249 TLB shootdowns
TRM: 0 0 0 Thermal event interrupts
THR: 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 Machine check exceptions
MCP: 180 180 180 Machine check polls
ERR: 1
MIS: 0
2015-06-25 14:10 GMT+01:00 Davor Spasoski <davor.spasoski at one.mk>:
> Hi,
> Thank you!
> Tried the proposed, but make is complaining " udp.c:206: error:
> ‘IP_MULTICAST_ALL’ undeclared (first use in this function)"
> When checking the man(7) page, it was evident that on SLES the
> IP_MULTICAST_ALL is not listed.
> This option was added in 2.6.31 and my SLES kernel is 3.0.76-0.11-default
> but strangely, it's missing.
>
> I moved the test environment to Ubuntu 14, compiled with the patch and
> dvblast works there with or without the patch.
> I have to observe more though to make sure it makes no difference.
>
> Thanks again
>
>
> -----Original Message-----
> From: dvblast-devel [mailto:dvblast-devel-bounces at videolan.org] On Behalf
> Of BENEDEK László
> Sent: 24 June 2015 16:16
> To: Mailing list for DVBlast developers
> Subject: Re: [dvblast-devel] TS discontinuity on multiple dvblast instances
>
> Hello,
>
> OK, then perhaps the problem may be not setting IP_MULTICAST_ALL
> (2.6.31+) to 0 on the receiving socket.
>
> In udp.c somewhere around IP_ADD_MEMBERSHIP try to write lines like this
> (untested code!):
>
> int mc_all = 0;
> if ((setsockopt( i_handle, IPPROTO_IP, IP_MULTICAST_ALL, (void*) &mc_all,
> sizeof(mc_all))) < 0) {
> msg_Warn( NULL, "multicast filtering setup failed (%s)",
> strerror(errno) );
> }
>
> See
>
> https://stackoverflow.com/questions/2741611/receving-multiple-multicast-feeds-on-the-same-port-c-linux/20919920#20919920
> for more info.
>
> L
>
> On 2015-06-24 14:30, Davor Spasoski wrote:
> > Hi Laszlo,
> >
> > Thank you for your valuable information. I checked all the points that
> you propose and the situation is such that:
> > - we have e1000 NICs
> > - there are no rx errors on the source NIC
> > - irq_balancer was automatically installed by Yast and running
> > - I increased socket buffer sizes as suggested.
> >
> > The results are the same with or without irq_balancer and with or
> > without increase socket buffer sizes I still notice the TS discontonuity
> errors and #cat /proc/interrupts still shows that most of the eth IRQs end
> up at CPU0
> > CPU0 CPU1 CPU2 CPU3
> > 0: 68 0 0 0 IO-APIC-edge
> timer
> > 1: 4 6 0 0 IO-APIC-edge
> i8042
> > 3: 1 0 0 0 IO-APIC-edge
> > 4: 0 0 0 1 IO-APIC-edge
> > 6: 7 2 0 0 IO-APIC-edge
> floppy
> > 7: 0 0 0 0 IO-APIC-edge
> parport0
> > 8: 35 0 0 0 IO-APIC-edge rtc0
> > 9: 0 0 0 0 IO-APIC-fasteoi acpi
> > 12: 1 136 0 0 IO-APIC-edge
> i8042
> > 14: 0 0 0 0 IO-APIC-edge
> ata_piix
> > 15: 2248 4244 949 1053 IO-APIC-edge
> ata_piix
> > 17: 8587714 2597 1924 54809 IO-APIC-fasteoi
> ioc0, eth3
> > 18: 3395767 0 0 1835 IO-APIC-fasteoi eth0
> > 19: 3013 0 0 0 IO-APIC-fasteoi eth1
> >
> > I stress that SLES11 is running as a VM on top of VMWare ESX, but I have
> experienced the same problems on a dual core PC with SLES11 on bare metal.
> > Most strange is that when three instances are running with completely
> different source MUXes, stopping the third instance makes the first two
> complain about discontinuity on SIDs that belonged to the third instance,
> not theirs. Like the instances are using the same input.
> >
> > I've seen a similar problem with more than one instance of ffmpeg with
> different source addresses, they end-up mixing the inputs.
> > It looks as the OS is the culprit. I'll try installing Debian as
> suggested. I'll try it as a VM first, it's easier for us.
> >
> > BR,
> > Davor
> >
> >
> > -----Original Message-----
> > From: dvblast-devel [mailto:dvblast-devel-bounces at videolan.org] On
> > Behalf Of BENEDEK László
> > Sent: 23 June 2015 14:13
> > To: Mailing list for DVBlast developers
> > Subject: Re: [dvblast-devel] TS discontinuity on multiple dvblast
> > instances
> >
> > Hi,
> >
> > Here we use ~20 instances, ~40Mbps each, without any problem, but we
> > use
> > Debian-amd64 on Intel based servers. Some tips:
> >
> > 1) I suggest to use Intel gigabit ethernet cards (e1000/e1000e), cheap
> ones can not handle such traffic reliably.
> >
> > 2) You can try to increase socket buffer sizes:
> > # echo 4096000 > /proc/sys/net/core/rmem_max # echo 4096000 >
> > /proc/sys/net/core/wmem_max # echo 4096000 >
> > /proc/sys/net/core/rmem_default # echo 4096000 >
> > /proc/sys/net/core/wmem_default
> >
> > 3) You can check if the drops are in the NIC itself:
> > # ethtool -S eth0 | grep rx
> >
> > If so, try to increase hw ring buffer size:
> > # ethtool -G eth0 rx {xxx}
> >
> > 4) On AMD platform the irqbalance daemon may also help, try to install
> it.
> >
> > Hope this helps!
> >
> > BENEDEK Laszlo
> >
> > On 2015-06-23 12:46, Davor Spasoski wrote:
> >> No one from the developers interested in this problem?
> >>
> >> I have direct access to DVB-T multiplexes and can do any test required.
> >> Anyone?
> >>
> >>
> >>
> >> BR
> >>
> >>
> >>
> >> *From:*Davor Spasoski
> >> *Sent:* 10 June 2015 14:03
> >> *To:* 'Mailing list for DVBlast developers'
> >> *Subject:* RE: [dvblast-devel] TS discontinuity on multiple dvblast
> >> instances
> >>
> >>
> >>
> >> Unfortunately, omitting –b didn’t help. After a few pages of
> >> continuity errors, CPU waiting for IO jumps and so does the load.
> >>
> >>
> >>
> >> Thank you for helping!
> >>
> >>
> >>
> >> *From:*dvblast-devel [mailto:dvblast-devel-bounces at videolan.org] *On
> >> Behalf Of *Dan Lita
> >> *Sent:* 10 June 2015 13:38
> >> *To:* dvblast-devel at videolan.org <mailto:dvblast-devel at videolan.org>
> >> *Subject:* Re: [dvblast-devel] TS discontinuity on multiple dvblast
> >> instances
> >>
> >>
> >>
> >> Try without -b switch.
> >>
> >>
> >>
> >> On 6/10/2015 1:56 PM, Davor Spasoski wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> I’m having a problem with dvblast with more than one instance
> running.
> >>
> >>
> >>
> >> My environment is:
> >>
> >> - SLES11 running on SUN AMD x86 server with 4 cores, 1Gb
> >> NICs, plenty of RAM. alternatively I test with a dual core Intel PC
> >> with same results
> >>
> >> - I use IP (UDP) MPTS sources from DVB-T headend, directly
> >> from production switches. The same sources feed the transmitters
> >> country-wide. The bitrate of the source is 27 mbit/s per MUX. At the
> >> moment I’m using clear only (unencrypted) muxes.
> >>
> >> - I filter up to 4 SIDs from the input MUX using the
> >> following configuration files
> >>
> >> mux1.conf
> >>
> >> 224.10.3.1:5004 at 10.146.16.86
> >> <mailto:224.10.3.1:5004 at 10.146.16.86>
> >> 1 611
> >>
> >> 224.10.3.2:5004 at 10.146.16.86
> >> <mailto:224.10.3.2:5004 at 10.146.16.86>
> >> 1 612
> >>
> >>
> >>
> >> mux8.conf
> >>
> >> 224.10.3.19:5004 at 10.146.16.86 <mailto:224.10.3.19:5004 at 10.146.16.86
> >
> >> 1 1312
> >>
> >> 224.10.3.16:5004 at 10.146.16.86 <mailto:224.10.3.16:5004 at 10.146.16.86
> >
> >> 1 1314
> >>
> >> 224.10.3.17:5004 at 10.146.16.86 <mailto:224.10.3.17:5004 at 10.146.16.86
> >
> >> 1 1311
> >>
> >> 224.10.3.18:5004 at 10.146.16.86 <mailto:224.10.3.18:5004 at 10.146.16.86
> >
> >> 1 1313
> >>
> >>
> >>
> >> Problem description:
> >>
> >>
> >>
> >> I start with a single instance of dvblast like:
> >>
> >> dvblast -D @239.3.1.13:5500/udp/ifaddr=10.0.33.5 -b 8 -l -g
> >> ip2ip_dvb_gw-D8 -c /home/user/mux8.conf
> >>
> >>
> >>
> >> CPU reading is:
> >>
> >> Cpu(s): 0.4%us, 0.7%sy, 0.0%ni, 98.0%id, 0.1%wa, 0.0%hi,
> >> 0.8%si, 0.0%st
> >>
> >> load average:
> >>
> >> 0.12, 0.56, 1.29
> >>
> >>
> >>
> >> This is usually working pretty stable. But if I run a second or n^th
> >> instance od dvblast, for example:
> >>
> >> dvblast -D @239.3.1.6:5500/udp/ifaddr=10.0.33.5 -b 8 -l -g
> >> ip2ip_dvb_gw-D1 -c /home/use/mux1.conf
> >>
> >>
> >>
> >> after a short while I get a flood of discontinuity errors in the
> >> log, CPU utilization of dvblast remains same, but overal the wa
> >> counter (waiting for io) jumps to around 25% and consequently the
> >> load increases dramatically (enough to saturate the PC)
> >>
> >>
> >>
> >> Cpu(s): 0.0%us, 0.5%sy, 0.0%ni, 74.7%id, 24.7%wa, 0.0%hi,
> >> 0.1%si, 0.0%st
> >>
> >> load average: 1.75, 1.17, 1.21
> >>
> >>
> >>
> >> We tested the sources with TS analyser on the same connection and
> >> there wasn’t a single continuity error in 24 hourse. Otherwise it
> >> would have been visible on STBs country-wide.
> >>
> >>
> >>
> >> It is worth noting that sometimes the problem appears with a single
> >> instance of dvblast.
> >>
> >>
> >>
> >> I tried with or without –u switch but nothing helps.
> >>
> >>
> >>
> >> Any ideas?
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> >> discontinuity on pid 5931 expected_cc 15 got 11 (H.264/14496-10
> >> video (MPEG-4/AVC), sid 1313)
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D1[4549]: warning: TS
> >> discontinuity on pid 5052 expected_cc 5 got 7 (13818-3 audio
> >> (MPEG-2), sid 605)
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> >> discontinuity on pid 5031 expected_cc 6 got 14 (H.264/14496-10
> >> video (MPEG-4/AVC), sid 1303)
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> >> discontinuity on pid 5012 expected_cc 5 got 13 (13818-3 audio
> >> (MPEG-2), sid 1301)
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D8[4548]: warning: TS
> >> discontinuity on pid 5042 expected_cc 9 got 7 (13818-3 audio
> >> (MPEG-2), sid 1304)
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D1[4549]: warning: TS
> >> discontinuity on pid 5521 expected_cc 10 got 5 (H.264/14496-10
> >> video (MPEG-4/AVC), sid 612)
> >>
> >> Jun 10 12:10:17 srd2 ip2ip_dvb_gw-D1[4549]: warning: TS
> >> discontinuity on pid 5211 expected_cc 7 got 14 (H.264/14496-10
> >> video (MPEG-4/AVC), sid 611)
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> --
> >>
> >>
> >> ONE Telecommunications Services Limited Liability Company Skopje
> >>
> >> This e-mail (including any attachments) is confidential and may be
> >> protected by legal privilege. If you are not the intended recipient,
> >> you should not copy it, re-transmit it, use it or disclose its
> >> contents, but should return it to the sender immediately and delete
> >> your copy from your system. Any unauthorized use or dissemination of
> >> this message in whole or in part is strictly prohibited. Please note
> >> that e-mails are susceptible to change. ONE Limited Liability
> >> Company Skopje shall not be liable for the improper or incomplete
> >> transmission of the information contained in this communication nor
> >> for any delay in its receipt or damage to your system.
> >>
> >>
> >> _______________________________________________
> >>
> >> dvblast-devel mailing list
> >>
> >> dvblast-devel at videolan.org <mailto:dvblast-devel at videolan.org>
> >>
> >> https://mailman.videolan.org/listinfo/dvblast-devel
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> --
> >>
> >> ONE Telecommunications Services Limited Liability Company Skopje
> >>
> >> This e-mail (including any attachments) is confidential and may be
> >> protected by legal privilege. If you are not the intended recipient,
> >> you should not copy it, re-transmit it, use it or disclose its
> >> contents, but should return it to the sender immediately and delete
> >> your copy from your system. Any unauthorized use or dissemination of
> >> this message in whole or in part is strictly prohibited. Please note
> >> that e-mails are susceptible to change. ONE Limited Liability Company
> >> Skopje shall not be liable for the improper or incomplete
> >> transmission of the information contained in this communication nor
> >> for any delay in its receipt or damage to your system.
> >>
> >>
> >> _______________________________________________
> >> dvblast-devel mailing list
> >> dvblast-devel at videolan.org
> >> https://mailman.videolan.org/listinfo/dvblast-devel
> >>
> >
> > _______________________________________________
> > dvblast-devel mailing list
> > dvblast-devel at videolan.org
> > https://mailman.videolan.org/listinfo/dvblast-devel
> >
> > ________________________________
> >
> > ONE Telecommunications Services Limited Liability Company Skopje
> >
> > This e-mail (including any attachments) is confidential and may be
> protected by legal privilege. If you are not the intended recipient, you
> should not copy it, re-transmit it, use it or disclose its contents, but
> should return it to the sender immediately and delete your copy from your
> system. Any unauthorized use or dissemination of this message in whole or
> in part is strictly prohibited. Please note that e-mails are susceptible to
> change. ONE Limited Liability Company Skopje shall not be liable for the
> improper or incomplete transmission of the information contained in this
> communication nor for any delay in its receipt or damage to your system.
> > _______________________________________________
> > dvblast-devel mailing list
> > dvblast-devel at videolan.org
> > https://mailman.videolan.org/listinfo/dvblast-devel
> >
>
> _______________________________________________
> dvblast-devel mailing list
> dvblast-devel at videolan.org
> https://mailman.videolan.org/listinfo/dvblast-devel
>
> ________________________________
>
> ONE Telecommunications Services Limited Liability Company Skopje
>
> This e-mail (including any attachments) is confidential and may be
> protected by legal privilege. If you are not the intended recipient, you
> should not copy it, re-transmit it, use it or disclose its contents, but
> should return it to the sender immediately and delete your copy from your
> system. Any unauthorized use or dissemination of this message in whole or
> in part is strictly prohibited. Please note that e-mails are susceptible to
> change. ONE Limited Liability Company Skopje shall not be liable for the
> improper or incomplete transmission of the information contained in this
> communication nor for any delay in its receipt or damage to your system.
> _______________________________________________
> dvblast-devel mailing list
> dvblast-devel at videolan.org
> https://mailman.videolan.org/listinfo/dvblast-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/dvblast-devel/attachments/20150626/2e0365f6/attachment-0001.html>
More information about the dvblast-devel
mailing list