dLive / SQ6 Dante De-Sync

[JamesNParticipant]

Hey,

We have a C3500 and MixRack for our main auditorium and an SQ6 for our secondary. The C3500/MixRack/Dante are on one main Cisco switch which trunks to a Cisco switch in the booth and then from that Cisco switch to a Cisco switch that the SQ6 is connected to. We started having an issue last year where the SQ6 and several of the patched Dante devices would loose sync and re-gain. This does not happen right away, but after some undetermined amount of time it will start and then every few minutes it will loose sync and then sync. Once it starts, it continues until we pull the trunk to the SQ6 side. Once we do, everything goes back to normal. We’ve tried a number of diagnostic steps, but we can’t get it to reliably start failing and so it is hard to work on since we have to have the trunk disconnected for normal operations.

Any suggestions would be GREATLY appreciated.

[andiaParticipant]

who is master (leader)? and how is leader configured?

[JamesNParticipant]

The MixRack (with Dante card) is set to Preferred Leader with “Sync To External”. The network that the audio equipment is on does not have access to the internet and there is no NTP server available. Would having one help?

[BrianParticipant]

It sure sounds like a setting in the network switches that is causing this issue. Dante has some very specific settings in order for it to work well. Ideally you just connect the two devices together without any switches in-between, but it sounds like the distance is great to easily accomplish this.

If you haven’t gone through the Dante Certification training (found on the Audiante site), I highly suggest that you go through level 1 and 2.

[TomParticipant]

What were the diagnostic steps you mentioned and what were the results?
Do you see a difference between unicast and multicast flows?

If you can’t simplify your network layout as Brian suggests, you should also look into Dante Certification Level 3.
Especially Clocking, IGMP and Troubleshooting.

The most simple reason could be that you set the latency lower than what the network can handle continuosly.
Cf. https://dev.audinate.com/GA/dante-controller/userguide/webhelp/content/latency%20tab.htm

Does the event log in Dante Controller give any clues?

[JamesNParticipant]

I’ve taken Level 3. It has been a while, so I will go review.
The Dante Controller logs don’t indicate why the issue is happening other than the SQ6 and a few other items desync.
The system had been working perfectly for some time before this started happening.
No changes were made to Dante prior to the issue (that I know of and I’ve reviewed settings).
I’ve used nmap to look for IP address conflicts and everything is as it should be.
I’ve rebooted all equipment (multiple times).
I’ve replaced network cables.
I’ve tried powering things up in different orders.
I can’t simplify the network layout as the equipment racks are far apart.
When it starts happening, I can only get it to stop if I disconnect the SQ6 from the network. The other items (amps, Dante devices, etc.) on the secondary auditorium side calm back down after the SQ6 is disconnected.

I can’t force it into the broken state, it seems to happen only after an indeterminate amount of time. This has been problematic for troubleshooting as each auditorium is used regularly.
I will look at the latency stuff and see if there is an issue there.
I have not checked the clocks to see if they are far out of sync. I am going to see if I can introduce an NTP server so that everything can sync to a real clock.

[SteffenRParticipant]

I have not checked the clocks to see if they are far out of sync. I am going to see if I can introduce an NTP server so that everything can sync to a real clock.

We talk about a digital clock signal for sample rate sync (To make the samples happen at the same time on the whole network).
NTP servers don’t help here.

When it starts happening, I can only get it to stop if I disconnect the SQ6 from the network.

That means you disconnect the Cisco switch and the SQ?
Can you try to connect the SQ direct to the Dante network without the switch?

I’ve used nmap to look for IP address conflicts and everything is as it should be.

Can you provide a network map?
How many devices are involved?

[BrianParticipant]

As a simple test, I would suggest that you physically move one console close enough to the other console that you can connect the two together via Dante and a single network cable. Connect a computer running Dante Controller to one of the other Dante network ports on one of the I/O cards. This will very quickly confirm if the problem is related to a setting in the console, I/O card, or Dante Controllers or if the problem is related to your network infrastructure (settings in switches, etc).

[JamesNParticipant]

We have around 35 devices on the Dante network (I’ll have to check on the exact number).
I can stop the de-sync if I pull the network cable from JUST the SQ6.

This is a rough network map:

[Backstage Main Rack]--TRUNK--[Main Booth Rack]-------------TRUNK----------------[Secondary Rack]--TRUNK--[Secondary Booth Rack]
-dLive MixRack                -Misc streaming                                    -Amp                     -Misc streaming
-Sure Receivers               -Mac Mini (Dante Controller, Sure monitor, etc.)   -Dante for misc sends    -SQ6
-Amps                                                                            -Sure Receivers
-Dante for misc sends
-Direct cable to CM3500  

I’ll try y’all’s suggestions here next Wed (the next time I will be able to try diagnosing). I’ll re-connect the trunk on Sunday and hopefully it will start doing it by Wed.

Thanks guys. I’ll report here what I tried and if anything changed.

[JamesNParticipant]

After trying many things, I discovered what the issues was. We have 8 different Cisco switches total that handle the AV network. Audio and Video are on different subnets. The trunks are single 1gb connections via Cat6. I would have preferred fiber trunks with 5gb or 10gb transceivers, but I was not involved in those decisions. There are two switches where a majority of the video equipment is racked. I discovered that two video streaming devices (for TVs around the facility) were together producing upwards of 75% utilization over the trunk ports. This traffic propagated to the entire network via the trunk even though no other switch needed this traffic. After isolating these switches from the rest of the trunks (wasn’t needed anyway), the problem is gone.

I had not thought to look at utilization because “I knew that we weren’t running enough in the network to possibly stress the utilization!” Let this be a lesson to you (and me for sure!) to check network utilization as part of your initial diagnostics when figuring out Dante issues.

[SteffenRParticipant]

That you are using the same switches for streaming, was missing information. With that info available, we had mentioned the trunks as possible bottlenecks before.

[Author]

Posts