[JamesNParticipant]

After trying many things, I discovered what the issues was. We have 8 different Cisco switches total that handle the AV network. Audio and Video are on different subnets. The trunks are single 1gb connections via Cat6. I would have preferred fiber trunks with 5gb or 10gb transceivers, but I was not involved in those decisions. There are two switches where a majority of the video equipment is racked. I discovered that two video streaming devices (for TVs around the facility) were together producing upwards of 75% utilization over the trunk ports. This traffic propagated to the entire network via the trunk even though no other switch needed this traffic. After isolating these switches from the rest of the trunks (wasn’t needed anyway), the problem is gone.

I had not thought to look at utilization because “I knew that we weren’t running enough in the network to possibly stress the utilization!” Let this be a lesson to you (and me for sure!) to check network utilization as part of your initial diagnostics when figuring out Dante issues.

[JamesNParticipant]

We have around 35 devices on the Dante network (I’ll have to check on the exact number).
I can stop the de-sync if I pull the network cable from JUST the SQ6.

This is a rough network map:

[Backstage Main Rack]--TRUNK--[Main Booth Rack]-------------TRUNK----------------[Secondary Rack]--TRUNK--[Secondary Booth Rack]
-dLive MixRack                -Misc streaming                                    -Amp                     -Misc streaming
-Sure Receivers               -Mac Mini (Dante Controller, Sure monitor, etc.)   -Dante for misc sends    -SQ6
-Amps                                                                            -Sure Receivers
-Dante for misc sends
-Direct cable to CM3500  

I’ll try y’all’s suggestions here next Wed (the next time I will be able to try diagnosing). I’ll re-connect the trunk on Sunday and hopefully it will start doing it by Wed.

Thanks guys. I’ll report here what I tried and if anything changed.

[JamesNParticipant]

I’ve taken Level 3. It has been a while, so I will go review.
The Dante Controller logs don’t indicate why the issue is happening other than the SQ6 and a few other items desync.
The system had been working perfectly for some time before this started happening.
No changes were made to Dante prior to the issue (that I know of and I’ve reviewed settings).
I’ve used nmap to look for IP address conflicts and everything is as it should be.
I’ve rebooted all equipment (multiple times).
I’ve replaced network cables.
I’ve tried powering things up in different orders.
I can’t simplify the network layout as the equipment racks are far apart.
When it starts happening, I can only get it to stop if I disconnect the SQ6 from the network. The other items (amps, Dante devices, etc.) on the secondary auditorium side calm back down after the SQ6 is disconnected.

I can’t force it into the broken state, it seems to happen only after an indeterminate amount of time. This has been problematic for troubleshooting as each auditorium is used regularly.
I will look at the latency stuff and see if there is an issue there.
I have not checked the clocks to see if they are far out of sync. I am going to see if I can introduce an NTP server so that everything can sync to a real clock.

[JamesNParticipant]

The MixRack (with Dante card) is set to Preferred Leader with “Sync To External”. The network that the audio equipment is on does not have access to the internet and there is no NTP server available. Would having one help?

[Author]

Posts