1 Topic by tucan

  • tucan
  • NWare Pioneer
  • Offline
  • From: London
  • Registered: 2007-09-07
  • Posts: 64

Topic: NWare 1.4.4 and CAB 8i's 8o's dropping offline

I am on site with a large system with  3 Nions and a conman and 18 CAB 8i and 27 CAB 8o.
The CABS have Cobranet firmware 2.9.12 and the Nions 2.9.16.

The cabs are not split evenly between the Nions as they are split across logical phases of the project instead, which are themselves split between Nions. Therefore,
Nion 1 - 7 x CAB8i and 15 x CAB8o - Nion has 8 Tx bundles, 7 Rx bundles
Nion 2 - 4 x CAB8i and 5 x CAB8o - Nion has 5 Tx bundles, 2 Rx bundles
Nion 3 - 7 x CAB8i and 7 x CAB8o - Nion has 7 Tx bundles, 2 Rx bundles

the CABS are largely present for their GPIO and the 8i's are not necessarily all TXing audio! We are also using serial bridging to transmit RS485 around site for monitoring other parts of the system

The problem I am having is that CAB8o's on Nion 1 seemingly randomly (but definately some more than others) drop offline ocasionally (their link light goes out in Nware) for a second or 2. In a 1 minute period one device will probably have gone off an online once. I have not been able to monitor to see whether the physical link light on the CAB goes out, or whether audio is halted as well, but I suspect not.

The only way i can see to improve this situation is to:-
Ensure that a CAB is a conductor rather than a Nion1
Try and combine the TX bundles on Nion1 into as few multi Unicast bundles as possible
Relocate some of the CABs from Nion1 to Nion2
Relocate the Cab control through Conman - would this make any difference?

Is anybody surprised by what i am seeing or is it what you would expect in the real world with so many CABS and bundles on one Nion (Nion1)?
I can not be certain but until last week this system was running on Cobranet 2.9.12 and Nware 1.2.6, and I dont remember seeing anything dropping on and offline.....although I was NOT monitoring it closely.


Any other advice or ideas gratefully received

2 Reply by Fergy

  • Fergy
  • MM Tech Support
  • Offline
  • From: The Home Office--Meridian
  • Registered: 2007-08-15
  • Posts: 334

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

The best solution is using the ControlManager to control the CABs.  NioNode #1's CM-1 is overwhelmed, and I would be surprised if this has not been a problem in the past, because the audio continues, but control of the CAB is lost.

Your list above is almost reverse order of what I would try, except I might move the conductor up (or down) the list.  A best practice would be to always load balance control of your CABs among your NioNodes.  Lastly, I would never expect a single NioNode CM-1 to be able to control 22 of anything.  This is not a limitation of Nion, but of the CM-1 itself.

Good luck.
Scott

Make it intuitive, never leave them guessing.

3 Reply by tucan

  • tucan
  • NWare Pioneer
  • Offline
  • From: London
  • Registered: 2007-09-07
  • Posts: 64

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

I did look in detail at putting the control of the CABs all on Conman, a while ago (Nware 1.2.4 I think) but the complication I came up against is that if one unit in the system say Nion1 has a fault and stops running then the conman carries on transmitting the control data. This is a problem because I have dual redundant systems (2 identical systems) and when a role in the Master system shuts down, the secondary system (running the same project) detects this and takes over. You can then end up with Conman on the SLAVE system controlling the same CABS as CONMAN on the MASTER system at the same time....and all the chattering relays and unpredicatable states that that means!

CABS are enabled and disabled by controlling their ID and bundle numbers to 0 when they are disabled.
So the problem to solve is what  happens if a CAB is located on Conman but its audio bundle is sent from Nion1....

I suppose what I need to do is to bite the bullet and have a moitoring script 'placed' on EVERY node in a system, that monitors the other nodes in that system, and if any node fails to respond, then the node where the script is, sets its local nodes bundles (in the case of Nions) or CAB ID and CAB bundles (in the case of Conman) to 0

.....it would be nice if the project option 'mute on failure' stopped transmitting and receiving audio bundles as well as sending CAB control data.....

Last edited by tucan (2008-11-23 19:31:47)

4 Reply by jvalenzuela

  • jvalenzuela
  • Initech programmer
  • Offline
  • From: Orlando, FL
  • Registered: 2007-10-10
  • Posts: 223

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

What would happen if you place Conman on its own project and project-link it to the master and slave projects?
It can then monitor each project and accept controls from one in order to direct them to the CABs.

5 Reply by tucan

  • tucan
  • NWare Pioneer
  • Offline
  • From: London
  • Registered: 2007-09-07
  • Posts: 64

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

The problem with that is that you would only have a single conman.... a single point of failure

6 Reply by jvalenzuela

  • jvalenzuela
  • Initech programmer
  • Offline
  • From: Orlando, FL
  • Registered: 2007-10-10
  • Posts: 223

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

To be honest, can you truly look at the system as a whole and not find any other single points of failure? Is the LAN designed to be fully redundant for both switches and links? Are there only one set of CABs? Last time I checked, the old style CABs do not utilize redundant Ethernet connections. Heck, every single analog audio connection that is not monitored and redundant is a possible single point of failure. At any rate I find that you have to pick your battles when designing 'redundant' audio systems. Are we trying to only have redundant Nion/Conman systems?

Why not use two Conmans? Two Conman projects and two Nion projects. It should be possible for each Conman to monitor the other Conman and both Nion projects.

7 Reply by tucan

  • tucan
  • NWare Pioneer
  • Offline
  • From: London
  • Registered: 2007-09-07
  • Posts: 64

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

I take your point about redundancy in general, however these are life safety systems and as configured, you can lose any one thing and the system as a whole stays up even though you may have lost a CAB and and therefore a particular zone, but eveything else stays up..... an importantly, you know that you have lost part of the system.

We do actually have a conman in each system but the thought of splitting the job into four seperate projects is scary when trying to make updates. We arrived out our architecture when project linking was far from reliable, and we use one identical project deployed to both the MASTER and SLAVE systems and then each instance of that project then communicates with the other to see whether it is there and active before deciding whether to take control of the CABS or not.

8 Reply by jvalenzuela

  • jvalenzuela
  • Initech programmer
  • Offline
  • From: Orlando, FL
  • Registered: 2007-10-10
  • Posts: 223

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

Are you actually modifying values within the CABs as part of normal operation? If not, have you considered Cobranet interfaces that do not require control? At risk of being excommunicated for mentioning a competitor's product, off the top of my head Whirlwind makes some simple Cobranet analog I/O interfaces that do not require software setup. I'm sure there are others. This approach solves the control issue by completely removing it, sometimes simpler is better. Of course this ship may have already sailed if the system is already installed.

9 Reply by tucan

  • tucan
  • NWare Pioneer
  • Offline
  • From: London
  • Registered: 2007-09-07
  • Posts: 64

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

System is already installed..... however  the real reason we have gone with the CAB 8o's and 8i's is their robust and simple GPIO. In these projects it is not unusual to have CABS NOT passing any audio!!!

We therefore do modify cab values frequently, iand also use the serial bridging functionality, however there are a few instances where the Whilwind boxes have come in handy in the past.

10 Reply by jvalenzuela

  • jvalenzuela
  • Initech programmer
  • Offline
  • From: Orlando, FL
  • Registered: 2007-10-10
  • Posts: 223

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

tucan wrote:

.....it would be nice if the project option 'mute on failure' stopped transmitting and receiving audio bundles as well as sending CAB control data.....

I've done something like this before. Take a GPIO output that goes false on failure and wire it to a relay. Use the relay's N/O contact to power a 24VDC network switch through which all the bundles and CAB control for that project flow. It would be conceivable to use the N/C contact on the same relay to power a similar switch that handles the bundles and CAB control for the backup project. When the relay is energized the master project has connectivity to the CABs. The break before make nature of the relay contacts ensures that only one project has access to the CABs at a time. Rube would be proud...

There are some flaws with that topology which I'll leave to others to find wink, but it would probably do what you want.

PS: go ahead and setup the CAB control with the Conman nodes like Fergy mentioned. It is definitely The Right(TM) thing to do.

Last edited by jvalenzuela (2008-11-26 17:10:31)

11 Reply by ivorr

  • ivorr
  • ivorr
  • DSP Geek
  • Offline
  • From: West Des Moines, IA
  • Registered: 2007-08-17
  • Posts: 56

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

Did anybody mention that a NION can only control 8 CABs? Just checking...

The only true wisdom is in knowing you know nothing. -Socrates

ivorr's Website

12 Reply by tucan

  • tucan
  • NWare Pioneer
  • Offline
  • From: London
  • Registered: 2007-09-07
  • Posts: 64

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

Just curious to know where I would ever have come across that information before?

13 Reply by ivorr

  • ivorr
  • ivorr
  • DSP Geek
  • Offline
  • From: West Des Moines, IA
  • Registered: 2007-08-17
  • Posts: 56

Re: NWare 1.4.4 and CAB 8i's 8o's dropping offline

I know we teach it in the NION training class but I'm presently unable to tell you where it's written down. The reasons for the limit are deficiencies in the CM-1 module. There are not enough processor and memory resources in it to manage any more than 8 devices reliably. It is possible for a NION to control more than 8 devices but its CM-1 module can't have much audio transport going on. Cirrus Logic says they'll soon be making improvements to the CM-1 including a bigger processor and more memory. We shall see when it arrives. Of course, NControl (formerly ConMan) can control almost any number of devices on the network.

As far as where to read all about it, I will make sure our technical writer is aware of the limitation and that it shows up in the next NWare helpfile.

The only true wisdom is in knowing you know nothing. -Socrates

ivorr's Website