CM-1 not present (Page 1 of 2)

We had similar issue with NX-32 just rolled out. After internal inspection of the NX we found loose ribbon cable connecting CM-1 card to the main board, reseated cable, reboot and the system has been working fine. This may have occured durring shipping???????

OK guys I think we got it. The "Managed Switch" appears to have been not so managed. It has now run for 12 days with no errors...........(*&%*%*&^!!!.... it's always the simple stuff.

Mike,

Is there is new line of switches that we have been hearing about that are considered a "managed" switch, but has NO user managable features.  They just set it up in a "standard" configuration.  This standard config greatly limits the broadcasts and multicasts, which will have great impact on CobraNet.

I have heard of this problem with a Linksys 4100 switch.  Is this what your problem relates to, or was it simply someone setting up a regular managed switch incorrectly?

Has anyone else on the forum run across these unmanagable--managed switch?

Could you be more specific about the incorrect switch settings? I am still having similar errors from one of my Nions and am quite interested in what happened in your case.

I do like that part about separating Ethernet from Cobranet.....

I learned a long time ago to keep you cobranet and eithnet separated. I have always accomplished this using separate hardware as apposed to using managed switches. This situation would mark the first time I have worked on system that used a managed switch. To fix the problem in this system we actually installed a small 8 port switch to handle the eithernet and left the cobranet on the managed switch. It has now been almost 4 weeks without an error of any type. The only thing that has shown up in the remote log is the users signing into Kiosk. I did not do the original install so I did not have access to the switch and I could not change the settings or see what they were. I checked it by plugging a network cable into the cobranet side of the switch and pinging an IP address on the eithernet side. After this I plan to always do it the way I have in the past. Separate the hardware. And I also have to tell you I have always used a 16 port, $60 Linksys switch. Nothing fancy, and I have never had any problems. I have one running in Baltimore for ESPN that has been running problrm free for 3 years.

Keep it Simple, It's only cool if it works....

ya, okay

I disagree with the argument that more hardware used in an unrelated fashion results in increased reliability. If you have a device with a MTBF of X, two devices now have a MTBF of X/2. If I want true reliability, I'll get switches with redundant, hot swappable power supplies/switching engines and use STP/RSTP with redundant links.

As for $60 switches, IMHO if I put all my eggs in one basket, I buy a nice basket(of which the switch is only one part). I've had problems w/ cheap network hardware(including HP Procurves for some reason), now I don't gamble. For the troubleshooting features alone, I think a decent managed network is well worth the investment.

The system does have an extremely high volume of traffic. 2 nionodes which are almost totally full on the transmit side going out to 20 or so Crown USP3 cards and 2 CAB4ns plus a couple of channels to a set of BSS Londons. It is a mess we are still trying to sort through. I am looking now at pulling all of the USP3 cards and installing a a couple of CAB 16os because all the amps are in the same rack.
The system was also originally installed with 2 HP switches and 2 back-up N3s. In my opinion it was way overkill for a BGM system. When the system was faulting it would take out both the primary and the back-up so it didn't do any good anyway.

Jason-

Not to speak for Mike, but if this is the project I am thinking of, they (the organization that put it in, not Mike or the company that he works for) put all the NIONs in one project rather than separating the "primary" system from the "backup" system with separate projects. That is, they had redundant processing cores with all the Cobranet I/O set to switch from one set of DSP to the the other. The consequence of what they did is that if something went wrong with anything in the "primary" part of the system, the entire project (both "primary" and "backup" sections) would go offline. This is quite opposed to the way this type of system should be done, with separate projects for "primary" and "backup" and project link devices to keep both projects current with one another.

I also agree with you, Jason, in that the simplification of rack wiring by using Cobranet to the amplifiers should not go unnoticed. I didn't realize how much easier it was to build this type of system until I actually started doing it. I mean, it's only a couple wires per amp... but it really is faster and easier... especially when you need to start reconfiguring things half way through building the project... but that never happens to anyone here... 

(edited for clarification of the way the original system was set up)

Last edited by JoshM (2008-05-28 14:37:43)

One thing I haven't seen anyone address here is the Cobranet configuration in the NIONs. I'll assume the CM-1 configuration is advanced mode in both NIONs from the looks of things. If the CM-1 is in advanced mode, it is absolutely mandatory that the value in field labeled "Num Chan"  match the number of channels in the bundle. If the value is left at the default value of 8, the CM-1 will transmit all 8 channels. If the bundle is only using 2 channels for audio, the CM-1 will transmit the 2 channels of audio plus 6 channels of null data. If this is overlooked, the CM-1 will be easily overwhelmed if there are more than 4 bundles transmitted. There just aren't enough processor cycles or memory to keep up. How many multicast bundles are in use? The limit is 4 on any single network. Excessive multicast traffic will bring a CM-1 to its knees in a hurry.

Regarding the log entries: The peek/poke errors and CAB errors are consistent with a CM-1 that is overwhelmed or broken. I tend to think the former. What is reported in the log if the CM-1 is disconnected from the network? Do the errors cease? Are the errors present when all bundle numbers are set to 0? How many bundles can be turned on before getting errors? Lots of things to look at before condemning the network.

Last edited by ivorr (2008-05-28 16:34:15)

Mike,

I'd like to back up a bit on this problem and take it in steps. In the initial post with the log, the CobraNet module appears completely unresponsive.
Question: What did you have to do to get the CM-1 back up and running?  Did a hard reset (or what) restore the functional state?
Did a watchdog in the CAB reset the card (does a CAB have a watchdog??).

Some of the prior posts are theorizing that it could be an issue with the CM-1 being configured such that the processor is overloaded and ceasing to function. This may be, but is probably not, the case. If this were true then the CM-1 would not work properly after separating the CobraNet and data networks.

And you said that there is a lot of CobraNet traffic on the net. This, in and of itself is unimportant. The important point, with this type of failure, is how much traffic is appearing at the Ethernet port of the failing device. The aggregate traffic should be of no concern in this type of failure on a switched network.

As of now, this sounds like a case of the Etnernet port on the CM-1 being inundated with too much traffic generated by data devices.
There are known cases where a CM-1 can experience a stack overflow and become unresponsive in the presence of massive Ethernet traffic.
Furthermore, there are almost certainly no data devices on the network that would or should 'know' about the CobraNet device and then  be able to specifically address it with any packets. So this excess data traffic is then almost certainly multicast or broadcast traffic. It would be easy enough to verify this. Run Ethereal or Wireshark on any PC connected to the network and capture the traffic. All multi/broadcast traffic will appear at the PC as it will appear at all other nodes.
Massive amounts of multicast traffic is abnormal on a properly configured data network. So look at the trace and figure out where the bad traffic is coming from and see if it can be corrected. This traffic is also not healthy for the now separated data network and should be corrected if possible. If it can be corrected you may be able to go back to a shared net. But if doing so, and considering that you have managed switches, I would suggest logically separating the traffic using VLANs.

BTW, multicast or broadcast traffic can be identified in the trace by looking at the least significant bit of the most significant byte of the destination address. If the bit == 0 then it is unicast, if it == 1 then it is multicast. If the whole address = 0xffffffff then it is the broadcast flavor of multicast.

Please look into the above issues and reply to us with what you find.

Thanks