Troubleshooting of a telemetry system (from RTU to the SCADA computer facilities)

Effective troubleshooting of a telemetry system

This technical article reviews certain methodologies that may be followed for effective troubleshooting of a telemetry system from the digital or analog field input/output at the RTU to the computer facilities at the central site.

Basic troubleshooting of the various components of a telemetry system will be covered here.

This includes the following:

1. The RTU and component modules
   1. Associated equipment interfaced to the RTU (such as PLCs)
   2. Radio transceivers
   3. Antennas and antenna feeder systems
2. The master station
3. The central site computer facilities
4. The operator station and software

IMPORTANT NOTES //

Ensure that components are not removed on line whilst the system is powered up unless the manufacturer specifically indicates that this is permissible. Damage to components and modules can occur when removing whilst the system is still powered up.

Ensure that the antenna system is not disconnected from the system unless a dummy load has been installed, otherwise the radio power amplifier may be damaged.

1. The RTU and component modules

A typical procedure to follow when reviewing the operation of the telemetry system for faults (either for intermittent or outright failure) is:

• Confirm that the power supply module is healthy. Check the main fuse or circuit breaker of the equipment rack or unit if no power is evident.
• If the power supply is not operating, check that there is power to the power supply module. If there is power to the module then replace the power supply module.
• Check central processing (CPU) module that the run or healthy light is on. Switch the CPU module to run mode if not running.
• Check earthing connections for low resistance to earth or whether some other drive hardware (such as a variable speed drive) has been added to the system.
• If CPU module will not run, check the configuration program to see whether this is faulty or not. Reload the program if indications are that it has become corrupted. Check that the configuration of the system matches that of the hardware.
  
  Load up a simpler program that you know works, if the program is possibly defective. Back up defective program onto disk (for future analysis) and reset the memory.
• Cycle power to the RTU before the new program is loaded.
• Replace the CPU and retry the test.
• Check that the modem module is operating. Confirm that the modem is operational and that it is transmitting and receiving data by examining the transmit (TX) and receive (RX) lights on the front panel.
• If the modem is not operational, replace the modem module (or desktop unit).
• If the modem module is not operating correctly, perform the local and remote loopback tests as described in the modem section.
• Check each analog and digital input/output module for status such as healthy or run.
• Check for possible heating problems in the system cabinet.
• This could be due to failure of an air-conditioning or fan unit (if installed) or excessive ambient temperature.
• If a module indicates no power at all – check the fuse for that module.
• Replace each module if the indications are not healthy.

Check each individual module as follows:

Analog input modules

• Check that there is current or voltage being injected into the signal inputs.
• Check the fuse is installed.
• Check the scale and span and compare with the appropriate register tables for accuracy.
• Adjust scale and span either via software or via pots on the card.

Digital input module

• Check that there is current or voltage at the signal input to the module.
• Check the fuse for each input.
• Check earthing connections.

Interface from RTU to PLC (RS-232/RS-485)

• Check for the transmit/receive/run lights on the interface unit.
• Check interface data communications link.
• Check that the radio unit is operating (if used). The on light should be on and no fault lights should be on.
• If the on light is not on, check that there is DC power to the radio. If there is power, check that the fuse or circuit breaker is healthy.
• Check that all coaxial connectors are secured properly.
• Check that the PTT – Push to talk (transmit) light on the radio comes on when the telemetry unit feeds data into it for transmitting. Check that there is sufficient audio level into the transmitter.
• Check that the mute (receive) light on the radio comes on when the radio receives RF data. Check that there is sufficient audio level into the telemetry unit from the receiver.
• Check that the voltage standing wave ratio into the antenna is 1.5 or less.
• Check that the RF output power from the transmitter is as specified.
• Check that the antenna is aligned in the correct direction and with correct polarization.
• If the radio is still not working correctly, a radio test set will be required to check transmitter deviation, RF distortion, audio distortion, receiver sensitivity at 12 dB SINAD, transmitter and receiver frequency errors, transmitter/receiver isolation and transmitter spurious outputs.
• If a landline is to be used, then first ensure that the telemetry and modem equipment is operational.

Then if the line is a:

Privately owned cable

• Check for end to end connectivity
• Measure noise level on line
• Check for crossed pairs
• Check main distribution frame (MDF) and intermediate distribution frame (IDF) connections
• Check earthing

Switched telephone line

• Listen for dial tone
• Connect up a standard telephone and make a normal telephone call
• Listen for noise levels
• Call out the telephone company

Analog or digital data links

• Check the run, connect, transmit and receive lights on the modem
• Check the operations manual and the communications software
• Call out the telephone company

Go back to contents ↑

2. The master sites

The master sites will generally consist of a more complete telemetry unit and higher quality radio equipment. The same troubleshooting techniques can be applied to the master site as are used at the RTUs. Additional equipment would include links to other master sites, to the central site and computer control facilities.

The additional checks that will be appropriate include:

1. Check that the link to the central site is operating correctly.
2. If it is a radio link, carry out the check as discussed in the last section.
3. If it is a microwave link, check that the transmit and receive lights are on.
4. Check that the BER alarm light is not on.
5. Check transmit power.
6. Check receiver sensitivity.
7. Check antenna alignment.
8. Check connectors are secure and the cable or waveguide has not been damaged.
9. Check individual multiplex cards for alarms and power fails.
10. Check input/output levels from multiplex cards.
11. Check for clean (noise free) healthy power supply to microwave equipment.
12. Check link fade margin.
13. As a last resort, carry out BER tests on each channel.
14. If there is a master site computer, check that it can carry out all its required functions, i.e. monitoring of radio and RTU performance, status and alarms, etc.

The next section offers some hints as to how to troubleshoot a computer system.

Go back to contents ↑

3. The central site

The areas in which to troubleshoot problems here are quite varied as the master station consists of:

1. The operator stations
2. The software for the system
3. The communications network for the operator stations

The radio and the antenna systems have already been covered in the previous section. However the operator stations, the communications network and the associated software will be covered in this section.

Go back to contents ↑

4. The operator station and software

There is not much that can be done here if a system fails or has intermittent problems except to systematically replace each connected unit to identify the faulty module.

There are however a few problems which can be examined:

• Operator terminal locks up intermittently
• Check the power supply to the system for possible electrical spikes or transients. This can be done with power analysis equipment (e.g. the dranetz) or by putting the entire system onto a battery supply.
• Check for any new electrical drives or pieces of equipment, which have recently been added to the system (and which may add harmonics to the system).
• Check the earthing cable connections that the impedance to earth is still to specifications (typically less than 1 ohm).
• Do a software check on the hard disk of the operator terminal for possible corruption of software or failure of the disk. Backup the system, reformat the hard disk and reinstall the software on the disk.
• Replace the motherboard on the operator terminal (this probably indicates that the operator terminal should be replaced with another system).
• Throughput of the operator station and associated system drops off dramatically
• This manifests itself in slow updates of data on the operator terminal.
• Check the system for errors being introduced on the data communications lines by electrical noise or earthing problems. The data communications system could be sending multiple messages due to errors introduced by electrical noise.
• Check the local area network for potential overload due to excessive traffic. Reduce the traffic by reducing the amount of data being transferred or split the systems up into separate networks (using bridges).
• Check the radio, microwave, landline, and antenna systems for possible introduction of noise and error problems.

Yokogawa Stardom CPU redundant test (VIDEO)

Go back to contents ↑

Reference // Practical SCADA for Industry by David Bailey and Edwin Wright