TERAGO NETWORKS – RF TECHNOLOGY

RDL-3000
Troubleshooting
Technology Guide

PREPARED BY:
Clarence Espejo

Date:
rd
November 23 , 2013

Revision:
V1.0
 TERAGO NETWORKS – RF TECHNOLOGY

Table of Revisions

Date Issue Description By

23-NOV-13 1.0 Origination of the document CE

27-Mar-15 1.a Detailed Link Summary & Auto Scan Status WA

Table of Contents
 TERAGO NETWORKS – RF TECHNOLOGY

List of Figures.......................................................................................................................3
Scope....................................................................................................................................4
System Event Log.................................................................................................................4
Statistic Screens...................................................................................................................4
Link Summary......................................................................................................................4
System Behavior..................................................................................................................7
Troubleshooting...................................................................................................................8
*Link Summary................................................................................................................8
Common Issues....................................................................................................................9
Subscriber Station Down.............................................................................................9
Subscriber Station Packet Loss..................................................................................10
When to Suspect RF Interference..................................................................................11
Channel Changing......................................................................................................12
RF Monitor.........................................................................................................................13
*Additional debug features showing detailed link summary ……………………………………....16
*Additional debug feature Auto Scan Status on Subscribers……………………………………….17

List of Figures
Figure 1-Link Summary........................................................................................................6
Figure 2-Link Summary........................................................................................................7
Figure 3- RF Monitor Login................................................................................................13
Figure 4- RF Monitor Logging – RDL3000..........................................................................14
Figure 5- RF Monitoring AN80i..........................................................................................15
 TERAGO NETWORKS – RF TECHNOLOGY

Scope

This section discusses the troubleshooting tools available for use during RDL-3000
troubleshooting. The RDL-3000 system has some tools built into its interfaces; these will
be covered first. There are also a number of external tools available, which will also be
highlighted.

System Event Log

System events for both the SC and SS on a PMP system are captured in their respective
event logs. The event log for both devices can be accessed via the CLI by entering the
command “show log” from the root menu.

172.17.126.204# show log

11/18/2013 14:02:32 1008 - Network Configuration Save: OK
11/18/2013 14:04:24 1008 - Network Configuration Save: OK
11/18/2013 23:39:29 1032 - SNTP: Time received: OK
11/22/2013 09:47:06 1075 - Link (7) [00:09:02:14:DD:21]: DOWN
11/22/2013 17:39:31 1032 - SNTP: Time received: OK
172.17.126.204#

Both devices order their event log with the most recent event last in the list. Each event
is time stamped with either the system time, or a time of day obtained from a time
server if it has been configured to do so.

Neither device stores its event log in non-volatile memory, meaning that all information
will be lost when the unit reboots. If you plan on rebooting, or otherwise interrupting
power to your device, make sure to capture the event logs before doing so.

Statistic Screens
The RDL-3000 provides various screens showing Wireless Ethernet Statistics and
Ethernet Port Statistics, Link information is also available for each links. Statistical
information for the links will be explained in detail in the sections below.

Link Summary
 TERAGO NETWORKS – RF TECHNOLOGY

Log into the SC and navigate the left menu to select Link Summary. The display will
identify all configured links by name (whether installed or not) and the link’s status.
Links which indicate a green check mark ( ) have wireless connections and are
registered with the sector controller. Links which indicate an orange triangle with an
exclamation mark ( ) have no wireless connection and are not registered with the
sector controller or are disabled.

For RDL-3000 Subscriber Links

For every link you will be provided the SINAD for the DL and UL for both RF ports 1 and
2. In addition, you will be provided the RSL in the DL and UL for both RF ports 1 and 2 on
RDL-3000. Statistics for both RF1 and RF2 must meet within designed specification for
RSL and SINAD to maintain the highest UBR achievable.

Redline AN80i Subscriber Links

With every link you will be provided the SINAD and RSL in the DL and UL for both RF
ports 1 and 2. Because the AN80i is SISO and only uses 1 RF port to transmit and
receive, you will observe the SINAD is sync’d for UL on RF1/2 and DL on RF1/2. In
addition, the operating RSL on the active RF port will be provided, however the inactive
port will show -90dB or less.

For both Radio types, the UL/DL UBR will be displayed and will dynamically shift when
adaptive modulation is enabled. The UBR will modulate up and down independently of
each other dependent on the Packet Error Rate

Link summary also displays the Total Wireless Packets transmitted over the wireless
interface. This is the Total number of packets processed over the wireless interface and
does not include any packets which were discarded or errored. Re-transmitted are the
number of wireless packets re-transmitted over the wireless interface. Lost are the
number of wireless packets discarded due to errors.

To obtain further statistics on a per link basis, Click on the ID status ( ) to bring up
the Subscriber Link status. This screen will provide additional details on the subscriber
such as:

 IP (which can be clicked on to bring up management to that device)

 Link Uptime since the wireless link has been active (not power up time). This will
reset, everytime the wireless link drops and recovers.
 Link Lost count which is the number of times the link drops.
 Active services which is the number of active services on the link
 Data Link Condition indicates whether the service is enabled or disabled
 TERAGO NETWORKS – RF TECHNOLOGY

Figure 1-Link Summary

In addition, the same RF performance statistics found in the link summary are displayed
subscriber link status.
 TERAGO NETWORKS – RF TECHNOLOGY

Figure 2-Link Summary

System Behavior

The system will try to maintain an error free wireless link to each subscriber dependent
on the RF parameters experienced by the system in both the uplink and downlink.

Each parameter such as RSL, SINAD, operating UBR and packet statistics are used by the
system to determine capabilities in capacity performance. The critical component in
achieving the maximum supportable modulation (UBR) is have the link in both UL and
DL separate from the general noise floor with the greatest margin possible or the
highest signal to interference and distortion ratio. The greater the margin from noise to
interference, the higher the system can operate in modulation. The system will always
be set to the maximum modulation to allow the most efficiency of the system scheduler
and therefore the most capacity dependent of RF conditions.

Should the SINAD start decreasing from the RSL dropping or the interference level
increasing the data transmission will start to generate errors in either the UL or DL.
When the DL or UL experiences a packet error rate of 10-9 the system will drop in
modulation till the system can achieve a PER of >10-9. This ensures the data is
transmitted error free and will continue downward till the lowest modulation is
reached.
 TERAGO NETWORKS – RF TECHNOLOGY

Keep in mind that as the modulation/burst rate drops the available bandwidth will also
be reduced and may go lower than the customer’s service agreement. In this case

corrective measures must be implemented to increase the SINADR and Burst rate to a
level which can support the customer’s service. This may include channel changes, re-
alignment or polarity changing.

Below is the SNR requirements for each Burst rate.

Modulation Burst Rate Burst Rate SNR

64QAM 3/4 108 54 20
64QAM 2/3 96 48 18
16QAM 3/4 72 36 14
16QAM 1/2 48 24 11
QPSK 3/4 36 18 8
QPSK 1/2 24 12 6
BPSK 1/2 12 6 4

Troubleshooting

Link Summary

Because the Link Summary displays the sector view of all links including their respective
SINADR,RSSI, Burst Rate and Retransmissions it is easier to identify issues which are
occurring sector wide or on specific customers. This will help in isolating issues and
actioning corrective measures. Included below are a few samples:

Sector (Uplink)
 Symptoms: Low UL SINADR from all SS, RSSI is within 3db from calculated,
incrementing UL re-transmissions on all SS

o Possible causes: interference at the Sector antenna causing degraded uplink

statistics, localized weather events or LOS obstructions

o Corrective Measures: Survey scan at the AP to identify interference and clear

channel selection
 TERAGO NETWORKS – RF TECHNOLOGY

 Symptoms: Low DL SINADR, RSSI is within 3db from calculated, incrementing DL

re-transmissions on all customers

o Possible causes: localized interference at the customer antenna causing

degraded uplink statistics, localized weather events or LOS obstructions.

o Corrective Measures: Survey scan at the Customers SS to identify

interference and clear channel selection

Subscriber (Downlink)
 Symptoms: Low DL/UL SINADR, RSSI is worse than 5db from calculated,
incrementing DL/UL re-transmissions
o Possible causes: interference at the customer antenna causing degraded
uplink statistics or customer antenna requires re-alignment

o Corrective Measures: Survey scan at the Customers SS to identify any

interference and action antenna re-alignment

In the link summary you may also click directly on each link to see individual customer
link statistics.

Common Issues
Subscriber Station Down

Log into the Sector Controller and go the Link Status screen. Verify the status of the link
(Up/Down).

 If the link is up and there is no connectivity to the Subscriber (NMS or pings)

verify that the clients NMS Group and Connection IDs are configured correctly on
the Sector Controller.

 If the client is Down, verify that the Units MAC address is entered correctly on
the Sector Controller via the Link Config. If the MAC is entered incorrectly the
Subscriber Station will not be allowed to register. If the MAC address is
configured correctly, verify whether there is RF interference on other links which
may be preventing Subscribers from registering (Down) or constantly
deregistering/registering (Up/Down). If RF interference is not the cause then the
client may require Field Technicians to investigate LOS/Cable/PoE adapter.
 TERAGO NETWORKS – RF TECHNOLOGY

Subscriber Station Packet Loss

Identify the Subscriber Station(s) which are experiencing packet loss by pinging directly
to the Radio with a maximum of 500Bytes. Once the Subscriber(s) experiencing packet
loss are identified log into the sector controller and go the Link Statistics or Link
Summary page to investigate the following parameters:

 Link Up Time: Link up time is the time since the link has been operational since
SS registration. Link up time is reset every time the link is lost either due to SS
radio reboot or due to severe interference. Every such incident of link lost will
increment the link lost counter. Links with short Uptimes should have their PoE,
Cable,LP and all ground points and also the Building power inspected if the link
lost cannot be attributed to user controlled reboots or to external interference.

 Link Lost Count: This value will indicate the amount of times the Subscriber
Station de-registers from the Sector Controller. Subscriber Stations with high
counts (>5) not attributed to initial turnup/alignment or Radio Reboots (as
indicated by the uptime) may be experiencing RF interference dropping the
wireless link, this may be attributed to faulty installations as well. Thus a check of
the connections should be made.

 SINADR: The Signal to Noise value displays the amount of separation between
the radios Carrier Signal and Interference in dB. Lower values under the
minimum required for the set or current burst rate indicate low signal and/or
high noise meaning the radio will have difficulty distinguishing its carrier signal
from noise. Verify the current Burst Rate for both UL and DL and compare the
current SINADR to the Minimum SINADR in the Redline Capacity Reference
(Fig.13). If the current SINADR is lower than the minimum required for that Burst
Rate then the link will be experiencing Blocks Retransmitted or Lost frames.

 RSSI: RSSI should be maintained within +/- 5db of calculated levels as provided
by the Link Budget tool.

 Link Burst Rate: For links with Adaptive Modulation enabled, the DL or UL burst
rate may fluctuate up and down dependent on the current packet error rate. As
RF conditions worsen due to interference or environmental conditions, Burst
rates will drop to ensure the data can be transmitted error free. Link Burst rates
should be maintained at the maximum rate supportable. Any link not operating
at the highest UBR setting must troubleshot through RF interference mitigation
techniques such as changing sector RF frequency to a cleaner channel, Shielding
antenna, moving antennas, increasing TX Power, or replacing antenna with
higher gain antennas etc. should be used (refer to RF Design prior to increasing
TX Power and antenna size). If the link cannot maintain the highest UBR the
 TERAGO NETWORKS – RF TECHNOLOGY

maximum DL/UL UBR may be set at a rate which supports the customer’s
service.

 Lost Frames: indicate the control field of the wireless frame is corrupted and the
wireless frame is discarded. This occurs during severe RF issues as the control
frame is transmitted using the most robust modulation. Use the RF monitor tool
to verify SINADR stability overtime. Ensure Lost frames do not exceed a BER of
10-3 or 1 lost frame per 100 blocks transmitted.

 Blocks Retransmitted: Incrementing Block retransmissions indicate the radios are

constantly retrying the transmission of data. If this value is increasing, check the
SINADR and RSSI and ensure they are within spec as this indicates a problem
with the wireless link. Ensure that Blocks re-transmitted count does not exceed a
BER of 10-3 or 1 retransmission per 100 total blocks transmitted.

 Duplex Mismatch: To identify speed/duplex mismatches log into the radio’s GUI
and go to System status. Under Ethernet Port Statistics there is a counter for Rx
Packets discard. An increasing value indicates Ethernet errors on the radios
Ethernet port. This can be caused by either a speed/duplex mismatch or
problems with the cabling (POE to radio or POE to switch).

In addition, you can also use the “show stats eth0” command in the CLI to
identify Duplex mismatches by displaying the Radio’s Ethernet port statistics for
any CRC error packets.

To verify the radios Ethernet port settings when the port is set to AUTO or fixed,
log into the radio using telnet and use the following command:

get ethsts

The system will return the current port status:

ethsts = 100hd

When to Suspect RF Interference

 
Before suspecting interference, first verify the customer’s traffic to ensure they are
utilizing under their subscribed amount and that the current burst rate can support that
requirement. If the customers burst rate is set to low or if the customer is over-utilizing
the link it may produce block discards and frame losses even without interference. If
burst rate is selected optimally, and you are still experiencing wireless discards and lost
frames, then there is a high probability that they are caused by interference.
 TERAGO NETWORKS – RF TECHNOLOGY

The location of the discards and the lost frames can also help isolate the source of the
problem. For example, if you are seeing a high rate of lost frames or discards in the
uplink, then the interference is most likely affecting the sector controller’s up link
reception. Similarly, if the lost frames or discards are experienced in the downlink, then
the interference is most likely affecting the subscriber links.
 

Channel Changing

Same Band
When it is deemed during troubleshooting that a channel change is required due to RF
interference, simply change the channel on the Sector Controller to the new frequency
when it’s in the same band as the original frequency. The Subscriber Stations will detect
a loss in carrier on the old frequency and start scanning the frequency band for the
sector controller. Once it detects the Sector Controller it will re-register on the new
frequency.

Changing Bands
When the band of the entire sector is changing, you must edit the frequency list of all
RESS in the deployment so that it can search for the sector within the specified
frequency range. To execute:

1) In the CLI of the RESS enter clear freqlist

2) Enter the frequency index of the new band

freq add 5260.0 5340.0

or
freq add 5480.0 5590.0
freq add 5660.0 5715.0
or
freq add 5735.0 5840.0

3) Save config
4) Conduct channel change

Both processes are dependent on the correct provisioning of the Subscriber Station with
Auto-Scan enabled. Also ensure that the frequency range is configured on the system to
the band the radio is operating in. This will allow the system to find and register to the
SC faster.
 TERAGO NETWORKS – RF TECHNOLOGY

RF Monitor

Redline provides an external RF monitoring tool application which continuously polls RSL
and SNR data from the radio and all subscribers. This tool can be found in the following
directory:

T:\Technology\Radio Documentation and Software\Redline\Redline Utilities\Redline RF

Monitor

Keep in mind that the tool uses Telnet to obtain statistics and requires exclusive access
to the CLI command while running.

1. Begin by launching the application and establishing a connection to the SC by

entering the user/pass IP address a clicking start.

Figure 3- RF Monitor Login

2. Click on the Start/Stop gathering statistics icon and select the subscriber you
wish to log.
 TERAGO NETWORKS – RF TECHNOLOGY

3. The application will begin graphically displaying the RSL and SNR over time. This
can be saved and exported to excel for review.
4. Watch for erratic dips in SNR which may indicate transient interference or
gradual dips or descending RSL which may indicate fading issues.

Figure 4- RF Monitor Logging – RDL3000

 TERAGO NETWORKS – RF TECHNOLOGY

Figure 5- RF Monitoring AN80i

 TERAGO NETWORKS – RF TECHNOLOGY

Detailed Link Summary

Regular Link Summary

NEW Detailed Link Summary showing additional stats not found in the Link
Summary to determine a link’s health and overall sector health

 Lost Wireless packets

 Lost frames
 Lost Link Count
 Distance to SC in km
 Reset stats button

Stats are updated in real time. As you can see ITWL33 is having issues as the Lost
Frames are incrementing only for this customer.
Example:
http://172.17.197.125/adm/dbg.html
 TERAGO NETWORKS – RF TECHNOLOGY

Auto Scan Status

Auto Scan survey can be completed on all RDL-3000 subscribers.

When a subscriber first boots up, it will scan the full frequency range specified in the
frequency list and display any Redline AN80i or RDL-3000 that the subscriber sees.

You can also force autoscan by navigating to Wireless and clicking on Rescan

To display the results, type in the following in the URL:

http://172.x.x.x./adm/ascansts.html

The table will show Frequency, RSSI, SINR and SC ID of the units that the subscriber
sees.

Example:
http://172.17.251.12/adm/ascansts.html

SC ID 379A is DMORESC09
SC ID A3DC is DMOSC08