Fortios Troubleshooting 524
Fortios Troubleshooting 524
Fortios Troubleshooting 524
VERSION 5.2.0
FORTINET DOCUMENT LIBRARY
http://docs.fortinet.com
FORTINET VIDEO GUIDE
http://video.fortinet.com
FORTINET BLOG
https://blog.fortinet.com
CUSTOMER SERVICE & SUPPORT
https://support.fortinet.com
http://cookbook.fortinet.com/how-to-work-with-fortinet-support/
FORTIGATE COOKBOOK
http://cookbook.fortinet.com
FORTINET TRAINING SERVICES
http://www.fortinet.com/training
FORTIGUARD CENTER
http://www.fortiguard.com
FEEDBACK
Email: techdocs@fortinet.com
October-18-15
Change Log 6
Introduction 7
Before you begin 7
How this guide is organized 7
Troubleshooting methodologies 8
Establish a baseline 8
Define the problem 9
Gathering Facts 10
Create a troubleshooting plan 10
Providing Supporting Elements 11
Obtain any required additional equipment 11
Ensure you have administrator level access to required equipment 11
Contact Fortinet customer support for assistance 12
Troubleshooting tools 13
FortiOS diagnostics 13
Check date and time 13
Resource usage 14
Proxy operation 15
Hardware NIC 19
Traffic trace 21
Session table 22
Firewall session setup rate 25
Finding object dependencies 25
Flow trace 26
Packet sniffing and packet capture 30
NPU based interfaces 33
Debug command 33
The execute tac report command 35
Other commands 36
FortiOS ports 37
FortiAnalyzer/FortiManager ports 38
FortiGuard troubleshooting 39
Troubleshooting process for FortiGuard updates 39
FortiGuard server settings 40
Common questions 42
How to check hardware connections 43
How to check FortiOS network settings 44
Interface settings 44
DNS settings 44
DHCP Server settings 45
How to check CPU and memory resources 45
How to troubleshoot high memory usage 46
How to troubleshoot high CPU usage 47
How to check modem status 50
How to run ping and traceroute 50
Ping 51
Traceroute 52
How to check the logs 54
How to verify the contents of the routing table (in NAT mode) 55
How to verify the correct route is being used 56
How to verify the correct firewall policy is being used 57
How to check the bridging information in Transparent mode 57
How to check the bridging information 57
How to display forwarding domain information 57
How to check number of sessions used by UTM proxy 59
Conserve or failopen mode 59
Checking sessions in use 60
Related commands 62
How to examine the firewall session list 63
Check source NAT information 63
How to check wireless information 64
Troubleshooting station connection issue 64
Enable diagnostic for particular station 64
How to verify FortiGuard connectivity 64
How to perform a sniffer trace (CLI and Packet Capture) 65
What can sniffing packets tell you 65
How do you sniff packets 65
Packet Capture 66
How to debug the packet flow 67
Troubleshooting resources 69
Technical Documentation 69
Fortinet Video Library 69
Release Notes 69
Knowledge Base 69
Fortinet Technical Discussion Forums 69
Fortinet Training Services Online Campus 70
Fortinet Customer Support 70
Technical Support Organization Overview 71
Fortinet Global Customer Services Organization 71
Creating an account 72
Registering a device 73
Reporting problems 75
Logging online tickets 75
Following up on online tickets 76
Telephoning a technical support center 76
Assisting technical support 76
Support priority levels 77
Priority 1 77
Priority 2 77
Priority 3 77
Priority 4 77
Return material authorization process 78
Change Log
Change Log
October 18, 2015 Moved Life of a Packet content into a new Life of a Packet handbook chapter.
January 05, 2015 Added "S" flag in FortiGuard server settings in "Troubleshooting tools" on page 40
Troubleshooting 6
Fortinet Technologies Inc.
Introduction Before you begin
Introduction
Welcome and thank you for selecting Fortinet products for your network protection.
This guide is intended for administrators who need guidance on different network needs and information on basic
and advanced troubleshooting.
Before you begin using this guide, take a moment to verify the following:
l Administrators are assumed to be super_admin administrators unless otherwise specified. Some restrictions will
apply to other administrators.
This handbook chapter describes concepts of troubleshooting and solving issues that may occur with FortiGate
units.
Verifying FortiGate admin access security explains how to verify and configure administrative access.
Troubleshooting tools describes some of the basic commands and parts of FortiOS that can help you with
troubleshooting.
Troubleshooting methodologies walks you through best practice concepts of FortiOS troubleshooting.
Technical Support Organization Overview describes how Fortinet Support operates, what they will need from you
if you contact them, and what you can expect in general.
Troubleshooting 7
Fortinet Technologies Inc.
Establish a baseline Troubleshooting methodologies
Troubleshooting methodologies
Before you begin troubleshooting anything but the most minor issues, you need to prepare. Doing so will shorten
the time to solve your issue. This section helps to explain how you prepare before troubleshooting, as well as
creating a troubleshooting plan and contacting support.
l Establish a baseline
l Define the problem
l Gathering Facts
l Create a troubleshooting plan
l Obtain any required additional equipment
l Ensure you have administrator level access to required equipment
l Contact Fortinet customer support for assistance
Establish a baseline
FortiGate units operate at all layers of the OSI model. For this reason troubleshooting problems can become
complex. If you establish a normal operation parameters, or baseline, for your system before the problem occurs
it will help reduce the complexity when you are troubleshooting.
Many of the guiding questions in the following sections are some form of comparing the current problem situation
to normal operation on your FortiGate unit. For this reason it is a best practice that you know what your normal
operating status is, and have a record of it you can refer to. This can easily be accomplished by monitoring the
system performance with logs, SNMP tools, or regularly running information gathering commands and saving the
output. This regular operation data will show trends, and enable you to see when changes happen and there may
be a problem.
Back up your FortiOS configuration on a regular basis. This is a good practice for
everyday as well as when troubleshooting. You can restore the backed up
configuration when needed and save the time and effort of re-creating it from the
factory default settings.
Some fundamental CLI commands you can use to obtain normal operating data for your system:
get system status Displays versions of firmware and FortiGuard engines, and
other system information.
8 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting methodologies Define the problem
get router info routing-table Displays all the routes in the routing table including their type,
all source, and other useful data.
get webfilter ftgd-statistics Displays list of FortiGuard related counts of status, errors, and
other data.
These commands are just a sample. Feel free to include any extra information gathering commands that apply to
your system. For example if you have active VPN connections, record information about them using the get
vpn * series of commands.
For an extensive snapshot of your system, run the CLI command used by TAC to gather extensive information
about a system — exec tac report. It runs many diagnostic commands that are for specific configurations.
This means no matter what features you are using, this command will record their current state. Then if you need
to perform troubleshooting at a later date, you can run the same command again and compare the differences to
quickly locate suspicious output you can investigate.
The following questions can help determine the scope of the problem and isolate it:
Troubleshooting 9
Fortinet Technologies Inc.
Gathering Facts Troubleshooting methodologies
changes were made. The change could be in the operating environment, for example, a gradual increase in load as
more sites are forwarded through the firewall.
If something has changed, see what the affect is if the change is rolled back.
l Determine the scope of the problem - after you have isolated the problem what applications, users,
devices, and operating systems does it effect?
Before you can solve a problem, you need to understand it. Often this step can be the longest in this process.
These questions will help you define the problem. Once the problem is defined, you can search for a solution and
then create a plan on how to solve it.
Gathering Facts
Fact gathering is an important part of defining the problem. Record the following information as it applies to the
problem:
Once you have defined the problem, and searched for a solution you can create a plan to solve that problem.
Even if your search didn’t find a solution to your problem you may have found some additional things to check to
further define your problem.
The plan should list all the possible causes of the problem that you can think of, and how to test for each possible
cause.
10 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting methodologies Obtain any required additional equipment
Your troubleshooting plan will act as a checklist so that you know what you have tried and what is left to check.
This is important to have if more than one person will be doing the troubleshooting. Without a written plan,
people will become easily confused and steps will be skipped. Also if you have to hand over the problem to
someone else, providing them with a detailed list of what data has been gathered and what solutions have been
already tried demonstrates a good level of professionalism.
Be ready to add to your plan as needed. After you are part way through, you may discover that you forgot some
tests or a test you performed discovered new information. This is normal.
Also if you contact support, they will require information about your problem as well as what you have already
tried to fix the problem. This should all be part of your plan.
l The firmware build version (use the get system status command)
l A network topology diagram
l A recent configuration file
l Optionally, a recent debug log
l Tell the support team what troubleshooting steps have already been performed and the results.
Do not provide the output from exec tac report unless Support requests it. The output
from that command is very large and is not required in many cases.
For additional information about contacting Fortinet Customer Support, see Technical Support Organization
Overview.
You may require additional networking equipment, computers, or other equipment to test your solution.
Normally network administrators have additional networking equipment available either to loan you, or a lab
where you can bring the FortiGate unit to test.
If you do not have access to equipment, check for shareware applications that can perform the same task. Often
there are software solutions when hardware is too expensive.
Before troubleshooting your FortiGate unit, you will need administrator access to the equipment. If you are a
client on a FortiGate unit with virtual domains enabled, often you can troubleshoot within your own VDOM.
However, you should inform your FortiGate unit’s super admin that you will be doing troubleshooting.
Troubleshooting 11
Fortinet Technologies Inc.
Contact Fortinet customer support for assistance Troubleshooting methodologies
Also, you may need access to other networking equipment such as switches, routers, and servers to help you test.
If you do not normally have access to this equipment, contact your network administrator for assistance.
You have defined your problem, researched a solution, put together a plan to find the solution, and executed that
plan. At this point if the problem has not been solved, its time to contact Fortinet Customer Support for
assistance.
12 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
Troubleshooting tools
FortiOS provides a number of tools that help with troubleshooting both hardware and software issues. These
tools include diagnostics and ports; ports are used when you need to understand the traffic coming in or going out
on a specific port, for example, UDP 53, which is used by the FortiGate unit for DNS lookup and RBL lookup.
l FortiOS diagnostics
l FortiOS ports
l FortiAnalyzer/FortiManager ports
l FortiGuard troubleshooting
FortiOS diagnostics
A collection of diagnostic commands are available in FortiOS for troubleshooting and performance monitoring.
Within the CLI commands, the two main groups of diagnostic commands are get and diagnose commands.
Both commands display information about system resources, connections, and settings that enable you to locate
and fix problems, or to monitor system performance.
Troubleshooting 13
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
Use Network Time Protocol (NTP) to set the date and time if possible. This is an automatic method that does not
require manual intervention. However, you must ensure the port is allowed through the firewalls on your network.
FortiToken synchronization requires NTP in many situations.
Resource usage
Each program running on a computer has one or more processes associated with it. For example if you open a
Telnet program, it will have an associated telnet process. The same is true in FortiOS. All the processes have to
share the system resources in FortiOS including memory and CPU.
Use get system performance status command to show the FortiOS performance status.
Sample output:
FGT#get system performance status
CPU states: 0% user 0% system 0% nice 100% idle
CPU0 states: 0% user 0% system 0% nice 100% idle
CPU1 states: 0% user 0% system 0% nice 100% idle
CPU2 states: 0% user 0% system 0% nice 100% idle
CPU3 states: 0% user 0% system 0% nice 100% idle
Memory states: 25% used
Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes, 0 kbps in 30 minutes
Average sessions: 5 sessions in 1 minute, 5 sessions in 10 minutes, 4 sessions in 30
minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second
in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 12 hours, 7 minutes
14 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
Monitor the CPU/memory usage of internal processes using the following command:
get system performance top <delay> <max_lines>
The data listed by the command includes the name of the daemon, the process ID, whether the process is
sleeping or running, the CPU percentage being used, and the memory percentage being used.
Sample output:
Proxy operation
Monitor proxy operations using the following command:
diag test application <application> <option>
The <application> value can include the following:
Troubleshooting 15
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
16 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
The <option> value depends from the application value used in the command. Here are some examples:
Troubleshooting 17
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
18 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
15 IPSA statistics
Hardware NIC
Monitor hardware network operations using the following command:
diag hardware deviceinfo nic <interface>
The information displayed by this command is important as errors at the interface are indicative of data link or
physical layer issues which may impact the performance of the FortiGate unit.
Troubleshooting 19
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
The diag hardware deviceinfo nic command displays a list of hardware related error names and
values. The following table explains the items in the list and their meanings.
Field Definition
Rx_CRC_Errors +
Rx_Length_Errors - This error is only valid in 10/100M mode.
Rx_Align_Errors
Tx_Errors = Tx_Aborted_ ECOL (Excessive Collisions Count). Only valid in half-duplex mode.
Errors
LATECOL (Late Collisions Count). Late collisions are collisions that occur
after 64-byte time into the transmission of the packet while working in 10
Tx_Window_Errors to100Mb/s data rate and 512-byte timeinto the transmission of the packet
while working in the 1000Mb/s data rate. This register only increments if
transmits are enabled and the device is in half-duplex mode.
20 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
Field Definition
The PHY should assert the internal carrier sense signal during every
transmission. Failure to do so may indicate that the link has failed or the
PHY has an incorrect link configuration. This register only increments if
Tx_Carrier_Errors
transmits are enabled. This register is not valid in internal SerDes 1 mode
(TBI mode for the 82544GC/EI) and is only valid when the Ethernet
controller is operating at full duplex.
Tx_Multiple_Collision_Frames A Multiple Collision Count which counts the number of times that a
transmit encountered more than one collision but less than 16. The value
only increments if transmits are enabled and the Ethernet controller is in
half-duplex mode.
Counts defer events. A defer event occurs when the transmitter cannot
immediately send a packet due to the medium being busy because another
device is transmitting, the IPG timer has not expired, half-duplex deferral
Tx_Deferred
events are occurring, XOFF frames are being received, or the link is not up.
This register only increments if transmits are enabled. This counter does
not increment for streaming transmits that are deferred due to TX IPG.
Traffic trace
Traffic tracing allows a specific packet stream to be followed. This is useful to confirm packets are taking the
route you expected on your network.
View the characteristics of a traffic session though specific security policies using:
diag sys session
Troubleshooting 21
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
Session table
A session is a communication channel between two devices or applications across the network. Sessions enable
FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each
packet individually. Each of these sessions has an entry in the session table that includes important information
about the session.
Use as a tool
Session tables are useful troubleshooting tools because they allow you to verify connections that you expect to
see open. For example, if you have a web browser open to browse the Fortinet website, you would expect a
session entry from your computer, on port 80, to the IP for the Fortinet website. Another troubleshooting method
is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this
is happening.
The FortiGate session table can be viewed from either the CLI or the web-based manager. The most useful
troubleshooting data comes from the CLI. The session table in web-based manager also provides some useful
summary information, particularly the current policy number that the session is using.
You may want to find information for a specific session, say a secure web browser session, for troubleshooting.
For example if that web browser session is not working properly, you can check the session table to ensure the
session is still active, and that it is going to the proper address. It can also tell you the security policy number it
matches, so you can check what is happening in that policy.
22 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
destination IP address if you have it, and port number. The policy ID is listed after the destination
information. If the list of sessions is very long, you can filter the list to make it easier to find your
session.
3. When there are many sessions, use a filter to help you find your session.
If there are multiple pages of sessions it is difficult to find a single session. To help you in your search
you can use a filter to block out sessions that you don’t want. Click the search icon on the column
heading to select the filter. Select Source IP and enter your source IP address. Now only sessions
that originate from your IP address will be displayed in the session table. If the list is still too long, you
can do the same for the Source port. That will make it easy to find your session and the security
policy ID. When you are finished remember to clear the filters.
You can filter a column in the web-based manager by clicking the search icon on the column heading or from the
CLI by creating a filter.
An entry is placed in the session table for each traffic session passing through a security policy. The following
command will list the information for a session in the table:
diag sys session list
Sample Output:
FGT# diag sys session list
session info: proto=6 proto_state=05 expire=89 timeout=3600 flags=00000000 av_idx=0 use=3
bandwidth=204800/sec guaranteed_bandwidth=102400/sec traffic=332/sec prio=0
logtype=session ha_id=0 hakey=4450
tunnel=/
state=log shape may_dirty
statistic(bytes/packets/err): org=3408/38/0 reply=3888/31/0 tuples=2
orgin->sink: org pre->post, reply pre->post oif=3/5 gwy=192.168.11.254/10.0.5.100
hook=post dir=org act=snat 10.0.5.100:1251->192.168.11.254:22(192.168.11.105:1251)
hook=pre dir=reply act=dnat 192.168.11.254:22->192.168.11.105:1251(10.0.5.100:1251)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 domain_info=0 auth_info=0 ftgd_info=0 ids=0x0 vd=0 serial=00007c33 tos=ff/ff
Since output can be verbose, the filter option allows specific information to be displayed, for example:
diag sys session filter <option>
Troubleshooting 23
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
duration duration
expire expire
Even though UDP is a sessionless protocol, the FortiGate unit still keeps track of the following two different
states:
State Meaning
Session is created by a policy. For example, the session for ftp control
may_dirty channel will have this state but ftp data channel will not. This is also seen
when NAT is enabled.
24 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
A reduced firewall session setup rate could be the result of a number of things from a lack of system resources on
the FortiGate unit, to reaching the limit of your session count for your VDOM.
The information you are looking for is the Average sessions section, highlighted in the above output. In this
example you can see there were 31 sessions in 1 minute, or an average of 0.5 sessions per second. The values
for 10 minutes and 30 minutes allow you to take a longer average for a more reliable value if your FortiGate unit is
working at maximum capacity. The smallest FortiGate unit can have 1 000 sessions established per second
across the unit.
Remember that session setup rate is a global command. If you have multiple VDOMs configured with many
sessions in each one, the session setup rate per VDOM will be slower than if there were no VDOMs configured.
Another use is if you have a virtual interface with objects that depend on it, you need to find and remove those
dependencies before you delete that interface.
CLI method
When running multiple VDOMs, this command is run in the Global configuration only and it searches for the
named object both in the Global and VDOM configuration most recently used:
diag sys checkused <path.object.mkey>
Troubleshooting 25
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
For example, to verify which objects are referred to in a security policy with an ID of 1, enter the command as
follows:
diag sys checkused firewall.policy.policyid 1
To show all the dependencies for an interface, enter the command as follows:
diag sys checkused system.interface.name <interface name>
Sample Output:
entry used by table firewall.address:name '10.98.23.23_host’
entry used by table firewall.address:name 'NAS'
entry used by table firewall.address:name 'all'
entry used by table firewall.address:name 'fortinet.com'
entry used by table firewall.vip:name 'TORRENT_10.0.0.70:6883'
entry used by table firewall.policy:policyid '21'
entry used by table firewall.policy:policyid '14'
entry used by table firewall.policy:policyid '19'
In this example, the interface has dependent objects, including four address objects, one VIP, and three security
policies.
2. Select the number in the Ref. column for the desired interface.
A Window listing the dependencies will appear.
3. Use these detailed entries to locate and remove object references to this interface.
The trash can icon will change from gray when all object dependencies have been removed.
4. Remove the interface by selecting the check box for the interface, and select Delete.
Flow trace
To trace the flow of packets through the FortiGate unit, use the following command:
diag debug flow trace start
If your network is using IPv4, follow packet flow by setting a flow filter using this command:
diag debug flow filter <option>
26 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
If your network is using IPv6, follow packet flow by setting a flow filter using this command:
diag debug flow filter6 <option>
Enable the output to be displayed to the CLI console using the following command:
diag debug flow show console enable
diag debug flow output is recorded as event log messages and are sent to a
FortiCloud or a FortiAnalyzer unit if connected. Do not let this command run longer
than necessary since it generates significant amounts of data.
Start flow monitoring with a specific number of packets using this command:
diag debug flow trace start <N>
The following is an example of the flow trace for the device at the following IP address: 203.160.224.97
diag debug enable
diag debug flow filter addr 203.160.224.97
diag debug flow show console enable
diag debug flow show function-name enable
diag debug flow trace start 100
Troubleshooting 27
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
Matched security policy. Check to see which policy this session matches:
id=20085 trace_id=209 func=fw_forward_handler line=317
msg="Allowed by Policy-3: SNAT"
ACK received:
id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700
msg="vd-root received a packet(proto=6,
192.168.3.221:1487->203.160.224.97:80) from port5."
28 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
direction"
Troubleshooting 29
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
Packet sniffing in the CLI is well suited for spot checking traffic from the CLI, but if you have complex filters to
enter it can be a lot of work to enter them each time. You can also save the sniffing output; however, you must log
to a file and then analyze the file later by hand.
Packet capture in the web-based manager makes it easy to set up multiple filters at once and just run one or two
as you need them. You also have controls to start and stop capturing as you wish. Packet capture output is
downloaded to your local computer as a *.pcap file which requires a third party application to read the file, such as
Wireshark. This method is useful to send Fortinet support information to help resolve an issue.
Third party software required puTTY to log plaintext output Wireshark to read *.pcap files
Packet sniffing
Before you start sniffing packets on the CLI, you should be prepared to capture the output to a file — there can be
huge amounts of data that you will not be able to see without saving it to a file. One method is to use a terminal
program like puTTY to connect to the FortiGate unit’s CLI. Then once the packet sniffing count is reached you
can end the session and analyze the output in the file.
Details within packets passing through particular interfaces can be displayed using the packet sniffer with the
following command:
30 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
The <interface> value is required, with the rest being optional. If not included the default values will be
“none”.
The <interface> value can be any physical or virtual interface name. Use any to sniff packets on all
interfaces.
The <filter> value limits the display of packets using filters, including Berkeley Packet Filtering (BPF) syntax.
The <filter> value must be enclosed in quotes.
'[[src|dst] host <host_name_or_IP1>] [[src|dst] host <host_name_or_IP2>]
[[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]]‘
If a second host is specified in the filter, only the traffic between the two hosts will be displayed. Optionally, you
can use logical OR to match only one of the hosts, or match one of multiple protocols or ports. When defining a
port, there are up to two parts — protocol and port number.
For example, to display UDP 1812 traffic or TCP 8080 traffic, use the following:
'udp port 1812 or tcp port 8080’
To display all IP traffic that has a source of 192.168.1.2 and a destination of 192.168.2.3:
'ip src host 192.168.1.2 and dst host 192.168.2.3’
The <verbose> option allows different levels of information to be displayed. The verbose levels include:
Packet capture
FortiOS 5.2 includes packet capture to the web-based manager. The FortiGate unit must have a disk and then
capture-packet feature can be enabled from the CLI within the firewall policy as below:
config firewall policy
Troubleshooting 31
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
edit <id>
set capture-packet enable
end
To configure packet capture filters, go to System > Network > Packet Capture.
When you add a packet capture filter, enter the following information and select OK.
You must select one interface. You cannot change the interface without
deleting the filter and creating a new one, unlike the other fields.
Separate multiple ports with commas. Enter a range using a dash without
spaces, for example 88-90
Protocol Enter one or more protocol. Separate multiple protocol with commas. Enter
a range using a dash without spaces, for example 1-6, 17, 21-25
Capture Non-IP packets The protocols available in the list are all IP based except for ICMP (ping).
To capture non-IP based packets select this feature. Some examples of
non-IP packets include IPsec, IGMP, ARP, and as mentioned ICMP.
If you select a filter and go back to edit it, you have the added option of starting and stopping packet capture in
the edit window, or downloading the captured packets. You can also see the filter status and the number of
packets captured.
You can also select the filter and select Start to start capturing packets. While the filter is running, you will see
the number of captured packets increasing until it reaches the max packet count or you select Stop. While the
filter is running you cannot download the output file.
32 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
When the packet capture is complete, you can select Download to send the packet capture filter captured
packets to your local computer as a *.pcap file. To read this file format, you will need to use Wireshark or a
similar third party application. Using this tool you will have extensive analytics available to you and the full
contents of the packets that were captured.
When using the NPU-based interfaces, only the initial session setup will be seen through the diag debug
flow command. If the session is correctly programmed into the ASIC (fastpath), the debug flow command will
no longer see the packets arriving at the CPU. If the NPU functionality is disabled, the CPU will see all the
packets, however, this should only be used for troubleshooting purposes.
First, obtain the NP4 id and the port numbers with the following command:
diag npu np4 list
Sample output:
ID Model Slot Interface
0 On-board port1 fabric1 fabric3 fabric5
1 On-board fabric2 port2 base2 fabric4
Sample output:
NP4 Fast Path Sniffer on port1 enabled
This will cause all traffic on port1 of NP4 to be sent to the CPU meaning a standard sniffer trace can be taken and
other diag commands should work if it was a standard CPU driven port.
Debug command
Debug output provides continuous, real-time event information. Debugging output continues until it is explicitly
stopped or until the unit is rebooted. Debugging output can affect system performance and will be continually
generated even though output might not be displayed in the CLI console.
Debug information displayed in the console will scroll in the console display and may prevent CLI commands from
being entered, for example, the command to disable the debug display. To turn off debugging output as the
display is scrolling by, press the á key to recall the recent diag debug command, press backspace, and type “0”,
followed by Enter.
Troubleshooting 33
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
Once enabled, indicate the debug information that is required using this command:
diag debug <option> <level>
application application.
kernel kernel.
remote-extender remote-extender.
console console.
urlfilter urlfilter.
The debug level can be set at the end of the command. Typical values are 2 and 3, for example:
34 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS diagnostics
Timestamps can be enabled to the debug output using the following command:
diag debug console timestamp enable
Sample Output:
FGh_FtiLog1: IPsec SA connect 0 192.168.11.2->192.168.10.201:500, natt_mode=0 rekey=0
phase2=FGh_FtiLog1
FGh_FtiLog1: using existing connection, dpd_fail=0
FGh_FtiLog1: found phase2 FGh_FtiLog1
FGh_FtiLog1: IPsec SA connect 0 192.168.11.2 -> 192.168.10.201:500 negotiating
FGh_FtiLog1: overriding selector 225.30.5.8 with 192.168.11.2
FGh_FtiLog1: initiator quick-mode set pfs=1536...
FGh_FtiLog1: try to negotiate with 1800 life seconds.
FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201,
ports=0/0, protocol=0/0
Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348
Initiator: sent 192.168.10.201 quick mode message #1 (OK)
FGh_FtiLog1: set retransmit: st=168, timeout=6.
In this example:
When you call Fortinet Customer Support, you will be asked to provide information about your unit and its current
state using the output from this CLI command.
Troubleshooting 35
Fortinet Technologies Inc.
FortiOS diagnostics Troubleshooting tools
Other commands
ARP table
To view the ARP cache, use the following command:
get sys arp
Sample output:
index=14 ifname=internal 224.0.0.5 01:00:5e:00:00:05 state=00000040 use=72203
confirm=78203 update=72203 ref=1
index=13 ifname=dmz 192.168.3.100 state=00000020 use=1843 confirm=650179 update=644179
ref=2 ? VIP
index=13 ifname=dmz 192.168.3.109 02:09:0f:78:69:ff state=00000004 use=71743 confirm=75743
update=75743 ref=1
index=14 ifname=internal 192.168.11.56 00:1c:23:10:f8:20 state=00000004 use=10532
confirm=10532 update=12658 ref=4
If all devices have the same time, it helps to correlate log entries from different devices.
36 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiOS ports
IP address
There may be times when you want to verify the IP addresses assigned to the FortiGate unit interfaces are what
you expect them to be. This is easily accomplished from the CLI using the following command.
diag ip address list
The output from this command lists the IP address and mask if available, the index of the interface (a sort of ID
number) and the devname is the name of the interface. While physical interface names are set, virtual interface
names can vary. Listing all the virtual interface names is a good use of this command. For vsys_ha and vsys_
fgfm, the IP addresses are the local host — these are internally used virtual interfaces.
# diag ip address list
IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal
IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha
IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm
FortiOS ports
In the TCP and UDP stacks, there are 65 535 ports available for applications to use when communicating with
each other. Many of these ports are commonly known to be associated with specific applications or protocols.
These known ports can be useful when troubleshooting your network.
Port(s) Functionality
UDP 53 (default) or UDP 8888 FDN Server List - source and destination port numbers vary by originating
and UDP 1027 or UDP 1031 or reply traffic. See the article “How do I troubleshoot performance issues
when FortiGuard Web Filtering is enabled?” in the Knowledge Base.
SYSLOG - All FortiOS versions can use syslog to send log messages to
UDP 514 remote syslog servers. FortiOS v2.80 and v3.0 can also view logs stored
remotely on a FortiAnalyzer unit.
Troubleshooting 37
Fortinet Technologies Inc.
FortiAnalyzer/FortiManager ports Troubleshooting tools
Port(s) Functionality
TCP 514 FortiGuard Analysis and Management Service log transmission (OFTP)
TCP 541 SSL Management Tunnel to FortiGuard Analysis and Management Service
(FortiOS v3.0 MR6 or later)
FortiAnalyzer/FortiManager ports
If you have a FortiAnalyzer unit or FortiManager unit on your network you may need to use the following ports for
troubleshooting network traffic.
Functionality Port(s)
38 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiGuard troubleshooting
Functionality Port(s)
FortiGuard troubleshooting
The FortiGuard service provides updates to Antivirus, Antispam, IPS, Webfiltering, and more. The FortiGuard
Distribution System (FDS) involves a number of servers across the world that provide updates to your FortiGate
unit. Problems can occur both with connection to FDS, and its configuration on your local FortiGate unit. Some of
the more common troubleshooting methods are listed here including
1. Does the device have a valid licence that includes these services?
Each device requires a valid FortiGuard license to access updates for some or all of these services. You can verify
the support contract status for your devices at the Fortinet Support website — https://support.fortinet.com/.
2. If the device is part of an HA cluster, do all members of the cluster have the same level of support?
As with the previous step, you can verify the support contract status for all the devices in your HA cluster at the
Fortinet Support website.
3. Have services been enabled on the device?
To see the FortiGuard information and status for a device, in the web-based manager go to System > Config >
FortiGuard. On that page you can verify the status of each component, and if required enable each service. If
there are problems, see the FortiGuard section of the FortiOS Handbook.
4. Is the device able to communicate with FortiGuard servers?
At System > Config > FortiGuard you can also attempt to update AV and IPS, or test the availability of WF and
AS default and alternate ports. If there are problems, see the FortiGuard section of the FortiOS Handbook.
5. Is there proper routing to reach the FortiGuard servers?
Ensure there is a static or dynamic route that enables your ForitGate unit to reach the FortiGuard servers. Usually
a generic default route to the internet is enough, but you may need to verify this if your network is complex.
6. Are there issues with DNS?
An easy way to test this is to attempt a traceroute from behind the FortiGate unit to an external network using the
FQDN for a location. If the traceroute FQDN name does not resolve, you have general DNS problems.
7. Is there anything upstream that might be blocking FortiGuard traffic, either on the network or ISP
side?
Many firewalls block all ports by default, and often ISPs block ports that are low. There may be a firewall between
the FortiGate unit and the FortiGuard servers that is blocking the traffic. FortiGuard uses port 53 by default, so if it
is being blocked you need to either open a hole for it, or change the port it is using.
Troubleshooting 39
Fortinet Technologies Inc.
FortiGuard troubleshooting Troubleshooting tools
You can optionally add a refresh rate to the end of this command and that will determine how often the server list
will be refreshed.
Sample Output:
Locale : english
License : Contract
Expiration : Thu Oct 9 02:00:00 2011
-=- Server List (Mon Feb 18 12:55:48 2008) -=-
IP Weight RTT Flags TZ Packets CurrLost TotalLost
Output Details
The Server List includes the IP addresses of alternate servers if the first entry cannot be reached. In this example
the IP addresses are not public addresses
The following flags in get webfilter status indicate the server status:
40 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting tools FortiGuard troubleshooting
l D - the server was found through the DNS lookup of the hostname. If the hostname returns more than one IP
address, all of them will be flagged with D and will be used first for INIT requests before falling back to the other
servers.
l I - the server to which the last INIT request was sent.
l F - the server has not responded to requests and is considered to have failed.
l T - the server is currently being timed.
l S - means that rating requests can be sent to the server. The flag is set for a server only in two cases:
1. The server exists in the servers list received from the Fortimanager or any other INIT server.
2. The servers list received from the FortiManager is empty so the Fortimanager itself would be the only
server known by Fortigate thus it should be used as the rating server.
The servers that are not currently serving will be pushed down to the bottom list (under the available
serving servers, and on top of the failed servers) in order for the load-balance-servers feature in
the config system fortiguard to work properly.
Calculating weight
The weight for each server increases with failed packets and decreases with successful packets. To lower the
possibility of using a remote server, the weight is not allowed to dip below a base weight, calculated as the
difference in hours between the FortiGate unit and the server times 10. The further away the server is, the higher
its base weight and the lower in the list it will appear.
Troubleshooting 41
Fortinet Technologies Inc.
FortiGuard troubleshooting Common questions
Common questions
The general troubleshooting tips include, and can help answer, the following questions:
42 Troubleshooting
Fortinet Technologies Inc.
Common questions How to check hardware connections
If there is no traffic flowing from the FortiGate unit, it may be a hardware problem.
l Ensure the network cables are properly plugged into the interfaces.
l Ensure there are connection lights for the network cables on the unit.
l Change the cable if the cable or its connector are damaged or you are unsure about the cable’s type or quality—
such as straight through or crossover, or possibly exposed wires at the connector.
l Connect the FortiGate unit to different hardware.
l Ensure the link status is set to Up for the interface, (see Network > Interface > Status). The link status is based
on the physical connection and cannot be set in FortiOS.
If any of these solve the problem, it was a hardware connection problem. You should still perform some basic
software connectivity tests to ensure complete connectivity. It might also be that the interface is disabled, or has
its Administrative Status set to Down.
1. Using the web-based management interface, go to System > Network > Interface.
2. Select and edit the interface to enable, such as port1.
3. Find Administrative Status at the bottom of the screen, and select Up.
4. Select Apply.
Troubleshooting 43
Fortinet Technologies Inc.
How to check FortiOS network settings Common questions
FortiOS network settings are present in both the web-based manager interface and the CLI. The following
information includes troubleshooting and best practice information. The network settings include:
Interface settings
DNS settings
DHCP Server settings
Interface settings
If you can access the FortiGate unit with the management cable only, the first step is to display the interface
settings. To display the settings for the internal interface, use the following CLI command:
FGT# show system interface <Interface_mane>
For a complete listing of all the possible interface settings, use the following CLI command:
config system interface
edit <Interface_name>
get
end
Check the interface settings to ensure they are not preventing traffic. Specific things to check include (only the
web-based manager names are shown, CLI names may vary slightly):
l Link Status — Down until a valid cable is plugged into this interface, after which it will be Up. The Link Status is
shown physically by the connection LED for the interface. If it lights up green, it is a good connection. If Link Status
is Down, the interface does not work. Link Status is also displayed on the System > Network > Interface screen
by default.
l Addressing mode — Do not use DHCP if you don’t have a DHCP server —you will not be able to logon to an
interface in DHCP mode as it will not have an IP address.
l IP/Netmask — An interface needs an IP address to be able to connect to other devices. Ensure there is a valid IP
address in this field. The one exception is if DHCP is enabled for this interface to get its IP address from an
external DHCP server.
l IPv6 address — The same protocol must be used by both ends to complete the connection. Ensure both this
interface and the remote connection are both using IPv4 or both using IPv6 addressing.
l Administrative access — If no protocols are selected, you will have to use the local management cable to
connect to the unit. If you are using IPv6, configure the IPv6 administrative access protocols.
l Administrative status — Set to Up or the interface will not work.
DNS settings
While this section is not complicated, many networking problems can be traced back to DNS problems. Things to
check in this area include:
44 Troubleshooting
Fortinet Technologies Inc.
Common questions How to check CPU and memory resources
l Are you using Dynamic DNS (DDNS)? If so, is it using the correct server, credentials, and interface?
l Can you contact both DNS servers to verify the servers are operational?
l If an interface addressing mode is set to DHCP and is set to override the internal DNS, is that interface receiving a
valid DNS entry from the DHCP server? Is it a reasonable address and can it be contacted to verify it’s operational?
l Are there any DENY security policies that need to allow DNS?
l Can any internal device perform a successful traceroute to a location using the FQDN?
l Is the DHCP server entry set to Relay? If so, verify there is another DHCP server to which requests can be relayed.
Otherwise, it should be set to Server.
l Is the DHCP server enabled?
l Does this DHCP server use a valid range of IP addresses? Are those addresses in use by other devices? If one or
more devices are using IP addresses in this range, you can use the IP reservation feature to ensure the DHCP
server does not use these addresses.
l Is there a gateway entry? Include a gateway entry to ensure clients of this server have a default route.
l Is the system DNS setting being used? The best practice is to avoid confusion by using the system DNS whenever
possible. However, the option to specify up to three custom DNS servers is available, and all three entries should be
used for redundancy.
There are some situations, such as a new wireless interface, or during the initial
FortiGate unit configuration, where interfaces override the system DNS entries. When
this happens, it often shows up as intermittent Internet connectivity. To fix the
problem, go to System > Network > DNS and ensure to enable Use FortiGuard
Servers.
System resources are shared and a number of processes run simultaneously on the FortiGate unit. If one of these
processes consumes nearly all the resources.
A quick way to monitor CPU and memory usage is on the System Dashboard using the System Resources
widgets. They have both a visual gauge displayed to show you the usage.
To check the system resources on your FortiGate unit, run the following CLI command:
FGT# get system performance status
The first line of output shows the CPU usage by category. A FortiGate that is doing nothing will look like:
CPU states: 0% user 0% system 0% nice 100% idle
However, if your network is running slow you might see something like:
CPU states: 1% user 98% system 0% nice 1% idle
Troubleshooting 45
Fortinet Technologies Inc.
How to check CPU and memory resources Common questions
This line shows that all the CPU is used up by system processes. Normally this should not happen as it shows the
FortiGate is overloaded for some reason. If you see this overloading, you should investigate farther as it’s
possible a process, such as scanunitid, is using all the resources to scan traffic, in which case you need to reduce
the amount of traffic being scanned by blocking unwanted protocols, configuring more security policies to limit
scanning to certain protocols, or similar actions. It is also possible that a hacker has gained access to your
network and is overloading it with malicious activity such as running a spam server or using zombie PCs to attack
other networks on the Internet. You can get additional CPU related information with the CLI command get
system performance top. This command shows you all the top processes running on the FortiGate unit
(names on the left) and their CPU usage. If a process is using most of the CPU cycles, investigate it to determine
if it’s normal activity.
The second line of output from get system performance status shows the memory usage. Memory
usage should not exceed 90 percent. If memory is too full, some processes will not be able to function properly.
For example, if the system is running low on memory, antivirus scanning will go into failopen mode where it will
start dropping connections or bypass the antivirus system.
The other lines of output, such as average network usage, average session setup rate, viruses caught, and IPS
attacks blocked can also help you determine why system resource usage it high. For example, if network usage is
high it will result in high traffic processing on the FortiGate, or if the session setup rate is very low or zero the
proxy may be overloaded and not able to do its job.
When high memory usage happens, you may experience services that appear to freeze up and connections are
lost or new connections are refused.
If you are seeing high memory usage in the System Resources widget, it could mean that the unit is dealing
with high traffic volume, which may be causing the problem, or it could be when the unit is dealing with
connection pool limits affecting a single proxy. If the unit is receiving large volumes of traffic on a specific proxy, it
is possible that the unit will exceed the connection pool limit. If the number of free connections within a proxy
connection pool reaches zero, problems may occur.
Use the following CLI command, which uses the antivirus failopen feature. Setting it to idledrop will drop
connections based on the clients that have the most connections open. This helps to determine the behavior of
the FortiGate antivirus system if it becomes overloaded in high traffic.
config system global
set av-failopen idledrop
end
Use the following CLI command, which gives you information about current memory usage:
diagnose hardware sysinfo memory
46 Troubleshooting
Fortinet Technologies Inc.
Common questions How to check CPU and memory resources
Sample output:
total: used: free: shared: buffers: cached: shm:
Mem: 2074185728 756936704 1317249024 0 20701184 194555904 161046528
Swap: 0 0 0
MemTotal: 2025572 kB
MemFree: 1286376 kB
MemShared: 0 kB
Buffers: 20216 kB
Cached: 189996 kB
SwapCached: 0 kB
Active: 56644 kB
Inactive: 153648 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 2025572 kB
LowFree: 1286376 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Some examples of features that are CPU intensive are VPN high level encryption, having all traffic undergo all
possible scanning, logging all traffic, and packets, and dashboard widgets that frequently update their data.
1. Determine how high the CPU usage is currently.There are two main ways to do this. The easiest is to go to
System > Dashboard > Status and look at the system resources widget. This is a dial gauge that displays a
percentage use for the CPU. If its at the red-line, you should take action. The other method is to use the
Dashboard CLI widget to enter diag sys top.
Sample output:
Run Time: 11 days, 23 hours and 36 minutes
0U, 0S, 98I; 1977T, 758F, 180KF
newcli 286 R 0.1 0.8
ipsengine 78 S < 0.0 3.1
ipsengine 64 S < 0.0 3.0
ipsengine 77 S < 0.0 3.0
ipsengine 68 S < 0.0 2.9
ipsengine 66 S < 0.0 2.9
ipsengine 79 S < 0.0 2.9
scanunitd 133 S < 0.0 1.8
pyfcgid 267 S 0.0 1.8
pyfcgid 269 S 0.0 1.7
pyfcgid 268 S 0.0 1.6
httpsd 139 S 0.0 1.6
pyfcgid 266 S 0.0 1.5
scanunitd 131 S < 0.0 1.4
scanunitd 132 S < 0.0 1.4
proxyworker 90 S 0.0 1.3
cmdbsvr 43 S 0.0 1.1
proxyworker 91 S 0.0 1.1
Troubleshooting 47
Fortinet Technologies Inc.
How to check CPU and memory resources Common questions
Where the codes displayed on the second output line mean the following:
l U is % of user space applications using CPU. In the example, 0U means 0% of the user space applications are
using CPU.
l S is % of system processes (or kernel processes) using CPU. In the example, 0S means 0% of the system
processes are using the CPU.
l I is % of idle CPU. In the example, 98I means the CPU is 98% idle.
l T is the total FortiOS system memory in Mb. In the example, 1977T means there are 1977 Mb of system
memory.
l F is free memory in Mb. In the example, 758F means there is 758 Mb of free memory.
l KF is the total shared memory pages used. In the example, 180KF means the system is using 180 shared
memory pages.
Each additional line of the command output displays information for each of the processes running on
the FortiGate unit. For example, the third line of the output is:
newcli 286 R 0.1 0.8
Where:
l newcli is the process name. Other process names can include ipsengine, sshd, cmdbsrv, httpsd,
scanunitd, and miglogd.
l 286 is the process ID. The process ID can be any number.
l R is the current state of the process. The process state can be:
l R running
l S sleep
l Z zombie
l D disk sleep.
l 0.1 is the amount of CPU that the process is using. CPU usage can range from 0.0 for a process that is
sleeping to higher values for a process that is taking a lot of CPU time.
l 0.8 is the amount of memory that the process is using. Memory usage can range from 0.1 to 5.5 and higher.
Enter the following single-key commands when diagnose sys top is running:
48 Troubleshooting
Fortinet Technologies Inc.
Common questions How to check CPU and memory resources
l iked — internet key exchange (IKE) in use with IPsec VPN tunnels
l newcli — active whenever you are accessing the CLI
l sshd — there are active secure socket connections
l cmdbsrv — the command database server application
Go to the features that are at the top of the list and look for evidence of them overusing the CPU.
Generally the monitor for a feature is a good place to start.
4. When CPU usage is under control, use SNMP to monitor CPU usage. Alternately, use logging to
record CPU and memory usage every 5 minutes.
Once things are back to normal, you should set up a warning system to alert you of future CPU overusage. A
common method to do this is with SNMP. SNMP monitors many values on the FortiOS and allows you to set high
water marks that will generate events. You run an application on your computer to watch for and record these
events. Go to System > Config > SNMP to enable and configure an SNMP community. If this method is too
Troubleshooting 49
Fortinet Technologies Inc.
How to check modem status Common questions
complicated, you can use the System Resources widget to record CPU usage. However, this method will not
alert you to problems - it will just record them as they happen.
Sometimes the modem may not work properly, or the unit may not be detecting the modem. Use the following
diagnostic commands to help you troubleshoot issues with the modem.
diagnose sys modem {cmd | com | detect | history | external-modem| query| reset}
You should always run the following diagnose command after inserting the USB modem into the unit:
diagnose sys modem detect
You can view the modem configuration by using the get system modem command. You can also view the
modem’s vendor identification as well as the custom product identification number from the information output
from the get system modem command.
When there are connectivity issues, use the following to help you resolve them:
Ping and traceroute are useful tools in network troubleshooting. Alone, either one can determine network
connectivity between two points. However, ping can be used to generate simple network traffic to view with
diagnose commands on the FortiGate unit. This combination can be very powerful when locating network
problems.
In addition to their normal uses, ping and traceroute can tell you if your computer or network device has access to
a domain name server (DNS). While both tools can use IP addresses alone, they can also use domain names for
devices. This is an added troubleshooting feature that can be useful in determining why particular services, such
as email or web browsing, may not be working properly.
If ping does not work, you likely have it disabled on at least one of the interface
settings, and security policies for that interface.
Both ping and traceroute require particular ports to be open on firewalls, or else they cannot function. Since you
typically use these tools to troubleshoot, you can allow them in the security policies and on interfaces only when
you need them, and otherwise keep the ports disabled for added security.
50 Troubleshooting
Fortinet Technologies Inc.
Common questions How to run ping and traceroute
Ping
The ping command sends a very small packet to the destination, and waits for a response. The response has a
timer that may expire, indicating the destination is unreachable. The behavior of ping is very much like a sonar
ping from a submarine, where the command gets its name.
Ping is part of Layer-3 on the OSI Networking Model. Ping sends Internet Control Message Protocol (ICMP) “echo
request” packets to the destination, and listens for “echo response” packets in reply. However, many public
networks block ICMP packets because ping can be used in a denial of service (DoS) attack (such as Ping of Death
or a smurf attack), or by an attacker to find active locations on the network. By default, FortiGate units have ping
enabled while broadcast-forward is disabled on the external interface.
Beyond the basic connectivity information, ping can tell you the amount of packet loss (if any), how long it takes
the packet to make the round trip, and the variation in that time from packet to packet.
If there is some packet loss detected, you should investigate the following:
l Hardware — ensure cabling is correct, and all equipment between the two locations is accounted for.
l Addresses and routes — ensure all IP addresses and routing information along the route is configured as
expected.
l Firewalls — ensure all firewalls, including FortiGate unit security policies allow PING to pass through.
Ping syntax is the same for nearly every type of system on a network.
1. Connect to the CLI either through telnet or through the CLI widget on the web-based manager dashboard.
2. Enter exec ping 10.11.101.101 to send 5 ping packets to the destination IP address. There are no options
for this command.
Sample output:
Head_Office_620b # exec ping 10.11.101.101
PING 10.11.101.101 (10.11.101.101): 56 data bytes
64 bytes from 10.11.101.101: icmp_seq=0 ttl=255 time=0.3 ms
64 bytes from 10.11.101.101: icmp_seq=1 ttl=255 time=0.2 ms
64 bytes from 10.11.101.101: icmp_seq=2 ttl=255 time=0.2 ms
64 bytes from 10.11.101.101: icmp_seq=3 ttl=255 time=0.2 ms
64 bytes from 10.11.101.101: icmp_seq=4 ttl=255 time=0.2 ms
Troubleshooting 51
Fortinet Technologies Inc.
How to run ping and traceroute Common questions
1. Go to a shell prompt.
2. Enter “ping 10.11.101.101”.
Traceroute
Where ping will only tell you if it reached its destination and came back successfully, traceroute will show each
step of its journey to its destination and how long each step takes. If ping finds an outage between two points,
traceroute can be used to locate exactly where the problem is.
What is traceroute
Traceroute works by sending ICMP packets to test each hop along the route. It will send out three packets, and
then increase the time to live (TTL) setting by one each time. This effectively allows the packets to go one hop
farther along the route. This is the reason why most traceroute commands display their maximum hop count
before they start tracing the route — that is the maximum number of steps it will take before declaring the
destination unreachable. Also, the TTL setting may result in steps along the route timing out due to slow
responses. There are many possible reasons for this to occur.
By default, traceroute uses UDP datagrams with destination ports numbered from 33434 to 33534. The
traceroute utility usually has an option to specify use of ICMP echo request (type 8) instead, as used by the
Windows tracert utility. If you have a firewall and if you want traceroute to work from both machines (Unix-like
52 Troubleshooting
Fortinet Technologies Inc.
Common questions How to run ping and traceroute
systems and Windows) you will need to allow both protocols inbound through your FortiGate security policies
(UDP with ports from 33434 to 33534 and ICMP type 8).
You can also use the packet count column of the Policy & Objects > Policy page to track traceroute packets.
This allows you to verify the connection, but also confirm which security policy the traceroute packets are using.
Ping and traceroute have similar functions—to verify connectivity between two points. The big difference is that
traceroute shows you each step of the way, where ping does not. Also, ping and traceroute use different protocols
and ports, so one may succeed where the other fails.
You can verify your DNS connection using traceroute. If you enter an FQDN instead of an IP address for the
traceroute, DNS will try to resolve that domain name. If the name does not get resolved, you know you have DNS
issues.
The traceroute command varies slightly between operating systems. Note that in MS Windows the command
name is shortened to “tracert”. Also, your output will list different domain names and IP addresses along your
route.
Trace complete.
The first, or the left column, is the hop count, which cannot go over 30 hops. When that number is reached, the
traceroute ends.
Troubleshooting 53
Fortinet Technologies Inc.
How to check the logs Common questions
The second, third, and fourth columns display how much time each of the three packets takes to reach this stage
of the route. These values are in milliseconds and normally vary quite a bit. Typically a value of <1ms indicates a
local connection.
The fifth, or the column farthest to the right, is the domain name of that device and its IP address or possibly just
the IP address.
1. Connect to the CLI either through telnet or through the CLI widget on the web-based manager dashboard.
2. Enter exec traceroute www.fortinet.com to trace the route to the destination IP address. There are no
options for this command.
Output appears as follows:
# execute traceroute www.fortinet.com
traceroute to www.fortinet.com (66.171.121.34), 32 hops max, 84 byte packets
1 172.20.120.2 0.637 ms 0.653 ms 0.279 ms
2 209.87.254.221 <static-209-87-254-221.storm.ca> 2.448 ms 2.519 ms 2.458 ms
3 209.87.239.129 <core-2-g0-2.storm.ca> 2.917 ms 2.828 ms 9.324 ms
4 209.87.239.199 <core-3-bdi1739.storm.ca> 13.248 ms 12.401 ms 13.009 ms
5 216.66.41.113 <v502.core1.tor1.he.net> 17.181 ms 12.422 ms 12.268 ms
6 184.105.80.9 <100ge1-2.core1.nyc4.he.net> 21.355 ms 21.518 ms 21.597 ms
7 198.32.118.41 <ny-paix-gni.twgate.net> 83.297 ms 84.416 ms 83.782 ms
8 203.160.228.217 <217-228-160-203.TWGATE-IP.twgate.net> 82.579 ms 82.187 ms 82.066 ms
9 203.160.228.229 <229-228-160-203.TWGATE-IP.twgate.net> 82.055 ms 82.455 ms 81.808 ms
10 203.78.181.2 82.262 ms 81.572 ms 82.015 ms
11 203.78.186.70 83.283 ms 83.243 ms 83.293 ms
12 66.171.127.177 84.030 ms 84.229 ms 83.550 ms
13 66.171.121.34 <www.fortinet.com> 84.023 ms 83.903 ms 84.032 ms
14 66.171.121.34 <www.fortinet.com> 83.874 ms 84.084 ms 83.810 ms
This step in troubleshooting can be forgotten, but its an important one. Logging records the traffic passing
through the FortiGate unit to your network and what action the FortiGate unit took during its scanning process of
the traffic. This recorded information is called a log message.
When you configure FortiOS initially, log as much information as you can. If needed, logging of unused features
can be turned off or scaled back if the logs generated are too large.
As with most troubleshooting steps, before you can determine if the logs indicate a problem, you need to know
what logs result from normal operation. Without a baseline it is difficult to properly troubleshoot.
54 Troubleshooting
Fortinet Technologies Inc.
Common questions How to verify the contents of the routing table (in NAT mode)
When increasing logging levels, ensure that alert email is configured and both disk usage and log quota are
selected. This ensures you will be notified if the increased logging causes problems. You can also use Logging
Monitor (located in Log&Report > Monitor > Logging volume Monitor) to determine the activities that
generate the most log entries.
For more information on Logging and Log Reports, see the Logging and Reporting handbook chapter.
How to verify the contents of the routing table (in NAT mode)
When you have some connectivity, or possibly none at all a good place to look for information is the routing table.
The routing table is where all the currently used routes are stored for both static and dynamic protocols. If a route
is in the routing table, it saves the time and resources of a lookup. If a route is not used for a while and a new
route needs to be added, the oldest least used route is bumped if the routing table is full. This ensures the most
recently used routes stay in the table. If your FortiGate unit is in Transparent mode, you are unable to perform
this step.
If the FortiGate is running in NAT mode, verify that all desired routes are in the routing table: local subnets,
default routes, specific static routes, and dynamic routing protocols.
To check the routing table in the web-based manager, use the Routing Monitor by going to Router > Monitor >
Routing Monitor.
Troubleshooting 55
Fortinet Technologies Inc.
How to verify the correct route is being used Common questions
In the CLI, use the command get router info routing-table all. Sample output:
FGT# get router info routing-table all
Codes:
K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default
If you have more than one default route and wants to make sure that traffic is flowing as expected via the right
route, you can run a trace route from a machine in the local area network, this will indicate you the first hop that
the traffic goes through.
Sample output:
C:\>tracert www.fortinet.com
Trace complete.
In this scenario, the first hop contains the IP address 10.10.1.99, which is the internal interface of the
FortiGate. The second hop contains the IP address 172.20.120.2, to which the wan1 interface of the
FortiGate is connected, so we can conclude that the route via wan1 interface is being used for this traffic.
Also debug the packet flow in the CLI shows the route taken for each session.
56 Troubleshooting
Fortinet Technologies Inc.
Common questions How to verify the correct firewall policy is being used
Sample output:
id=20085 trace_id=319 func=vf_ip4_route_input line=1597 msg="find a route: gw-172.20.120.2
via wan1"
For more information on debuging the packet flow, see How to debug the packet flow.
If you have more than one firewall policy, use the count column to check which policy is being used, the count
must show traffic increasing. To do so, go to Policy & Objects > Policy page.
Also debuging the packet flow in the CLI shows the policy id allowing the traffic.
Sample output:
id=13 trace_id=1 func=fw_forward_handler line=650 msg="Allowed by Policy-14: SNAT"
For more information on debuging the packet flow, see How to debug the packet flow.
When FortiOS is in Transparent mode, the unit acts like a bridge sending all incoming traffic out on the other
interfaces. The bridge is between interfaces on the FortiGate unit.
Each bridge listed is a link between interfaces. Where traffic is flowing between interfaces, you expect to find
bridges listed. If you are having connectivity issues, and there are no bridges listed, that is a likely cause. Check
for the MAC address of the interface or device in question.
Sample output:
#diagnose netlink brctl list
list bridge information
1. root.b fdb: size=256 used=6 num=7 depth=2 simple=no
Total 1 bridges
Troubleshooting 57
Fortinet Technologies Inc.
How to check the bridging information in Transparent mode Common questions
Collision domains prevent the forwarding of ARP packets to all VLANs on an interface. Without collision domains,
duplicate MAC addresses on VLANs may cause ARP packets to be duplicated. Duplicate ARP packets can cause
some switches to reset. It is important to know what interfaces are part of which forwarding domains as this
determines which interfaces can communicate with each other.
To manually configure forwarding domains in Transparent mode, use the following FortiOS CLI command:
config system interface
edit <interface_name>
set forward-domain <integer>
end
where <name> is the name of the forwarding domain to display and <id> is the domain id.
Sample output
diagnose netlink brctl domain ione 101
show bridge root.b ione forward domain.
id=101 dev=trunk_1 6
To list the existing bridge MAC table, use the following command:
diagnose netlink brctl name host <name>
Sample output
show bridge control interface root.b host.
fdb: size=256, used=6, num=7, depth=2, simple=no
Bridge root.b host table
4 9 internal 02:09:0f:78:67:68 8
Local
1 3 wan1 00:09:0f:78:69:fe 0
Static
58 Troubleshooting
Fortinet Technologies Inc.
Common questions How to check number of sessions used by UTM proxy
Sample Output:
show bridge root.b data port.
trunk_1 peer_dev=0
internal peer_dev=0
dmz peer_dev=0
wan2 peer_dev=0
wan1 peer_dev=0
Each FortiGate model has a set limit of the maximum number of sessions the UTM proxy supports. The UTM
proxy handles all the traffic for the following protocols: HTTP, SMTP, POP3, IMAP, FTP, and NNTP. If the proxy
for a protocol fills up its session table, the FortiGate unit will enter conserve mode, where it behaves differently,
until entries and memory free up again.
av-failopen determines the behavior of the proxy until entries are free in the session table again for that
proxy.
l idledrop — This option removes idle sessions from the session table, starting with the clients that have the most
sessions currently open. This method assumes that idle sessions are not being used and it will not cause problems
to close these sessions. This is usually true, but some applications may have problems with this and start
complaining about either not having or being able to open a session. If this occurs, try another method to check if
this is really the problem. This is a secure option as no unscanned traffic is allowed to pass.
l off — This option turns off accepting any new AV sessions, but will continue to process any existing AV sessions
that are currently active. All the protocols listed (HTTP, SMTP, POP3, IMAP, FTP, and NNTP) are scanned by
FortiGate Antivirus. If AV scanning is enabled, av-failopen off is selected, and the proxy session table fills up, then
no new sessions of that type will be accepted. For example, if POP3 session table is filled and email AV scanning is
enabled, no more POP3 connections will be allowed until the session table gets some free space. This is a secure
option because no unscanned traffic is allowed to pass.
l one-shot — When memory is low, bypass the antivirus system. The name one-shot comes from the fact that once
you are in one-shot av-failopen mode, you must set av-failopen to either pass or off to restart AV scanning. This is a
Troubleshooting 59
Fortinet Technologies Inc.
How to check number of sessions used by UTM proxy Common questions
very unsecure option because it allows all traffic without AV scanning, and it never reverts to normal without manual
assistance.
l pass — When memory is low, bypass the antivirus system much as one-shot. The difference is that when memory
is freed up, the system will start AV scanning automatically again. This is an unsecure option because it allows
traffic to pass without AV scanning. However, it is better than one-shot because it automatically restarts AV
scanning when possible.
If the proxy session table is full for one or more protocols and your FortiGate unit enters into conserve or failopen
mode, it will appear as if you have lost connections, network services are intermittent or non-existent, and yet
other services work normally for a while until their sessions end and they join the queue of session-starved
applications.
Due to the amount of output from this command, you should connect to the CLI with a
terminal program, such as puTTY, that logs output. Otherwise, you will likely not be
able to access all the output information from the command.
In the following output, only the HTTP entries are displayed. The other protocols have been removed in an
attempt to shorten the output. There will be separate entries for each supported protocol (HTTP, SMTP, POP3,
IMAP, FTP, and NNTP) in each section of the output.
Worker[0]
HTTP Common
Current Connections 8/8032
Max Concurrent Connections 76
Worker Stat
Running time (HH:MM:SS:usec) 29:06:27:369365
Time in loop scanning 2:08:000198
Error Count (accept) 0
Error Count (read) 0
Error Count (write) 0
Error Count (poll) 0
Error Count (alloc) 0
Last Error 0
Acceptor Read 6386
Acceptor Write 19621
Acceptor Close 0
HTTP Stat
Bytes sent 667012 (kb)
Bytes received 680347 (kb)
Error Count (alloc) 0
Error Count (accept) 0
Error Count (bind) 0
60 Troubleshooting
Fortinet Technologies Inc.
Common questions How to check number of sessions used by UTM proxy
Worker Accounting
poll=721392/649809/42 pollfail=0 cmdb=85 scan=19266 acceptor=25975
HTTP Accounting
setup_ok=8316 setup_fail=0 conn_ok=0 conn_inp=8316
urlfilter=16553/21491/20 uf_lookupf=0
scan=23786 clt=278876 srv=368557
SMTP Accounting
setup_ok=12 setup_fail=0 conn_ok=0 conn_inp=12
scan=12 suspend=0 resume=0 reject=0 spamadd=0 spamdel=0 clt=275 srv=279
POP3 Accounting
Troubleshooting 61
Fortinet Technologies Inc.
How to check number of sessions used by UTM proxy Common questions
IMAP Accounting
setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0
scan=0 clt=0 srv=0
FTP Accounting
setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0
scan=0 clt=0 srv=0 datalisten=0 dataclt=0 datasrv=0
NNTP Accounting
setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0
scan=0 clt=0 srv=0
The output from this command falls into the following sections:
l HTTP Common current connections — There is an entry for each protocol that displays the connections
currently used, and the maximum connections allowed. This maximum is for the UTM proxy, which means all the
protocols connections combined cannot be larger than this number. To support this, note that the maximum
session count for each protocol is the same. You may also see a line titled Max Concurrent Connections for
each protocol. This number is the maximum connections of this type allowed at one time. If VDOMs are enabled,
this value is defined either on the global or per-VDOM level at VDOM > Global Resources.
l Worker Stat — This is statistics about the UTM proxy including how long it has been running, and how many errors
it has found.
l HTTP Stat — This section includes statistics about the HTTP protocol proxy. This is a very extensive list covering
errors, web responses, and any UTM positive matches. There are similar sections for each protocol, but the specific
entries in each vary based on what UTM scanning is looking for in each — spam control for email, file transfer
blocking for FTP, and so on.
l Worker Accounting — Lists accounting information about the UTM proxy such as polling statistics, how many
sessions were scanned, and how many were just accepted. This information can tell you if expect AV scanning is
taking place or not. Under normal operation there should be no errors or fails.
l HTTP Accounting — The accounting sections for each protocol provide information about successful session
creation, failures, how many sessions are being scanned or filtered, and how many are client or server originated. If
setup_fail is larger than zero, run the command again to see if it is increasing quickly. If it is, your FortiGate unit
may be in conserve mode.
Related commands
To dump memory usage:
# get test proxyworker 1
62 Troubleshooting
Fortinet Technologies Inc.
Common questions How to examine the firewall session list
One further step is to examine the firewall session. The firewall session list displays all the sessions the FortiGate
unit has open. You will be able to see if there are strange patterns such as no sessions apart from the internal
network, or all sessions are only to one IP address.
When examining the firewall session list in the CLI, filters may be used to reduce the output. In the web-based
manager, the filters are part of the interface.
When examining the firewall session list, there may be too many sessions to display. In this case it will be
necessary to limit or filter the sessions displayed by source or destination address, or NATed address or port. If
you want to filter by more than one of these, you need to enter a separate line for each value.
The following example shows filtering the session list based on a source address of 10.11.101.112.
FGT# diag sys session filter src 10.11.101.112
FGT# diag sys session list
The following example shows filtering the session list based on a destination address of 172.20.120.222.
FGT# diag sys session filter dst 172.20.120.222
FGT# diag sys session list
The NAT values can be helpful to ensure they are the values you expect, and to ensure the remote end of the
sessions can see the expected IP address and port number.
When displaying the session list in the CLI, you can match the NATed source address (nsrc) and port (nport).
This can be useful if multiple internal IP addresses are NATed to a common external facing source IP address.
FGT# diag sys session filter nsrc 172.20.120.122
FGT# diag sys session filter nport 8888
FGT# diag sys session list
Troubleshooting 63
Fortinet Technologies Inc.
How to check wireless information Common questions
Wireless connections, stations, and interfaces have different issues than other physical interfaces.
You can verify the FortiGuard connectivity in the License Information widget under System > Dashboard >
Status. When FortiGate is connected to FortiGuard, a green check mark appears for available FortiGuard
services.
Sample output:
FG100D# execute ping service.fortiguard.net
PING guard.fortinet.net (208.91.112.196): 56 data bytes
64 bytes from 208.91.112.196: icmp_seq=0 ttl=51 time=61.0 ms
64 bytes from 208.91.112.196: icmp_seq=1 ttl=51 time=60.0 ms
64 bytes from 208.91.112.196: icmp_seq=2 ttl=51 time=59.6 ms
64 bytes from 208.91.112.196: icmp_seq=3 ttl=51 time=58.9 ms
64 bytes from 208.91.112.196: icmp_seq=4 ttl=51 time=59.2 ms
64 Troubleshooting
Fortinet Technologies Inc.
Common questions How to perform a sniffer trace (CLI and Packet Capture)
When troubleshooting networks and routing in particular, it helps to look inside the headers of packets to
determine if they are traveling along the expected route. Packet sniffing can also be called a network tap, packet
capture, or logic analyzing.
If your FortiGate unit has NP2/NP4 interfaces that are offloading traffic, this will
change the sniffer trace. Before performing a trace on any NP2/NP4 interfaces, you
should disable offloading on those interfaces.
Sniffing packets can also tell you if the FortiGate unit is silently dropping packets for reasons such as Reverse
Path Forwarding (RPF), also called Anti Spoofing, which prevents an IP packet from being forwarded if its Source
IP does not either belong to a locally attached subnet (local interface), or be part of the routing between the
FortiGate unit and another source (static route, RIP, OSPF, BGP). Note that RPF can be disabled by turning on
asymmetric routing in the CLI (config system setting, set asymetric enable), however this will
disable stateful inspection on the FortiGate unit and cause many features to be turned off.
If you configure virtual IP addresses on your FortiGate unit, it will use those addresses
in preference to the physical IP addresses. You will notice this when you are sniffing
packets because all the traffic will be using the virtual IP addresses. This is due to the
ARP update that is sent out when the VIP address is configured.
<interface_name> The name of the interface to sniff, such as “port1” or “internal”. This can
also be “any” to sniff all interfaces.
Troubleshooting 65
Fortinet Technologies Inc.
How to perform a sniffer trace (CLI and Packet Capture) Common questions
What to look for in the information the sniffer reads. “none” indicates no
filtering, and all packets will be displayed as the other arguments indicate.
<‘filter’>
The filter must be inside single quotes (‘).
The number of packets the sniffer reads before stopping. If you do not put
<count>
a number here, the sniffer will run forever unit you stop it with <CTRL C>.
For a simple sniffing example, enter the CLI command diag sniffer packet port1 none 1 3. This
will display the next three packets on the port1 interface using no filtering, and using verbose level 1. At this
verbosity level you can see the source IP and port, the destination IP and port, action (such as ack), and sequence
numbers.
In the output below, port 443 indicates these are HTTPS packets, and 172.20.120.17 is both sending and
receiving traffic.
Head_Office_620b # diag sniffer packet port1 none 1 3
interfaces=[port1]
filters=[none]
For a more advanced example of packet sniffing, the following commands will report packets on any interface
travelling between a computer with the host name of “PC1” and the computer with the host name of “PC2”. With
verbosity 4 and above, the sniffer trace will display the interface names where traffic enters or leaves the
FortiGate unit. Remember to stop the sniffer, type CTRL+C.
FGT# diagnose sniffer packet any "host <PC1> or host <PC2>" 4
or
FGT# diagnose sniffer packet any "(host <PC1> or host <PC2>) and icmp" 4
The following sniffer CLI command includes the ARP protocol in the filter which may be useful to troubleshoot a
failure in the ARP resolution (for instance PC2 may be down and not responding to the FortiGate ARP requests).
FGT# diagnose sniffer packet any "host <PC1> or host <PC2> or arp" 4
Packet Capture
When troubleshooting networks, it helps to look inside the header of the packets. This helps to determine if the
packets, route, and destination are all what you expect. Packet capture can also be called a network tap, packet
66 Troubleshooting
Fortinet Technologies Inc.
Common questions How to debug the packet flow
When the capture is complete, click the Download icon to save the packet capture file to your hard disk for
further analysis.
Packet capture tells you what is happening on the network at a low level. This can be very useful for
troubleshooting problems, such as:
Before you start capturing packets, you need to have a good idea of what you are looking for. Capture is used to
confirm or deny your ideas about what is happening on the network. If you try capture without a plan to narrow
your search, you could end up with too much data to effectively analyze. On the other hand, you need to capture
enough packets to really understand all of the patterns and behavior that you are looking for.
Traffic should come in and leave the FortiGate unit. If you have determined that network traffic is not entering
and leaving the FortiGate unit as expected, debug the packet flow.
Debugging can only be performed using CLI commands. Debugging the packet flow requires a number of debug
commands to be entered as each one configures part of the debug action, with the final command starting the
debug.
Troubleshooting 67
Fortinet Technologies Inc.
How to debug the packet flow Common questions
If your FortiGate unit has FortiASIC NP4 interface pairs that are offloading traffic, this
will change the packet flow. Before performing the debug on any NP4 interfaces, you
should disable offloading on those interfaces.
The following configuration assumes that PC1 is connected to the internal interface of the FortiGate unit and has
an IP address of 10.11.101.200. PC1 is the host name of the computer.
To debug the packet flow in the CLI, enter the following commands:
FGT# diag debug disable
FGT# diag debug flow filter add <PC1>
FGT# diag debug flow show console enable
FGT# diag debug flow show function-name enable
FGT# diag debug flow trace start 100
FGT# diag debug enable
The start 100 argument in the above list of commands will limit the output to 100 packets from the flow. This
is useful for looking at the flow without flooding your log or displaying too much information.
The following is an example of debug flow output for traffic that has no matching security policy, and is in turn
blocked by the FortiGate unit. The denied message indicates that the traffic was blocked.
id=20085 trace_id=319 func=resolve_ip_tuple_fast line=2825 msg="vd-root received a packet
(proto=6, 192.168.129.136:2854->192.168.96.153:1863) from port3."
68 Troubleshooting
Fortinet Technologies Inc.
Troubleshooting resources Technical Documentation
Troubleshooting resources
Before you begin troubleshooting, you need to know Fortinet’s troubleshooting resources. Doing so will shorten
the time to solve your issue. Indeed, an administrator can save time and effort during the troubleshooting process
by first checking if the issue has been experienced before. Several self-help resources are available to provide
valuable information about FortiOS technical issues, including:
Technical Documentation
Installation Guides, Administration Guides, Quick Start Guides, and other technical documents are available
online at the following URL:
http://docs.fortinet.com
The Fortinet Video Library hosts a collection of video which provide valuable information about Fortinet products.
http://video.fortinet.com
Release Notes
Issues that are uncovered after the technical documentation has been published will often be listed in the
Release Notes that accompany the device.
Knowledge Base
The Fortinet Knowledge Base provides access to a variety of articles, white papers, and other documentation
providing technical insight into a range of Fortinet products. The Knowledge Base is available online at the
following URL:
http://kb.fortinet.com
An online technical forums allow administrators to contribute to discussions about issues related to their Fortinet
products. Searching the forum can help the administrator identify if an issue has been experienced by another
user. The support forums can be accessed at the following URL:
http://forum.fortinet.com
Troubleshooting 69
Fortinet Technologies Inc.
Fortinet Training Services Online Campus Troubleshooting resources
The Fortinet Training Services Online Campus hosts a collection of tutorials and training materials which can be
used to increase knowledge of the Fortinet products.
http://www.fortinet.com/training/
You have defined your problem, researched a solution, put together a plan to find the solution, and executed that
plan. At this point if the problem has not been solved, its time to contact Fortinet Customer Support for
assistance.
http://support.fortinet.com
70 Troubleshooting
Fortinet Technologies Inc.
Technical Support Organization Overview Fortinet Global Customer Services Organization
This section explains how Fortinet’s technical support works, as well as how you can easily create an account to
get technical support for when issues arise that you cannot solve yourself.
The Fortinet Global Customer Services Organization is composed of three regional Technical Assistance
Centers (TAC):
l The Americas (AMER)
l Europe, Middle East, and Africa (EMEA)
l Asia Pacific (APAC)
The regional TACs are contacted through a global call center. Incoming service requests are then routed to the
appropriate TAC. Each regional TAC delivers technical support to the customers in its regions during its hours of
operation. These TACs also combine to provide seamless, around-the-clock support for all customers.
Troubleshooting 71
Fortinet Technologies Inc.
Creating an account Technical Support Organization Overview
Creating an account
To receive technical support and service updates, Fortinet products in the organization must be registered. The
Product Registration Form on the support website will allow the registration to be completed online. Creating
an account on the support website is the first step in registering products.
https://support.fortinet.com/
72 Troubleshooting
Fortinet Technologies Inc.
Technical Support Organization Overview Registering a device
Once the support account has been created, product details can be provided by going to the Product
Register/Renew and Manage Product buttons displayed on the home page. Alternately, the product
registration can be completed at a later time.
Registering a device
Complete the following steps when registering a device for support purposes:
1. Log in using the Username and Password defined when the account was created
2. Under the Asset section, select Register/Renew to go to the Registration Wizard. Alternatively, use the Asset
menu at the top of the page.
Troubleshooting 73
Fortinet Technologies Inc.
Registering a device Technical Support Organization Overview
3. Get a serial number from the back of the FortiGate unit or from the exterior of the FortiGate shipping box.
4. Enter the serial number, service contract registration code or license certificate number to start the product
registration.
74 Troubleshooting
Fortinet Technologies Inc.
Technical Support Organization Overview Reporting problems
Reporting problems
Problems can be reported to a Fortinet Technical Assistance Center in the following ways:
Fortinet partners
Fortinet Partners are entitled to priority web-based technical support. This service is designed for partners who
provide initial support to their customers and who need to open a support ticket with Fortinet on their behalf. We
strongly encourage submission and follow up of support tickets using this service.
The support ticket can be submitted after logging into the partner website using one of the following links using
FortiPartner account details:
http://partners.fortinet.com
This link will redirect to the general Fortinet Partner Portal extranet website. Click Support > Online Support
Ticket.
https://forticare.fortinet.com/customersupport/Login/CommonLogin.aspx
Fortinet customers
There are two methods to report a technical issue on the Fortinet Support website: creating a technical support
ticket by product or creating any type of ticket with the Ticket Wizard for more options.
Fortinet customers should complete the following steps to create a support ticket by product:
1. Log in to the support website at the following address with the account credentials used when the account was
created: https://support.fortinet.com
2. Navigate to the top menu, click Asset and select Manage/View Products.
3. In the product list, select the product that is causing the problem.
4. On the left side bar, go to the Assistance category, and select Technical Request to create a TA Ticket.
5. Complete the Create TA Ticket fields.
6. Click View Products.
7. In the Products List, select the product that is causing the problem.
8. Complete the Create Support Ticket fields.
9. Select Finish to complete the support ticket.
Fortinet customers who would like to submit a customer service ticket, DOA ticket, RMA ticket, or FortiGuard
service ticket should use the Ticket Wizard and complete the following steps:
Troubleshooting 75
Fortinet Technologies Inc.
Assisting technical support Technical Support Organization Overview
1. Log in to the support website at the following address with the account credentials used when the account was
created: https://support.fortinet.com
2. Navigate to the top menu, click Assistance and select Create a Ticket from the drop down menu.
3. Select a ticket type and complete the remaining steps in the Ticket Wizard.
4. Select Finish to complete the ticket.
http://partners.fortinet.com
Customers should log into the following site:
http://support.fortinet.com
1. Log in with the account credentials used when the account was created.
2. Navigate to the top menu, click Assistance, and select Manage Tickets.
3. Use the search field on the View Tickets page to locate the tickets assigned to the account.
4. Select the appropriate ticket number. Closed tickets cannot be updated. A new ticket must be submitted if it
concerns the same problem.
5. Add a New Comment or Attachment.
6. Click Submit when complete.
Every web ticket update triggers a notification to the ticket owner, or ticket queue
supervisor.
The more information that can be provided to Fortinet technical support, the better they can assist in resolving
the issue. Every new support request should contain the following information:
76 Troubleshooting
Fortinet Technologies Inc.
Technical Support Organization Overview Support priority levels
l Additional log files such as Antivirus log, Attack log, Event log, Debug log or similar information to include in the
ticket as an attachment. If a third-party product is involved, for example, email server, FTP server, router, or switch,
please provide the information on its software revision version, configuration, and brand name.
Fortinet technical support assigns the following priority levels to support cases:
Priority 1
This Critical priority is assigned to support cases in which:
l The network or system is down causing customers to experience a total loss of service.
l There are continuous or frequent instabilities affecting traffic-handling capability on a significant portion of the
network.
l There is a loss of connectivity or isolation to a significant portion of the network.
l This issue has created a hazard or an emergency.
Priority 2
This Major priority is assigned to support cases in which:
Priority 3
This Medium priority is assigned to support cases in which:
Priority 4
This Minor priority is assigned to support cases in which:
l The customer is making information requests and asking standard questions about the configuration or functionality
of equipment.
Customers must report Priority 1 and 2 issues by phone directly to the Fortinet EMEA Support Center.
For lower priority issues, you may submit an assistance request (ticket) via the web system.
The web ticket system also provides a global overview of all ongoing support requests.
Troubleshooting 77
Fortinet Technologies Inc.
Return material authorization process Technical Support Organization Overview
In some cases hardware issues are experienced and a replacement unit must be sent. This is referred to as a
Return Material Authorization (RMA). In these cases or RMAs, the support contract must be moved to the new
device. Customers can move the support contract from the failing production unit to the new device through the
support web site.
1. Log in to the support web site with the credentials indicated when the account was created.
2. From Manage Products, locate the serial number of the defective unit from the list of devices displayed for the
account. The Product Info for the selected device will be displayed.
3. In the left side bar under the Assistance section, select RMA Transfer.
4. Enter the Original Serial Number of the original device, enter the New Serial Number, and click Replace to
complete the transfer.
This will transfer the support contract from the defective unit to the new unit with the serial number provided.
78 Troubleshooting
Fortinet Technologies Inc.
Copyright© 2015 Fortinet, Inc. All rights reserved. Fortinet®, FortiGate®, FortiCare® and FortiGuard®, and certain other marks are registered trademarks of Fortinet,
Inc., in the U.S. and other jurisdictions, and other Fortinet names herein may also be registered and/or common law trademarks of Fortinet. All other product or company
names may be trademarks of their respective owners. Performance and other metrics contained herein were attained in internal lab tests under ideal conditions, and
actual performance and other results may vary. Network variables, different network environments and other conditions may affect performance results. Nothing herein
represents any binding commitment by Fortinet, and Fortinet disclaims all warranties, whether express or implied, except to the extent Fortinet enters a binding written
contract, signed by Fortinet’s General Counsel, with a purchaser that expressly warrants that the identified product will perform according to certain expressly-identified
performance metrics and, in such event, only the specific performance metrics expressly identified in such binding written contract shall be binding on Fortinet. For
absolute clarity, any such warranty will be limited to performance in the same ideal conditions as in Fortinet’s internal lab tests. In no event does Fortinet make any
commitment related to future deliverables, features, or development, and circumstances may change such that any forward-looking statements herein are not accurate.
Fortinet disclaims in full any covenants, representations,and guarantees pursuant hereto, whether express or implied. Fortinet reserves the right to change, modify,
transfer, or otherwise revise this publication without notice, and the most current version of the publication shall be applicable.