VMware ESXi Troubleshooting

Paudie O’Riordan
Technical Support
pp Engineer
g
Disclaimer
This session may contain product features that are
currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in
any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility
feasibilit and market demand will
ill affect final delivery.
deli er

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.
“These features are representative of feature areas under development. Feature commitments are
subject to change, and must not be included in contracts, purchase orders, or sales agreements of
any kind. Technical feasibility and market demand will affect final delivery.”
Agenda

Technical Support Mode and access to the ESXi command line

ESXi Boot Sequence
ESXi Direct Console User Interface (DCUI)
Logs – where are they and how to manage them?
Troubleshooting Network issues on ESXi
Troubleshooting Storage issues on ESXi
VIMSH and VIM-CMD
ESXi P
Patching
t hi
Saving and Restoring ESXi configurations
ESXi Boot Sequence

ESXi firmware boot starts from grub screen

– Hypervisor loads necessary drivers
– Hypervisor runs config scripts
– Hypervisor starts daemons (services)
– Boot process ends when DCUI screen is loaded
• Alt+F2 : DCUI (Default)
• Alt+F1: inactive console
y information ((version,, server,, CPU,,
• Alt+F11 : Summary
memory)
• Alt+F12 : vmkernel log
ESXi Direct Console User Interface Navigation
For troubleshooting use the DCUI to
– Configure
C fi Networking
N t ki
– Verify Networking
– Restart
R t t Management
M t Agents
A t
– View Logs
– Access Tech Support Mode
ESXi Direct Console User Interface Navigation
Ensure your KVM or remote console solution can map to the
corresponding function keys below
Actions Keys
View and change configuration F2 Key
Shutdown or restart the Host F3 Key
Move between selection Fields Arrow Keys
Select a menu Item Enter Key
Toggle a value Spacebar
Confirm sensitive commands F11 Key / F2 Key
Save and Exit Enter key
Exit without saving ESC key
ESXI DCUI – Configure and Verify Networking
ESXI DCUI – Restart Management Agents and
Networking
Technical Support Mode and access to the ESXi
Shell prompt
Tech Support Mode provides a command-line interface that can be
used to diagnose and repair VMware ESXi hosts. Tech Support
Mode should only be used at the request of VMware Technical
Support. Tech Support Mode is
– Enabled but inactive for default VMware ESXi installations.
– Only usable with physical / remote access to the server console and
root privileges.
p g
– Unsupported unless used in consultation with VMware Tech Support.
– Accessing Tech Support Mode will be logged on the system
– However, individual commands issued at this interface are not logged
Technical Support Mode and access to the ESXi
Shell prompt
Customers should be aware of current limitations of Tech Support
Mode that May violate corporate security policies or best practices
– Commands that can be issued from this interface can result in an
unusable system
– The interface only supports logging in as root. No other user account
or role can use this management interface
– VMware only supports the use of Tech Support Mode as a last resort
to troubleshooting an ESXi host.
– As a result, customers should expect to use the Direct Console UI
(DCUI),
(DCUI) Remote Command Line Interface (RCLI) or the VI Client as
available alternatives
Technical Support Mode and access to the ESXi
Shell prompt
To use Tech Support Mode:
– Log in to your ESXi host at the DCUI console.
– Press Alt+F1 to switch to the console window.
– Enter unsupported to start the Tech Support Mode login process.
Note that no text will appear on the console window.
– Enter the password for the root user. Tech Support Mode is now
ti
active.
– Complete tasks in Tech Support Mode.
– Enter the command exit to exit Tech Support Mode.
– Press Alt+F2 to return the server to DCUI mode.
Technical Support Mode and access to the ESXi
Shell prompt
To disable Tech Support Mode:
– Connect VMware Infrastructure Client
(VI Client) to an ESXi host or a
VirtualCenter Server.
– Click the Configuration tab.
– Click the Advanced Settings link.
– Click "VMkernel"
VMkernel in the left-hand
side pane.
– Deselect
VMkernel.Boot.techSupportMode
– Restart the ESXi host.
Logs – where are they and how to manage them?
Compared to ESX 3.0.X and ESX3.5 the locations of the system or
vmkernel logs have changed slightly
vmkernel log file names are no longer used
All system
y events are logged
gg to the /var/log/messages
g g file(s)
( )
System logs are still in the /var/log location
ESXi has a quite a small storage footprint, thus logs get rotated
very quickly
The current /var/log/messages logs will not be retained after
performing
p g a reboot
If you require the system logs beyond the last reboot, it may be
advisable to log to both a remote disk and a remote syslog server.
Logs – where are they and how to manage them?
Configuring ESXi to log to a remote syslog server

From the VI Client, select your

ESXi Server
Go to Configuration Æ
Advanced SettingsÆ
y gÆ
Syslog
Remote
Enter in syslog server in the
following field
“Syslog.Remote.Hostname”
Logs – where are they and how to manage them?
Configuring ESXi to log to a local or shared datastore

From the VI Client, select

your ESXi Server
Go to
G ti Æ
t Configuration
C fi
Advanced SettingsÆ Syslog
Æ Local
Enter in the datastore and
path for the location of the
syslog directory
The format will be
[datastore]/dir
Logs – where are they and how to manage them?
ESXi Diagnostic Cores
– Instead of going to the /root dir as on ESX 3.0.X
3 0 X or ESX3.5
ESX3 5 cores will
be located on /var/core dir on ESXi
– A vm-support script will extract the diagnostic core as part of
exporting diagnostic data
– On ESX 3.0.X and ESX 3.5 the diagnostic core is extracted on restart
the server, on ESXi the core is extracted when the vm-support script
is invoked
invoked, to save on space
– ESXi will have a diagnostic core partition or VMKcore partition
enabled by default
– Use VMware Remote CLI interface to verify if a VMKcore partition is
configured
– vicfg-dumpart.pl –h for command usage
Logs – where are they and how to manage them?
VMware Agent Logs and Virtual Machine logs
h td are still
hostd till iin th
the same llocation
ti as ESX 3
3.5
5
– /var/log/vmware/hostd*
vpxa logs are still in the same location as ESX 3
3.5
5
– /var/log/vmware/vpx/vpxa*
VMware HA logs are still in the same location as ESX 3.5
– /var/log/vmware/aam
Virtual Machine log files are still stored in the same datastore
directory as the Virtual Machine configuration files
– /vmfs/volumes/my datastore/my vm/vmware.*
Logs – where are they and how to manage them?
VMware hostd logging options
– There may be times that VMware Technical Support will have to
change or enable hostd logging options for debugging reasons
– Use the VMware Remote CLI interface to copy the hostd
configuration file to your local machine for offline editing
– For example you can copy the hostd configuration file locally to your
workstation using vifs.pl
• vifs.pl --get /host/hostAgentConfig.xml localfile.xml
– Use an editor to change the logging options to trivia levels
– Copy
py the modified file back
• vifs.pl --put localfile.xml /host/hostAgentConfig.xml
– Restart your Management agents from the DCUI
Troubleshooting Network issues on ESXi
Use VMware Remote CLI interface to verify networking
List network cards
– vicfg-nics.pl –l
List interfaces
– vicfg-vmknics
icfg mknics –ll
List vswitch configurations
– esxcfg-vswitch.pl -l
If required vmkping commands are available on ESXi shell
– vmkping –D pings all configured Management Networks, VMkernel
interfaces as well as default routes
Standard networking commands such as nslookup and ping are available
on ESXi
Troubleshooting Network issues on ESXi
ESXi does not have tcpdump available as was the case with ESX
3.0.X and ESX 3.5
To Capture a NetworkTrace
– Create a new port group on the desired vSwitch
– Attach a Virtual Machine Network Interface to that Port Group
– Install, for example Wireshark or Tcpdump on the Virtual
Machine
– Enable Promiscuous mode on your vSwitch
– Select vSwitch Properties Æ Security Tab Æ Policy
Exceptions Æ Promiscuous Mode “Accept”
– If using VLANS , use VLAN 4095 on your Port Group to
capture VLAN traffic
Troubleshooting Storage issues on ESXi
Using VMware Remote CLI gives many options to view and
troubleshoot storage configurations on ESXi
Command syntax is similar to what is available on ESX 3.0.X and
ESX 3.5 service consoles
esxcfg-vmhbadevs pl
esxcfg-vmhbadevs.pl
– Lists available storage devices
esxcfg-rescan.pl
g p
– Rescans your storage adapters
vmkfstools.pl
– Manipulation of VMFS3 filesystems and virtual disk files
All equivalent commands are also available on ESXi command
shell
Troubleshooting Storage issues on ESXi
VMware Technical Support may
request to increase the logging
l
level
l off th
the ESXi storage
t llayer
Via GUI
– Go to Configuration Æ
Advanced Settings Æ
Scsi Æ
Scsi.PrintCmdErrors
sett to
t 1 to
t enable
bl
(0 is default)
Via VMware Remote CLI
interface
i t f
– esxcfg-advcfg.pl -s “1"
Scsi.PrintCmdErrors
Troubleshooting Storage issues on ESXi

Modifying Storage module parameters

This is quite similar to how it is done on ESX 3.0.X and ESX 3.5
Use VMware Remote CLI interface
For example I want to change the queue depth on my qlogic driver
Change the queue depth on your qlogic driver
– esxcfg-module.pl -s "ql2xmaxqdepth=64" qla2300_707_vmw
Verify Change
– esxcfg-module.pl
esxcfg module pl -g qla2300_707_vmw
g qla2300 707 vmw
A graceful reboot of your ESXi Server will save and apply the
changes
Troubleshooting Storage issues on ESXi
Again with the assistance of Emulex LightPulse FC SCSI elx_7.4.0.13-2
VMware Technical Support, Emulex LP10000DC-M2 2Gb 2-port PCI-X Fibre Channel Adapter on PCI bus 06
we can help customers to device 08 irqq 209
BoardNum: 0
diagnose storage connectivity SerialNum: MS63794337
issues Firmware Version: 1.91A5 (T2D1.91A5)
Hdw: 1001206d
VendorId: 0xfa0010df
ESXi compared
p to ESX 3.5 Portname: 10:00:00:00:c9:5a:78:fc Nodename: 20:00:00:00:c9:5a:78:fc
have very few proc nodes Link Up - Ready:
available to view PortID 0x6a1e00
Fabric
However certain proc nodes Current speed 2G

are still a
available
ailable for Current Mapped Nodes on Physical Port:
troubleshooting certain issues, lpfc0t00 DID 6a0a00 WWPN 50:06:01:61:30:21:b9:df
lpfc0t01 DID 6a0800 WWPN 50:06:01:69:30:21:b9:df
WWNN
WWNN
50:06:01:60:b0:21:b9:df
50:06:01:60:b0:21:b9:df
namely fibre-channel proc lpfc0t02 DID 6b0600 WWPN 50:06:01:60:41:e0:b2:0c WWNN 50:06:01:60:c1:e0:b2:0c
nodes, e.g. qlogic and emulex lpfc0t03 DID 6b0500 WWPN 50:06:01:68:41:e0:b2:0c WWNN 50:06:01:60:c1:e0:b2:0c

drivers vmkadapter: vmhba2

– /proc/scsi/vendor
Troubleshooting Storage issues on ESXi
For iSCSI connectivity
troubleshooting
g there are also
proc nodes available to quickly
diagnose connectivity issues
Software iSCSI procnodes are
available at
– /proc/scsi/vmkiscsi/
Using vmkiscsi-util on the
ESXi shell we can quickly
diagnose and repair iscsi
connectivity
ti it issues
i
– vmkiscsi-util –h for usage
Troubleshooting Storage issues on ESXi

A common scenario when using shared storage is the loss of a

partition table on shared VMFS volumes
Common symptoms of a missing partition table is
– Your VMFS volume will not mount
– If you attempt to add storage your missing VMFS volume LUN ID is
available to format
Question: How can you recover your partition using ESXi ?
At this point the only choice you will have is to enable Tech Support
mode with the help of VMware Technical Support
Luckily the tools to rebuild your partition on ESXi are available
Troubleshooting Storage issues on ESXi

The solution is to use Tech Support Mode on your ESXi server

Get help from VMware Technical Support
Your typical storage commands such as esxcfg-vmhbadevs,
esxcfg-mpath
f th are available
il bl on th
the ESXi shell,
h ll as wellll as more
advanced commands such as “fdisk” , “dd”, “hexdump”
Isolate your missing VMFS volume using “esxcfg-vmhbadevs”
– The –q switch will list all accessible luns
– The –m switch will list all mounted and healthy VMFS volumes
– Use “fdisk –lu” on the lun that does not mount
– If the partition entry is blank then it needs to be repaired
Troubleshooting Storage issues on ESXi

VMware Technical Support can help you rebuild your partition

For example on the below “fdisk –lu” output there is no partition
table entry
You
Y willill note
t on ESXi there
th iis no conceptt off /d
/dev/sdX
/ dX instances
i t for
f
storage devices, use /vmfs/devices/disks/vml.* format
Troubleshooting Storage issues on ESXi

Rebuild the Partition information with the help of VMware Technical

Support
Always verify before modifying, use esxcfg-mpath –lv
Always
Al use the
th //vmfs/devices/disks/vml.XXX
f /d i /di k / l XXX ffrom th
the esxcfg-
f
mpath output name for modifying partitions
Troubleshooting Storage issues on ESXi

There maybe times that VMware Technical Support will request a

dump of metadata of a particular lun
– To verify VMFS header information
– To verify data integrity of a VMFS volume
There are more advanced tools available on the ESXi console to
VMware Technical support to gather such information
– dd is supplied on the ESXi to copy out disk header data to a file
– hexdump is also available for live troubleshooting
As
A always
l never use T
Tech
hSSupportt M
Mode
d unless
l requested
t dbby
VMware Technical Support.
VIMSH/VIM-CMD

VIMSH is an interactive shell buffer available on ESXi and ESX 3.5

In the context of VMware ESX it has been implemented as an API for
the ESX host agent.
VIM-CMD
VIM CMD was introduced in ESX 3 3.5. wrapper for
5 This command is a ‘wrapper’
the vimsh shell. It simplifies the command and means that vimsh can
be run without having to enter and exit the shell and use previously
documented command switches and “quotes”. q
The vim-cmd command is executed from the ESXi shell.
– Example: vim-cmd /hostsvc/maintenance_mode_enter
– This runs a non-interactive command which puts an ESXi server into
Maintenance Mode.
VIM-CMD

Within the vim-cmd there are a vast array of interactive commands

that allow reporting and configuration of most aspects of the ESXi
host.
Commands consist of 2 main types; either plugin or shell based.
The shell commands revolve around management of plugins or
execution of simple help, list or exit type controls. Shell commands
are common to the vimsh interface
The plugin commands are specific to a group of particular ESXi
functions or Command Sets
Changes made by using this command appear in the VI Client
console at normal refresh schedules.
VIM-CMD

Shell commands
– The following list outlines the available shell commands for the vim-
cmd interface. To start, Enter Tech Support Mode and type vim-cmd
at the command prompt.
– If you are unfamiliar with the command ensure you only use non-
production test servers when experimenting.
vim-cmd
[/]$ vim cmd
Commands available under /:
y
hostsvc/ proxysvc/ vimsvc/ help
internalsvc/ solo/ vmsvc/
VIM-CMD

It is to be noted that vmware-cmd is also available from VMware

Remote CLI interface to control Virtual Machine power and
snapshots. vmware-cmd is not available on the ESXi shell
– vmware cmd.pl –h
vmware-cmd.pl h for usage
However for troubleshooting you could use vim-cmd with VMware
Technical Support supervision if RCL or the VI Client is not
available
– List virtual machines registered on esx host
• vim cmd /vmsvc/getallvms
vim-cmd
– List running virtual Machines
• vm-support
pp -x
VIM-CMD

From the vm-support –x output on running Virtual Machines

You Can check the status of a Virtual Machine using
– vim-cmd /vmsvc/get.guest <vim id>
At this point VMware Technical Support can help you to
– Power off a Virtual Machine
• vim-cmd /vmsvc/power off <vim id>
/vmsvc/power.off
– Power on a Virtual Machine
• vim-cmd /vmsvc/power.on <vim id>
– Reset a Virtual Machine
• vim-cmd /vmsvc/power.reset <vm id>
VIM-CMD

Again if there was a situation that VMware Remote CLI or the VI

Client was not available the below methods could be used to
manipulate Virtual Machines and Snapshots. VMware Technical
Support assist you here.
This set of commands allows the full manipulation of VM snapshots
including creating, removing, reverting and reporting.
– vim-cmd help vmsvc/snapshot create
vmsvc/snapshot.create
– vim-cmd help vmsvc/snapshot.get
– vim-cmd help vmsvc/snapshot.remove
– vim-cmd help vmsvc/snapshot.removeall
– vim-cmd help vmsvc/snapshot.revert
ESXi and Patching

Each ESXi host can store two builds, one boot build and one
standby
t db b build.
ild Th
The b
boott b
build
ild iis th
the one ffrom which
hi h th
the system
t iis
booted.
For each update, the update utility updates the standby build. After
the
th update,
d t you th
then reboot
b t ththe h
host. On reboot,
t O b t the
th newly
l updated
d t d
build becomes the boot build.
If the update is successful, the host continues to boot from the new
b tb
boot ild until
build til th
the nextt update.
d t
There are three methods to Patch your ESXi server
p
– VMware vCenter Update g
Manager
– Performing Maintenance with Infrastructure Update
– Performing Maintenance with the vihostupdate Remote CLI Utility
ESXi and Patching

Rolling back an update

During the boot process, you can manually boot into the standby
build. This causes an irreversible rollback.
In
I other
th words,d when
h you manually ll select the standby
l t th t db build,
b ild the
th
standby build becomes the new boot build and remains the boot
build until you perform another update or manually select the
standby build
You will need physical access to the ESXi DCUI to rollback to a
standby build.
ESXi and Patching

Rolling back an update

Reboot the ESXi host.
Connect to the DCUI
When you see the page that
displays the Loading VMware
Hypervisor…, press Shift + r to
l
select h standby
the db b ild
build.
Press Shift + y to confirm the
selection and press Enter.
This will load the Standby build
and the ESXi server will now
boot from the standby build
Saving and Restoring ESXi Configurations

You perform a backup of your ESXi system by using the vicfg-

cfgbackup command, which you run from the VMware Remote CLI
The Remote CLI is available in two forms:
pp
– As a virtual appliance that yyou import
p into ESXi Server, VMware
Workstation, or VMware Player.
– As a package that you install on Microsoft Windows and Linux
machines.
Always back up your host configuration after you change the
configuration or upgrade the ESXi Server image.
To
T restore
t a configuration
fi ti you mustt run th
the R
Remote
t CLI virtual
it l
appliance from a remote host. When you restore the configuration,
the target host must be in maintenance mode.
Saving and Restoring ESXi Configurations

Command usage of vicfg-cfgbackup.pl

Saving your ESXi configuration
– vicfg-cfgbackup --server <3i-host-ip> --portnumber <port_number> --
protocol <protocol
<protocol_type>
type> --username
username root --password
password
<root_password> -s <backup-filename>
Restoring your ESXi configuration
– vicfg-cfgbackup --server <3i-host-ip> --portnumber <port_number> --
protocol <protocol_type> --username root --password
<root_password> -l <backup-filename>
– This action will prompt you to reboot your host.
 Thank you for coming.

Rate your session and

g
watch for the highest scores!