Basic F5 LTM Troubleshooting SSL Ciphersuits - Using Httpwatch and Long Run Tcpdumps

1
Basic F5 LTM troubleshooting

SSL ciphersuits, using httpwatch
and long run tcpdumps
Presented by:
Peter Nowacki
(Sr. Network Support Engineer)
F5 Company Confidential
2
No good news yet.
The master key dump in tmos needs a bit more work

on the serverside SSL
3
Issues in new versions

1. 11.3.0 – 11.5.1 upgrade
SSL SNI default profile
Below errors are logged when SSL log level is set to “Debug”
Feb 28 03:17:54 scooter info tmm1[21402]: 01260013:6: SSL Handshake

failed for TCP 192.168.163.149:51510 -> 172.29.20.88:443
Feb 28 03:18:06 scooter debug tmm1[21402]: 01260009:7: Connection

error: ssl_hs_rxhello:5785: name not available (112)
4
Issues in new versions

1. 11.3.0 – 11.5.1 upgrade
11.3.0 allows the Default SNI profile

allows connections from clients that do NOT
Support SNI extension even if the profile has
Require Peer SNI support – enabled
(Despite what help says)
11.5.1 works as doc says it does.

It resets the connections from non SNI clients
Upgrade breaks what worked in 11.3.0
5
Issues in (not so)new versions
Beginning in BIG-IP 11.4.0, as part of the firewall integration, the

tm.minipfragsize BigDB variable was introduced. This is part of the denial-of-
service (DoS) protection vectors incorporated within BIG-IP AFM and TMOS.
tmsh list sys db tm.minipfragsize

sys db tm.minipfragsize {
value "576“ (later changed to 556)
}
We see recently often Radius UDP traffic fragmented below 100 bytes
tmsh modify sys db tm.minipfragsize value 20

tmsh save sys config
6
tmctl ip_stat
The incremental counter for the err_len column may indicate that
the BIG-IP system is possibly encountering tm.minipfragsize
threshold packet drops.
To verify, you should capture traffic and then confirm if traffic is
affected.
7

For a BIG-IP system that is licensed with AFM
You may observe the packet drops as a result of this new setting if you review the
dos_stat table using the tmctl command. The useful information from the counters on
the columns related to drops can be observed from the following vectors:
ICMP fragmented
IP short fragment
To view the dos_stat table's ICMP fragmented vector, type the following command:
tmctl dos_stat vector_name="ICMP fragmented"
To view the dos_stat table's IP short fragment vector, type the following command:
tmctl dos_stat vector_name="IP short fragment"
The counters with incremental drops for ICMP fragmented or IP short fragment may
indicate that the BIG-IP system is possibly encountering tm.minipfragsize threshold
8
Issues in new versions ???
https://support.f5.com/kb/en-
us/solutions/public/15000/000/sol15003.html
The odd-numbered hyper-threads (1, 3, 5, 7, etc.) for host daemons

The even-numbered hyper-threads (0, 2, 4, 6, etc.) are dedicated to TMM
This is not confirmed yet if the below is caused

by the new split planes behavior.
9

More of an ASM problem than pure +LTM
All ASM and AVR daemons sit on Core 7
Mysql, monpd, bd, policy builder
10
This is normal and would be trouble-less

behaviour, If cores 0-6 were not that lazy
11
SSL Ciphersets - Lab

Recent changes in SSL stack(s)
and SSL vulnerabilities require changes
to SSL Profiles
But what cipherstring results in what cipher sets ???

“tmm –-clientciphers / --serverciphers” commands
can help us here
– the problem is the presentation layer
https://support.f5.com/kb/en-
us/solutions/public/15000/100/sol15194.html
12
Long run tcpdump - Lab

Sometimes when the issue occurs really rarely
and randomly, we may need to run the tcpdump
for a long period of time and then the disk space
becomes a concern.
We have 2 scripts that can help us with this.

They monitor a chosen log file for a defined
error message to collect the data for us,
while rotating older dumps to save the disk space.
13
find_error.pl - Lab
Included in all recent versions
/usr/share/ts/bin/find_error.pl -h
/usr/share/ts/bin/find_error.pl -l /var/log/ltm -t "monitor

status down" -p 2000 -s 0 -nnn -o /root/find_error.tar
find_error.pl does NOT stop when the string is found

So if we get the message logged 3 times during the run
we get 3 dump files.
I saw it sometimes missing the message.
14
ringdump.pl - Lab
Custom script distributed via Devcentral
https://devcentral.f5.com/wiki/advdesignconfig.ringdump.a
shx
I modified mine to use –i 0.0
my %SETTINGS =(
"0.0:nnn" => { filter => "host 10.0.0.100 and port 80 or arp or icmp" },
);
And added “kill me” message

15
ringdump.pl - Lab
./ringdump.pl (default destination /var/tmp)
./ringdump.pl /var/tmp/dumps/
All can be configured my directly editing the script
16
ringdump.pl - Lab
By default the output file is in /var/tmp
With the name of the interface used
/var/tmp/0.0:nnn.dump.2
(escape the : with \ when working on the command line)
./ringdump.pl /var/tmp/dumps/
Makes the /var/tmp/dumps/ the destination directory
If you miss the “kill me” message, run:

cat /var/run/ring_dump.pid
This is the parent PID so children will be killed too.
17
ringdump.pl - Lab
1. Rotates the files over and over
2. Stops automatically when the error is seen
So only one occurrence will be captured
Make sure you properly balance the number of files

vs the size of the file vs the sleeping time.
If you have only 2 small files, lots of traffic and a timeout of
30 seconds you may actually have the files rotated out
during the 30 second sleep time and loose the precious
packet trace.
18
HTTPWatch / Fiddler - Lab

Sometimes tcpdump is NOT enough
Some browser request may simply not be
hitting the expected VIP (80 vs 443)
or LTM at all
(redirects or external src tags in html)
Then we need to see the problem from the

browser’s perspective.
19
HTTPWatch / Fiddler
Lab 1
Problem description - F5 is eating my pictures

sometimes called –
“SSL handshake is eating my pictures”
(Usually new implementations)
Peter.html
20
HTTPWatch / Fiddler
Lab 2
Problem description -
F5 delays my traffic 20 – 40 seconds
It is taking long if the remote website
silently drops connections.
If it resets them, the browser does not wait
the timeout time.
(worked before, now broken)
Peter_1.html F5 Company Confidential
21
HTTPWatch / Fiddler
Real life examples where httpwatch helped a lot
1. Ads injecting browser plugins causing delays

2. 4.5 sec delay for every request
(rev DNS lookup for LTM’s SNAT ip address)
3. Content modified between LTM and the client
(English error page replaced German content on
the fly by AKAMAI)
- compare tcpdump to httpwatch
22
HTTPWatch / Fiddler
Free version of HTTPWatch has some features disabled

(i.e search)
Full Fiddler is free but works a Proxy
which sometimes may be an advantage – mobile devices
We will always ask for hwl (httpwatch) output files.
23
Summary of commands presented

tmm --clientcipher 'ALL'
tmm --clientcipher 'DEFAULT'
tmm --clientcipher 'NATIVE'
tmm --clientcipher 'COMPAT'
tmm --clientcipher 'HIGH'
tmm --clientcipher 'EXPORT'
openssl ciphers -v 'ALL'

openssl ciphers -v 'DEFAULT‘
tmm --clientcipher 'ALL:!RSA' | grep RSA

tmm --clientcipher 'DEFAULT:!DH:!EDH:!DHE:!ADH:!ECDHE-RSA' | grep ECDHE-RSA
tmm --clientcipher 'DEFAULT:!DH:!EDH:!DHE:!ADH:!ECDHE-RSA-DES-CBC3-SHA' | grep DH
tmm --clientcipher 'ALL:!SSL3' | grep -i ssl
tmm --clientcipher 'ALL:!SSLv3' | grep -i ssl
tmm --clientcipher 'ALL:!TLS1.2' | grep -i tls
tmm --clientcipher 'ALL:!TLSv1_2' | grep -i tls
tmm --clientcipher 'ALL:!DES' | grep -i des
tmm --clientcipher 'ALL:!3DES' | grep -i des
24
This 4th webex session completes

the presentation of the basic tools and
techniques we use here at F5 Support
department for troubleshooting LTM devices.
Now we need you feedback for choosing
topics for the future sessions.
25

Basic F5 LTM Troubleshooting SSL Ciphersuits - Using Httpwatch and Long Run Tcpdumps

Uploaded by

Copyright:

Available Formats

Basic F5 LTM Troubleshooting SSL Ciphersuits - Using Httpwatch and Long Run Tcpdumps

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Basic F5 LTM Troubleshooting SSL Ciphersuits - Using Httpwatch and Long Run Tcpdumps

Uploaded by

Copyright:

Available Formats

1

Basic F5 LTM troubleshooting

No good news yet.

The master key dump in tmos needs a bit more work

Issues in new versions

Feb 28 03:17:54 scooter info tmm1[21402]: 01260013:6: SSL Handshake

Feb 28 03:18:06 scooter debug tmm1[21402]: 01260009:7: Connection

Issues in new versions

11.3.0 allows the Default SNI profile

11.5.1 works as doc says it does.

Issues in (not so)new versions

Beginning in BIG-IP 11.4.0, as part of the firewall integration, the

tmsh list sys db tm.minipfragsize

tmsh modify sys db tm.minipfragsize value 20

Issues in (not so)new versions

Issues in (not so)new versions

tmctl dos_stat vector_name="ICMP fragmented"

tmctl dos_stat vector_name="IP short fragment"

Issues in new versions ???

The odd-numbered hyper-threads (1, 3, 5, 7, etc.) for host daemons

This is not confirmed yet if the below is caused

Issues in new versions ???

Mysql, monpd, bd, policy builder

Issues in new versions ???

This is normal and would be trouble-less

SSL Ciphersets - Lab

But what cipherstring results in what cipher sets ???

Long run tcpdump - Lab

We have 2 scripts that can help us with this.

/usr/share/ts/bin/find_error.pl -l /var/log/ltm -t "monitor

find_error.pl does NOT stop when the string is found

I modified mine to use –i 0.0

And added “kill me” message

./ringdump.pl (default destination /var/tmp)

All can be configured my directly editing the script

If you miss the “kill me” message, run:

Make sure you properly balance the number of files

HTTPWatch / Fiddler - Lab

Then we need to see the problem from the

Problem description - F5 is eating my pictures

1. Ads injecting browser plugins causing delays

Free version of HTTPWatch has some features disabled

We will always ask for hwl (httpwatch) output files.

Summary of commands presented

openssl ciphers -v 'ALL'

tmm --clientcipher 'ALL:!RSA' | grep RSA

This 4th webex session completes

You might also like