• Author: Kevin Grigorenko
• Version: V19 (March 20, 2023)

• Introduction
• Core Concepts
• Installation
  • With podman
  • With Docker Desktop
• Starting the lab
  • Start with podman
  • Start with Docker Desktop
• Request Timing
• Admin Center
• IBM Java and IBM Semeru Runtimes Thread Dumps
• Garbage Collection
• Health Center Sampling Profiler
• HTTP NCSA Access Log
• Methodology
• Appendix
  • Windows Remote Desktop Client

IBM® WebSphere® Application Server (WAS) is a platform for serving Java™-based applications. WAS comes in two major product forms:

1. WAS traditional (colloquially named tWAS): Released in 1998 and still fully supported and used by many.

2. WebSphere Liberty: Released in 2012 and designed for fast startup, composability, and the cloud. The commercial WebSphere Liberty product is built on top of the open source OpenLiberty. The colloquial term 'Liberty' may refer to WebSphere Liberty, OpenLiberty, or both.

WAS traditional and Liberty share some source code but differ in significant ways.

This lab covers the major tools and techniques for performance tuning WebSphere Liberty. This is a subset of the WebSphere Performance and Troubleshooting Lab which also covers WAS traditional.

This lab container image comes with WebSphere Liberty pre-installed so installation and configuration steps are skipped.

The way we are using these container images is to run multiple services in the same container (e.g. VNC, Remote Desktop, WebSphere Liberty, a full GUI server, etc.) and although this approach is valid and supported, it is generally not recommended for real-world container images. For labs that demonstrate how to use WebSphere in containers in production, see WebSphere Application Server and Docker Tutorials.

This lab is built on top of Linux (specifically, Fedora Linux, which is the open source foundation of RHEL). The concepts and techniques apply generally to other supported operating systems although details of other operating systems may vary significantly and are covered elsewhere.

WebSphere Liberty supports Java 8, 11, 17, or 19 editions.

This lab uses IBM Java 8. The concepts and techniques apply generally to other Java runtimes although details of other Java runtimes (e.g. HotSpot) vary significantly and are covered elsewhere.

The IBM Java virtual machine (named J9) has become largely open sourced into the Eclipse OpenJ9 project. OpenJ9 ships with OpenJDK through the IBM Semeru Runtimes offering. OpenJDK is somewhat different than the JDK that IBM Java uses. Note that some IBM Java tooling such as HealthCenter is not yet available in IBM Semeru Runtimes (except on z/OS) which is why we chose IBM Java 8 for this lab; however, generally, more modern versions of Java such as IBM Semeru Runtimes would be recommended.

Performance tuning is best done with all layers of the stack in mind. This lab will focus on the layers in bold below:

The lab image is about 20GB. If you plan to run this in a classroom setting, perform the installation steps beforehand which includes downloading the image.

This lab assumes the installation and use of podman or Docker Desktop to run the lab (other container systems may also work but have not been tested). Choose one or the other:

• Installing podman
• Installing Docker Desktop

If you are using podman instead of Docker Desktop, perform the following steps to install podman and then perform the podman post-installation steps. If you are using Docker Desktop, skip down to Installing Docker Desktop.

podman installation instructions:

• Windows: https://podman.io/getting-started/installation#windows
• macOS: https://podman.io/getting-started/installation#macos
• For a Linux host, simply install podman

1. On macOS and Windows:
   1. Create the podman virtual machine with sufficient memory (preferrably at least 4GB and, ideally, at least 8GB), CPU, and disk. For example (memory is in MB):

      podman machine init --memory 10240 --cpus 4 --disk-size 100
      

   2. Start the podman virtual machine:

      podman machine start
      

   3. Switch to a "root" podman connection:

      podman system connection default podman-machine-default-root
      

   4. Run the following command to allow producing core dumps within the container:

      podman machine ssh "sh -c 'mkdir -p /etc/sysctl.d/; ln -sf /dev/null /etc/sysctl.d/50-coredump.conf && sysctl -w kernel.core_pattern=core'"
      

2. Download the image:

   podman pull quay.io/ibm/webspherelab
   

   This command may not show any output for a long time while the download completes.

The following section on Docker Desktop should be skipped since you are using podman. The next section for podman is Start with podman.

If you are using Docker Desktop instead of podman, perform the following steps to install Docker Desktop and then perform the Docker Desktop post-installation steps:

• Windows (we recommend using WSL2):
  • Required:

    • Windows 11 64-bit: Home or Pro version 21H2 or higher, or Enterprise or Education version 21H2 or higher.
    • Windows 10 64-bit: Home or Pro 21H1 (build 19043) or higher, or Enterprise or Education 20H2 (build 19042) or higher.

  • Download: https://docs.docker.com/desktop/install/windows-install/
• macOS ("macOS must be version 11 or newer"):
  • Download: https://docs.docker.com/desktop/install/mac-install/
• For a Linux host, simply install and start Docker (e.g. sudo systemctl start docker):
  • For an example, see https://docs.docker.com/engine/install/fedora/

1. Ensure that Docker is started. For example, start Docker Desktop and ensure it is running:
   
   macOS:
   
   
   Windows:
   

2. Ensure that Docker receives sufficient resources, particularly memory (at least 4GB and, ideally, at least 8GB), CPU, and disk:

   1. macOS:

      1. Click the Docker Desktop icon and select Dashboard

      2. Click the Settings gear icon in the top right, then click Resources on the left.

      3. Configure sufficient memory (preferrably at least 4GB and, ideally, at least 8GB), CPU, and disk.

      4. Click Apply & Restart
         
         macOS:
         

   2. The Windows WSL2 backend defaults to 50% of RAM or 8GB, whichever is less, and the same number of CPUs as the host. This may be overridden with a %UserProfile%\.wslconfig file with, for example (preferrably at least 4GB and, ideally, at least 8GB):

      [wsl2]
      memory=10GB
      processors=4
      

3. Open a terminal or command prompt and download the image:

   docker pull quay.io/ibm/webspherelab
   

Depending on whether you installed podman or Docker Desktop, start the container:

• Start with podman
• Start with Docker Desktop

The following section is the start of the lab. If you were only preparing for the lab by installing and downloading the lab before the lab begins, then you may stop at this point until the instructor provides further direction.

If you are using podman for this lab instead of Docker Desktop, then perform the following steps. If you are using Docker Desktop, skip down to Start with Docker Desktop.

1. Open a terminal or command prompt:
   
   macOS:
   
   
   Windows:
   

2. Start the lab:

   podman run --cap-add sys_chroot --rm -p 5901:5901 -p 5902:5902 -p 3390:3389 -p 9080:9080 -p 9443:9443 -it quay.io/ibm/webspherelab
   

3. Wait about 5 minutes until you see the following in the output (if not seen, review any errors):

   =========
   = READY =
   =========
   

4. VNC or Remote Desktop into the container:

   1. macOS built-in VNC client:

      1. Open another tab in the terminal and run:

         1. open vnc://localhost:5902

         2. Password: websphere

   2. Linux VNC client:

      1. Open another tab in the terminal and run:

         1. vncviewer localhost:5902
            • You may need to install vncviewer first.
         2. Password: websphere

   3. Windows 3^rd party VNC client:

      i. If you are able to install and use a 3^rd party VNC client, then connect to localhost on port 5902 with password websphere.

   4. Windows Remote Desktop client:

      i. Windows requires a few steps to make Remote Desktop work with a Docker container. See Appendix: Windows Remote Desktop Client for instructions.

5. When using VNC, you may change the display resolution from within the container and the VNC client will automatically adapt. For example:
   
   

The following section on Docker Desktop should be skipped since you are using podman. The next section for podman is Apache JMeter.

The following section is the start of the lab. If you were only preparing for the lab by installing and downloading the lab before the lab begins, then you may stop at this point until the instructor provides further direction.

If you are using Docker Desktop for this lab instead of podman:

1. Open a terminal or command prompt:
   
   macOS:
   
   
   Windows:
   

2. Start the lab by starting the Docker container from the command line:

   docker run --rm -p 5901:5901 -p 5902:5902 -p 3390:3389 -p 9080:9080 -p 9443:9443 -it quay.io/ibm/webspherelab
   

3. Wait about 5 minutes until you see the following in the output (if not seen, review any errors):

   =========
   = READY =
   =========
   

4. VNC or Remote Desktop into the container:

   1. macOS built-in VNC client:

      1. Open another tab in the terminal and run:

         1. open vnc://localhost:5902

         2. Password: websphere

   2. Linux VNC client:

      1. Open another tab in the terminal and run:

         1. vncviewer localhost:5902
            • You may need to install vncviewer first.
         2. Password: websphere

   3. Windows 3^rd party VNC client:

      i. If you are able to install and use a 3^rd party VNC client (there are a few free options online), then connect to localhost on port 5902 with password websphere.

   4. Windows Remote Desktop client:

      i. Windows requires a few steps to make Remote Desktop work with a Docker container. See Appendix: Windows Remote Desktop Client for instructions.

5. When using VNC, you may change the display resolution from within the container and the VNC client will automatically adapt. For example:
   
   

Apache JMeter is a free tool that drives artificial, concurrent user load on a website. The tool is pre-installed in the lab image and we'll be using it to simulate website traffic to the DayTrader7 sample application pre-installed in the lab image.

1. Double click on JMeter on the desktop:
   
   

2. Click File → Open and select:

   1. /opt/daytrader7/jmeter_files/daytrader7_liberty.jmx

3. By default, the script will execute 4 concurrent users. You may change this if you want (e.g. based on the number of CPUs available):
   
   

4. Click the green run button to start the stress test and click the Aggregate Report item to see the real-time results.
   
   

5. It will take some time for the responses to start coming back and for all of the pages to be exercised.

6. Ensure that the Error % value for the TOTAL row at the bottom is always 0%.
   
   

   1. If there are any errors, review the logs:

      1. /logs/messages.log

1. Stop a JMeter test by clicking the STOP button:

   

2. Click the broom button to clear the results in preparation for the next test:

   

3. If it asks what to do with the JMeter log files from the previous test, you may just click Overwrite existing file:

   

If you would like to view or edit text files in the container using a GUI tool, you may use a program such as mousepad:

This lab demonstrates how to enable and use WebSphere Liberty's slow and hung request detection feature. You can specify a threshold for what the application stakeholders define as an excessively slow web request and Liberty watches requests and prints a stack trace and other details when a request exceeds such a threshold. Note that enabling requestTiming-1.0 may have a large overhead in high volume environments and ideally its overhead should be measured in a test environment before applying to production; if the overhead is too high, test with request sampling through the sampleRate attribute.

To detect slow and hung requests, the Liberty requestTiming-1.0 feature is required in a Liberty server's configuration. In addition to enabling the feature, the feature must be configured with a requestTiming element that specifies the thresholds and other configuration. This lab will be using the following:

<requestTiming slowRequestThreshold="60s" hungRequestThreshold="180s" sampleRate="1" />

The key configuration is the slowRequestThreshold which specifies the time after which a request is considered to be "slow" and diagnostics are printed to Liberty logs. In general, this value must be decided with application stakeholders based on service level agreements. Note that if a high volume of requests start to exceed this threshold, there will be some overhead of printing diagnostics to the logs, in which case you can consider increasing the threshold or the sampleRate.

The difference between the slow and hung thresholds is that if the hung threshold is exceeded, then Liberty will gather 3 thread dumps, one minute apart for more in-depth diagnostics. This will not be demonstrated in this lab but it is generally advised to configure both thresholds.

1. Modify /config/server.xml (for example, using the Mousepad program on the Desktop) to add the following before </server> and save the file:

   <featureManager><feature>requestTiming-1.0</feature></featureManager>
   <requestTiming slowRequestThreshold="60s" hungRequestThreshold="180s" sampleRate="1" />
   

2. Execute a request that takes more than one minute by opening a browser to http://localhost:9080/swat/Sleep?duration=65000
3. After about a minute and the request completes, review the requestTiming warning in /logs/messages.log (for example, using the Mousepad program) -- for example:

   [3/20/22 16:16:52:250 UTC] 0000007b com.ibm.ws.request.timing.manager.SlowRequestManager         W TRAS0112W: Request AAAAPOsEvAG_AAAAAAAAAAA has been running on thread 00000079 for at least 60003.299ms. The following stack trace shows what this thread is currently running.
   
     at java.lang.Thread.sleep(Native Method)
     at java.lang.Thread.sleep(Thread.java:956)
     at com.ibm.Sleep.doSleep(Sleep.java:35)
     at com.ibm.Sleep.doWork(Sleep.java:18)
     at com.ibm.BaseServlet.service(BaseServlet.java:73)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
     [...]
   
   The following table shows the events that have run during this request.
   
   Duration      Operation
   60008.080ms + websphere.servlet.service | swat | Sleep?duration=65000 
   

   1. The warning shows a stack at the time requestTiming notices the threshold is breached and it's followed be a tree of components of the request. The plus sign (+) indicates that an operation is still in progress. The indentation level indicates which events requested which other events.
4. Execute a request that takes about three minutes by opening a browser to http://localhost:9080/swat/Sleep?duration=200000
5. After about five minutes, review the requestTiming warning in /logs/messages.log -- in addition to the previous warning, multiple thread dumps are produced:

   [3/20/22 16:22:23:565 UTC] 0000007d com.ibm.ws.kernel.launch.internal.FrameworkManager           A CWWKE0067I: Java dump request received.
   [3/20/22 16:22:23:662 UTC] 0000007d com.ibm.ws.kernel.launch.internal.FrameworkManager           A CWWKE0068I: Java dump created: /opt/ibm/wlp/output/defaultServer/javacore.20220320.162223.17.0001.txt
   

   1. Thread dumps will be captured, one minute apart, after the threshold is breached.
6. Detailed steps on analyzing thread dumps are covered in a subsequent lab.

In general, it is a good practice to use requestTiming, even in production. Configure the thresholds to values that are at the upper end of acceptable times for the users and the business. Configure and test the sampleRate to ensure the overhead of requestTiming is acceptable in production.

When the requestTiming feature is enabled, the server dump command will include a snapshot of all the event trees for all requests thus giving a very nice and lightweight way to see active requests in the system at a detailed level (including URI, etc.), in a similar way that thread dumps do the same for thread stacks.

WebSphere Liberty's Admin Center is an optional, web-based administration and monitoring tool for Liberty servers. Admin Center is enabled with the adminCenter-1.0 feature as well as configuring administrator credentials.

If you are already using another monitoring product, you may skip this Admin Center lab.

For monitoring statistics to be produced and visualized in Admin Center (or published to monitoring products such as Instana), the Liberty monitor-1.0 feature is required in a Liberty server's configuration. This feature enables the collection of monitoring statistics which can then be consumed by clients such as the Admin Center, Instana, etc. Note that enabling monitor-1.0 may have an overhead of up to a few percent although this may be minimized by filtering to specific statistics, if needed.

This lab will demonstrate how to enable Admin Center and monitor-1.0 and visualize statistics for the sample application.

1. Modify /config/server.xml (for example, using the Mousepad program on the Desktop) to add the following before </server> and save the file:

   <featureManager><feature>adminCenter-1.0</feature><feature>monitor-1.0</feature></featureManager>
   

2. Open the browser (for example, using Applications in the top left and then Web Browser)

3. Navigate to https://localhost:9443/adminCenter and accept the self-signed certificate.

4. Log in with the user name wsadmin and the password websphere.

5. Click on the Explore button:
   

6. Click on the "Monitor" button:
   

7. Start JMeter

8. Go back to the Admin Center Monitor screen and you will see graphs of various statistics for this server:
   

9. The default statistics are:

   • Used Heap Memory: Java heap usage. Note that most modern garbage collectors are generational meaning that trash tends to accumulate in old regions and is cleaned up less often thus leading to a common sawtooth pattern. In some cases, the rise on such a tooth might look like a leak, but may not be.
   • Loaded Classes: The total number of loaded and unloaded classes.
   • Active JVM Threads: The number of live, total, and peak threads. Note that this shows all threads, not just the Default Executor threads.
   • CPU Usage: Average CPU utilization for the process.

10. Click the pencil button in the top right to edit the displayed statistics:

    

11. Click the plus buttons to the right of Thread Pool, Connection Pool, and Sessions. Note that it's expected that the + button won't turn into a checkbox when you click on Connection Pool and Sessions; these will become checked after one of the subsequent steps below.

    

12. Close the metrics side bar:

    

13. Scroll down to the new statistics boxes. For the In Use Connections and Average Wait Time (ms) boxes, use the Select data sources... dropdown to select jdbc/TraceDataSource. Note that if you want to view multiple data sources, it's generally better to create multiple boxes and choose one data source per box instead of checking the All box as it's harder to interpret aggregated statistics.

    

14. Scroll down to the Active Sessions box and use the Select sessions... dropdown to select default_host/daytrader. As above, if you want to view sessions for multiple applications, it's generally better to create separate boxes.

    

15. Click the Save button at the top right to make the box selections permanent:

    

16. This completes the section on server-level statistics. The available statistics are limited but generally these statistics are the most important for 80% or more of monitoring situations. For more detailed monitoring, use monitoring products such as Instana.

17. Next, you will view application-level statistics. Click on Applications on the left side:

    

18. Click on the daytrader box title:

    

19. Click on the Monitor button on the left:

    

20. By default, there are boxes for Request Count which is the total number of HTTP requests, and Average Response Time (ns) which is the average response time in nanoseconds.

    

21. For each box, click on the dropdown and select Show Legend to show the legends:

    

22. As with server-level statistics, available application-level statistics in the Admin Center are limited but generally these statistics are the most important for 80% or more of monitoring situations. For more detailed monitoring, use monitoring products such as Instana.

Thread dumps are snapshots of process activity, including the thread stacks that show what each thread is doing. Thread dumps are one of the best places to start to investigate problems. If a lot of threads are in similar stacks, then that behavior might be an issue or a symptom of an issue.

For IBM Java or IBM Semeru Runtimes, a thread dump is also called a javacore or javadump. HotSpot-based thread dumps are covered elsewhere.

This exercise will demonstrate how to gather and review thread dumps in the free IBM Thread and Monitor Dump Analyzer (TMDA) tool.

Thread dumps may be gathered in many different ways. This lab will demonstrate both the IBM performance, hang, and high CPU MustGather as well as the Liberty server dump tool.

An IBM Java or IBM Semeru Runtimes thread dump is generated in a javacore*.txt file with a snapshot of process activity, including:

• Each Java thread and its stack.
• A list of all Java synchronization monitors, which thread owns each monitor, and which threads are waiting for the lock on a monitor.
• Environment information, including Java command line arguments and operating system ulimits.
• Java heap usage and information about the last few garbage collections.
• Detailed native memory and classloader information.

Thread dumps generally do not contain sensitive information about user requests, but they may contain sensitive information about the application or environment such as host names and environment variables, so they should be treated sensitively. In general, a thread dump is cheap (pauses the JVM for only a few hundred milliseconds), small (generally less than a few MB), and low risk.

Liberty offers a server dump utility which gathers useful state about the Liberty process, logs, configuration and, optionally, requests a thread dump.

This lab demonstrates how to gather a server dump and review its output.

1. Open a terminal on the lab image; for example, double click on Xfce Terminal on the Desktop.
2. Run the following command to start the Liberty server dump:

   /opt/ibm/wlp/bin/server dump defaultServer --include=thread
   

3. Once the command completes, it shows the path to the dump ZIP; for example:

   Dumping server defaultServer.
   Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/defaultServer.dump-23.03.22_14.13.16.zip.
   

4. Move the dump ZIP to your current directory:

   mv /opt/ibm/wlp/output/defaultServer/*dump*zip .
   

5. Unzip the dump ZIP:

   unzip *dump*zip
   

6. List the current directory with ls -l to find the dump output; in particular, there is a javacore*txt file which was taken due to --include=thread, Liberty logs under logs/, Liberty configuration such as server.xml, and Liberty introspections under the dump_* subdirectory.
7. The server dump ZIP file is generally what you would send to your development team to review a potential problem, or upload it to an IBM Support case.
8. The next lab will demonstrate how to analyze the javacore*txt thread dump files in more detail.

IBM WebSphere Support provides a script called linperf.sh as part of the document, "MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on Linux" (similar scripts exist for other operating systems such as AIX, z/OS, and Windows). This script will be used to gather thread dumps in this lab. Such a script should be pre-installed on all machines where you run Liberty and it should be run when you have performance or hang issues and the resulting files should be uploaded if you open a support case with IBM. This script is complementary to the Liberty server dump as the server dump also gathers Liberty logs, configuration, and introspections. It's best to run both during an issue.

The linperf.sh script is pre-installed in the lab image at /opt/linperf/linperf.sh.

We will gather and review thread dumps:

Note: You may skip the data collection steps 1-8 and start at step 9 with example data packaged at /opt/webspherelab/supplemental/exampledata/liberty/linperf/

1. Start JMeter

2. Open a terminal on the lab image; for example, double click on Xfce Terminal on the Desktop.

3. First, we'll need to find the PID(s) of Liberty. There are a few ways to do this, and you only need to choose one method:

   1. Show all processes (ps -elf), search for the process using something unique in its command line (grep defaultServer), exclude the search command itself (grep -v grep), and then select the fourth column (in bold below):

      $ ps -elf | grep defaultServer | grep -v grep
      4 S was       1567     1 99  80   0 - 802601 -     19:26 pts/1    00:03:35 java -javaagent:/opt/ibm/wlp/bin/tools/ws-javaagent.jar -Djava.awt.headless=true -Xshareclasses:name=liberty,nonfatal,cacheDir=/output/.classCache/ -jar /opt/ibm/wlp/bin/tools/ws-server.jar defaultServer
      

   2. Search for the process using something unique in its command line using pgrep -f:

      $ pgrep -f defaultServer
      1567
      

4. Execute the linperf.sh command and pass the PID gathered above (replace 1567 with your PID from the output above):

   $ /opt/linperf/linperf.sh 1567
   Tue Apr 23 19:29:26 UTC 2019 MustGather>> linperf.sh script starting [...]
   

5. Wait for 4 minutes for the script to finish before continuing to the next step:

   [...]
   Tue Apr 23 19:33:33 UTC 2019 MustGather>> linperf.sh script complete.
   Tue Apr 23 19:33:33 UTC 2019 MustGather>> Output files are contained within ---->   linperf_RESULTS.tar.gz.   <----
   Tue Apr 23 19:33:33 UTC 2019 MustGather>> The javacores that were created are NOT included in the linperf_RESULTS.tar.gz.
   Tue Apr 23 19:33:33 UTC 2019 MustGather>> Check the <profile_root> for the javacores.
   Tue Apr 23 19:33:33 UTC 2019 MustGather>> Be sure to submit linperf_RESULTS.tar.gz, the javacores, and the server logs as noted in the MustGather.
   

6. As mentioned at the end of the script output, the resulting linperf_RESULTS.tar.gz does not include the thread dumps. Move them over to the current directory:

   mv /opt/ibm/wlp/output/defaultServer/javacore.* .
   

7. Stop JMeter

8. At this point, if you were creating a support case, you would upload linperf_RESULTS.tar.gz, javacore*, and all the Liberty logs; however, instead, we will analyze the results.

9. From the desktop, double click on TMDA.

10. Click Open Thread Dumps and select all of the javacore*.txt files using the Shift key. These will be in your home directory (/home/was) if you moved them above; otherwise, they're in the default working directory (/opt/ibm/wlp/output/defaultServer):
    
    
    
    
    

11. Select a thread dump and click the Thread Detail button:
    
    

12. Click on the Stack Depth column to sort by thread stack depth in ascending order.

13. Click on the Stack Depth column again to sort again in descending order:
    
    

14. Generally, the threads of interest are those with stack depths greater than ~20. Select any such rows and review the stack on the right (if you don't see any, then close this thread dump and select another from the list):
    
    

    1. Generally, to understand which code is driving the thread, skip any non-application stack frames. In the above example, the first application stack frame is TradeAction.getQuote.

    2. Thread dumps are simply snapshots of activity, so just because you capture threads in some stack does not mean there is necessarily a problem. However, if you have a large number of thread dumps, and an application stack frame appears with high frequency, then this may be a problem or an area of optimization. You may send the stack to the developer of that component for further research.

15. In some cases, you may see that one thread is blocked on another thread. For example:
    
    

    1. The Monitor line shows which monitor this thread is waiting for, and the stack shows the path to the request for the monitor. In this example, the application is trying to commit a database transaction. This lab uses the Apache Derby database engine which is not a very scalable database. In this example, optimizing this bottleneck may not be easy and may require deep Apache Derby expertise.

    2. You may click on the thread name in the Blocked by view to quickly see the thread stack of the other thread that owns the monitor.

    3. Lock contention is a common cause of performance issues and may manifest with poor performance and low CPU usage.

16. An alternative way to review lock contention is by selecting a thread dump and clicking Monitor Detail:
    
    
    
    

    1. This shows a tree view of the monitor contention which makes it easier to explore the relationships and number of threads contending on monitors. In the above example, Default Executor-thread-153 owns the monitor and Default Executor-thread-202 is waiting for the monitor.

17. You may also select multiple thread dumps and click the Compare Threads button to see thread movement over time:
    
    
    
    

    1. Each column is a thread dump and shows the state of each thread (if it exists in that thread dump) over time. Generally, you're interested in threads that are runnable (Green Arrow) or blocked or otherwise in the same concerning top stack frame. Click on each cell in that row and review the thread dump on the right. If the thread dump is always in the same stack, this is a potential issue. If the thread stack is changing a lot, then this is usually normal behavior.

    2. In general, focus on the main application thread pools such as DefaultExecutor, WebContainer, etc.

Next, let's simulate a hung thread situation and analyze the problem with thread dumps:

Note: You may skip the data collection steps and use example data packaged at /opt/webspherelab/supplemental/exampledata/liberty/threaddump_deadlock/

1. Open a browser to: http://localhost:9080/swat/

2. Scroll down and click on Deadlocker:
   
   

3. Wait until the continuous browser output stops writing new lines of "Socrates [...]" which signifies that the threads have become deadlocked and then gather a thread dump of the WAS process by sending it the SIGQUIT (3) signal. Although the name of the signal includes the word "QUIT", the signal is captured by the JVM, the JVM pauses for a few hundred milliseconds to produce the thread dump, and then the JVM continues. This same command is performed by linperf.sh. It is a quick and cheap way to quickly understand what your JVM is doing:

   kill -3 $(pgrep -f defaultServer)
   

   1. Note that here we are using a sub-shell to send the output of the pgrep command (which finds the PID of WAS) as the argument for the kill command.

   2. This can be simplified even further with the pkill command which combines pgrep functionality:

      pkill -3 -f defaultServer
      

4. In the TMDA tool, clear the previous list of thread dumps:
   
   

5. Click File > Open Thread Dumps and navigate to /opt/ibm/wlp/output/defaultServer and select both new thread dumps and click Open:
   
   

6. When you select the first thread dump, TMDA will warn you that a deadlock has been detected:
   
   

   1. Deadlocks are not common and mean that there is a bug in the application or product.

7. Use the same procedure as above to review the Monitor Details and Compare Threads to find the thread that is stuck. In this example, the DefaultExecutor application thread actually spawns threads and waits for them to finish, so the application thread is just in a Thread.join:
   
   

8. The actual spawned threads are named differently and show the blocking:
   
   

Next, let's simulate a thread that is using a lot of CPU:

Note: You may skip the data collection steps and use example data packaged at /opt/webspherelab/supplemental/exampledata/liberty/threaddump_infiniteloop/

1. Go to: http://localhost:9080/swat/

2. Scroll down and click on InfiniteLoop:
   
   

3. Go to the container terminal and start top -H with a 10 second interval:

   top -H -d 10
   

   

4. Notice that a single thread is consistently consuming ~100% of a single CPU thread.

5. Convert the PID to hexadecimal. In the example above, 22129 = 0x5671.

   1. In the container, open Galculator:
      
      

   2. Click View > Scientific Mode:
      
      

   3. Enter the decimal number (in this example, 22129), and then click on HEX:
      
      

   4. The result is 0x5671:

      

6. Take a thread dump of the parent process:

   pkill -3 -f defaultServer
   

7. Open the most recent thread dump from /opt/ibm/wlp/output/defaultServer/ in a text editor such as mousepad:
   
   

8. Search for the native thread ID in hex (in this example, 0x5671) to find the stack trace consuming the CPU (if captured during the thread dump):
   
   

9. Finally, kill the server destructively (kill -9) because trying to stop it gracefully will not work due to the infinitely looping request:

   pkill -9 -f defaultServer
   

10. Restart the Liberty server for the next lab:

    /opt/ibm/wlp/bin/server start defaultServer
    

Garbage collection (GC) automatically frees unused objects. Healthy garbage collection is one of the most important aspects of Java programs. The proportion of time spent in garbage collection versus application time should be less than 10% and ideally less than 1%.

This lab will demonstrate how to enable verbose garbage collection in WAS for the sample DayTrader application, exercise the application using Apache JMeter, and review verbose garbage collection data in the free IBM Garbage Collection and Memory Visualizer (GCMV) tool.

All major Java Virtual Machines (JVMs) are designed to work with a maximum Java heap size. When the Java heap is full (or various sub-heaps), an allocation failure occurs and the garbage collector will run to try to find space. Verbose garbage collection (verbosegc) prints detailed information about each one of these allocation failures.

Always enable verbose garbage collection, including in production (benchmarks show an overhead of less than 1% for IBM Java), using the options to rotate the verbosegc logs. For IBM Java - 5 historical files of roughly 20MB each:

-Xverbosegclog:verbosegc.%seq.log,5,50000

Add the verbosegc option to the jvm.options file:

Note: You may skip the data collection steps and use example data packaged at /opt/webspherelab/supplemental/exampledata/liberty/verbosegc_and_oom/

1. Stop JMeter if it is started.

2. WebSphere Liberty:

   1. Stop the Liberty server.

      /opt/ibm/wlp/bin/server stop defaultServer
      

   2. Open a text editor such as mousepad and add the following line to it:

      -Xverbosegclog:logs/verbosegc.%seq.log,5,50000
      

   3. Save the file to /opt/ibm/wlp/usr/servers/defaultServer/jvm.options
      
      

   4. Start the Liberty server

      /opt/ibm/wlp/bin/server start defaultServer
      

3. Start JMeter

4. Run the test for about 5 minutes.

5. Stop JMeter

6. From the desktop, double click on GCMV:

7. Click File > Load File... and select the verbosegc.001.log file. For example:
   
   

8. Select /opt/ibm/wlp/output/defaultServer/logs/verbosegc.001.log
   
   

9. Once the file is loaded, you will see the default line plot view. It is common to change the X-axis to date to see absolute timestamps:
   
   

10. Click the Data Selector tab in the top left, choose VGC Pause and check Total pause time to add the total garbage collection pause time plot to the graph:
    
    

11. Do the same as above using VGC Heap and check Used heap (after global collection):
    
    
    

12. Observe the heap usage and pause time magnitude and frequency over time. For example:
    
    

    1. This shows that the heap size reaches 145MB and the heap usage (after global collection) reached ~80MB.

13. More importantly, we want to know the proportion of time spent in GC. Click the Report tab and review the Proportion of time spent in garbage collection pauses (%):
    
    

    1. If this number is less than 1%, then this is very healthy. If it's less than 5% then it's okay. If it's less than 10%, then there is significant room for improvement. If it's greater than 10%, then this is concerning.

Next, let's simulate a memory issue.

Note: You may skip the data collection steps and use example data packaged at /opt/webspherelab/supplemental/exampledata/liberty/verbosegc_and_oom/

1. Stop JMeter if it is started.

2. Liberty:

   1. Stop Liberty:

      /opt/ibm/wlp/bin/server stop defaultServer

   2. Edit /opt/ibm/wlp/usr/servers/defaultServer/jvm.options, add an explicit maximum heap size of 256MB on a new line and save the file:

      -Xmx256m

      

   3. Start Liberty

      /opt/ibm/wlp/bin/server start defaultServer

3. Start JMeter

4. Let the JMeter test run for about 5 minutes.

5. Do not stop the JMeter test but leave it running as you continue to the next step.

6. Open your browser to the following page: http://localhost:9080/swat/AllocateObject?size=1048576&iterations=300&waittime=1000&retainData=true

   1. This will allocate three hundred 1MB objects with a delay of 1 second between each allocation, and hold on to all of them to simulate a leak.

   2. This will take about 5 minutes to run and you can watch your browser output for progress.

   3. You can run top -H while this is running. As memory pressure builds, you'll start to see GC Slave threads consuming most of the CPUs instead of application threads (garbage collection also happens on the thread where the allocation failure occurs, so you may also see a single application thread consuming a similar amount of CPU as the GC Slave threads):

      top -H -p $(pgrep -f defaultServer) -d 5

      

   4. At some point, browser output will stop because the JVM has thrown an OutOfMemoryError.

7. Stop JMeter

8. Forcefully kill the JVM because an OutOfMemoryError does not stop the JVM; it will continue garbage collection thrashing and consume all of your CPU.

       pkill -9 -f defaultServer
   

9. Close and re-open the verbosegc*log file in GCMV:
   
   

   1. We can quickly see how the heap usage reaches 256MB and the pause time magnitude and durations increase significantly.

10. Click on the Report tab and review the Proportion of time spent in garbage collection pauses (%):
    
    

11. 24% seems pretty bad but not terrible and doesn't line up with what we know about what happened. This is because, by default, the GCMV Report tab shows statistics for the entire duration of the verbosegc log file. Since we had run the JMeter test for 5 minutes and it was healthy, the average proportion of time in GC is lower for the whole duration.

12. Click on the Line plot tab and zoom in to the area of high pause times by using your mouse button to draw a box around those times:
    
    

13. This will zoom the view to that bounding box:
    
    

14. However, zooming in is just a visual aid. To change the report statistics, we need to match the X-axis to the period of interest.

15. Hover your mouse over the approximate start and end points of the section of concern (frequent pause time spikes) and note the times of those points (in terms of your selected X Axis type):
    
    

16. Enter each of the values in the minimum and maximum input boxes and press Enter on your keyboard in each one to apply the value. The tool will show vertical lines with triangles showing the area of the graph that you've cropped to.
    
    

17. Click on the Report tab at the bottom and observe the proportion of time spent in garbage collection for this period is very high (in this example, ~87%).
    
    

18. This means that the application is doing very little work and is very unhealthy. In general, there are a few, non-exclusive ways to resolve this problem:

    1. Increase the maximum heap size.

    2. Decrease the object allocation rate of the application.

    3. Resolve memory leaks through heapdump analysis.

    4. Decrease the maximum thread pool size.

IBM Monitoring and Diagnostics for Java - Health Center is free and shipped with IBM Java 8. Among other things, Health Center includes a statistical CPU profiler that samples Java stacks that are using CPU at a very high rate to determine what Java methods are using CPU. Health Center generally has an overhead of less than 1% and is suitable for production use. In recent versions, it may also be enabled dynamically without restarting the JVM.

This lab will demonstrate how to enable Java Health Center, exercise the sample DayTrader application using Apache JMeter, and review the Health Center file in the IBM Java Health Center Client Tool.

The Health Center agent gathers sampled CPU profiling data, along with other information:

• Classes: Information about classes being loaded

• Environment: Details of the configuration and system of the monitored application

• Garbage collection: Information about the Java heap and pause times

• I/O: Information about I/O activities that take place.

• Locking: Information about contention on inflated locks

• Memory: Information about the native memory usage

• Profiling: Provides a sampling profile of Java methods including call paths

The Health Center agent can be enabled in two ways:

1. At startup by adding -Xhealthcenter:level=headless to the JVM arguments

2. At runtime, by running ${IBM_JAVA}/bin/java -jar ${IBM_JAVA}/jre/lib/ext/healthcenter.jar ID=${PID} level=headless

Note: For both items, you may add the following arguments to limit and roll the total file usage of Health Center data:

-Dcom.ibm.java.diagnostics.healthcenter.headless.files.max.size=BYTES
-Dcom.ibm.java.diagnostics.healthcenter.headless.files.to.keep=N (N=0 for unlimited)

The key to produce the final Health Center HCD file is that the JVM should be gracefully stopped (there are alternatives to this by packaging the temporary files but this isn't generally recommended).

Consider always enabling HealthCenter in headless mode for post-mortem debugging of issues.

Note: You may skip the data collection steps and use example data packaged at /opt/webspherelab/supplemental/exampledata/liberty/healthcenter/

1. Stop JMeter if it is started.

2. Add Health Center arguments to the JVM:

   1. Add the following line to /opt/ibm/wlp/usr/servers/defaultServer/jvm.options:

      -Xhealthcenter:level=headless

3. Stop the server:

   /opt/ibm/wlp/bin/server stop defaultServer
   

4. Start the server

   /opt/ibm/wlp/bin/server start defaultServer
   

5. Start JMeter and run it for 5 minutes.

6. Stop JMeter

7. Stop WAS as in step 3 above.

8. From the desktop, double click on HealthCenter.

9. Click File > Load Data... (note that it's towards the bottom of the File menu; Open File does not work):
   
   

10. Select the healthcenter*.hcd file from /opt/ibm/wlp/output/defaultServer:
    

11. Wait for the data to complete loading:
    
    

12. Click on CPU:
    

13. Review the overall system and Java application CPU usage:
    

14. Right click anywhere in the graph and change the X-axis to date (which changes all other views to date as well):
    

    1. For large Health Center captures, this may take significant time to change and there is no obvious indication when it's complete. The best way to know is when the CPU usage of Health Center drops to a low amount.

15. Click Data > Crop data...
    
    

16. Change the Start time and End time to match the period of interest. For example, usually you want to exclude the start-up time of the process and only focus on user activity:
    

17. Click Window > Preferences:
    

18. Check the Show package names box under Health Center > Profiling and press OK so that we can see more details in the profiling view:
    

19. Click on Method profiling to review the CPU sampling data:
    

20. The Method profiling view will show CPU samples by method:
    

21. The Self (%) column reports the percent of samples where a method was at the top of the stack. The Tree (%) column reports the percent of samples where a method was somewhere else in the stack. Make sure to check that the Samples column is at least in the hundreds or thousands; otherwise, the CPU usage is likely not that high or a problem did not occur. The Self and Tree percentages are a percent of samples, not of total CPU.

22. Any methods over ~1% are worthy of considering how to optimize or to avoid. For example, ~2% of samples were in method 0x2273c68 (for various reasons, some methods may not resolve but you can usually figure things out from the invocation paths). Selecting that row and switching to the Invocation Paths view shows the percent of samples leading to those calls:
    

    1. In the above example, 63.11% of samples (i.e. of 2.9% of total samples) were invoked by org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.doCommit.

23. If you sort by Tree %, skip the framework methods from Java and WAS, and find the first application method. In this example, about 32% of total samples was consumed by com.ibm.websphere.samples.daytrader.web.TradeAppServlet.performTask and all of the methods it called. The Called Methods view may be further reviewed to investigate the details of this usage; in this example, doPortfolio drove most of the CPU samples.
    
    

The Liberty HTTP access log is optionally enabled with the httpEndpoint accessLogging element. When enabled, a separate access.log file is produced with an NCSA standardized (i.e. httpd-style) line for each HTTP request, including items such as the URI and response time, useful for post-mortem corelation and performance analysis.

1. Modify /config/server.xml to change the following element:

   <httpEndpoint id="defaultHttpEndpoint"
              host="*"
              httpPort="9080"
              httpsPort="9443" />
   

2. To the following (note that the <httpEndpoint> element must no longer be self-closing):

   <httpEndpoint id="defaultHttpEndpoint"
              host="*"
              httpPort="9080"
              httpsPort="9443">
     <accessLogging filepath="${server.output.dir}/logs/access.log" maxFileSize="250" maxFiles="2" logFormat="%h %i %u %t &quot;%r&quot; %s %b %D %{R}W" />
   </httpEndpoint>
   

3. Start JMeter
4. Run the test for a couple of minutes.
5. Stop JMeter
6. Review /opt/ibm/wlp/output/defaultServer/logs/access.log to see HTTP responses. For example:

   127.0.0.1 - Admin1 [20/Mar/2022:16:28:23 +0000] "GET /daytrader/app?action=portfolio HTTP/1.1" 200 12927 397114 390825
   127.0.0.1 - Admin1 [20/Mar/2022:16:28:23 +0000] "GET /daytrader/app?action=quotes&symbol=s%3A6651 HTTP/1.1" 200 7849 528280 526508
   127.0.0.1 - Admin1 [20/Mar/2022:16:28:23 +0000] "GET /daytrader/app?action=quotes&symbol=s%3A9206 HTTP/1.1" 200 7849 528556 526676
   

7. The second-to-last number is the response time in microseconds. In the example above, the first response time was 397.114 milliseconds. The last number is the time until the first byte of the response was sent back which may help investigate network slowdowns in front of WebSphere.

There are various scripts and tools available publicly (example) to post-process NCSA-style access logs to create statistics and graphs.

In general, it is a good practice to use accessLogging, even in production, if the performance overhead is acceptable.

Consider reviewing other common performance tuning and troubleshooting operations and capabilities:

• Review performance tuning guidance and the WebSphere Performance Cookbook.
• A key differentiator of WebSphere Liberty is its auto-tuning main thread pool that dynamically changes the maximum thread pool size to try to maximize throughput. In most cases, this means that an administrator does not need to specify nor tune the main application thread pool. Note that other pools such as database connection pools, web service client pools and others do not auto-tune as those are backend resources out of the control of Liberty, so they must still be tuned.
• MicroProfile Metrics provides the ability to produce the same statistics as monitor-1.0 but exposed through a Prometheus-style /metrics HTTP endpoint for consumption by Prometheus or custom scripts.
• Distributed tracing to monitor the flow of JAX-RS web service calls through a web of Liberty processes.
• WebSphere Liberty has a rich set of configuration capabilites through XML files and variables (server.xml, etc.), Java options files (jvm.options), and environment variables (server.env or through the process environment). Most WebSphere Liberty configuration changes may be done dynamically, by default.
• An optional servlet response cache that takes significant effort to configure but may have a dramatic return on investment in reduced server utilization and improved response times.
• If using collectives:
  • Consider grouping Liberty servers running the same application into Liberty clusters for easier management.
  • Consider the auto scaling features to dynamically scale the number of Liberty servers based on CPU and memory.
  • Dynamic routing rules in a collective may be used to route requests to specific servers, redirect requests, or reject requests. This includes the capability of handling multiple editions of an application and routing a percentage of requests to certain editions, for example.
  • Consider using maintenance mode for clusters during diagnostics or maintenance operations.
  • Consider using health policies to capture diagnostics and perform other operations based on different conditions.
• The most common performance tuning that needs to be done is to change the maximum Java heap size (-Xmx or -XX:MaxRAMPercentage), the maximum Java nursery size (-Xmn), and maximum pool sizes such as database connection pools.
• Flexible diagnostic trace capabilities with the option of using a binary output format for reduced overhead.
• Consider integrating the WebSphere Automation product that helps with server inventory, security patching, and automatic memory leak detection and diagnosis.
• If running WebSphere Liberty on IBM Java or IBM Semeru Runtimes:
  • A shared class cache to improve startup time.
  • Various garbage collection policies for different workloads.
  • Method tracing to take diagnostic actions on method entry or exit, or to time method calls.
  • A dump engine to gather diagnostics on various events such as large object allocations, thrown or caught exceptions, etc.
  • A JITServer to separate Just-In-Time compilation into a separate process and share compiled code across processes.
  • For richer memory analysis, consider enabling and configuring core dumps (e.g. core and file ulimits, kernel.core_pattern truncation settings, etc.) after reviewing the security, disk and performance risks.
• Gather basic operating system metrics such as CPU, memory, disk, and network utilization, saturation, and errors.
• Gather operating system logs and watch for warnings and errors.
• Monitor for common network issues such as TCP retransmissions.
• Monitor the web server's access and error logs for warnings and errors.
• Monitor the web server's utilization with tools such as mod_mpmstats and mod_status.
• For newer applications, advanced capabilities for fault tolerance such as automatic retries, circuit breakers, fallbacks, and bulkheads. In addition, health checks may be enabled using readiness and liveness probes.
• When running in a container environment such as OpenShift:
  • Consider deploying applications using the WebSphere Liberty Operator and use capabilities such as the WebSphereLibertyTrace and WebSphereLibertyDump custom resources.
  • Enable JSON logging and publish native logs of pods to OpenShift centralized logging using, most commonly, EFK, and then search logs in the Kibana log viewer. Optionally install sample Kibana dashboards that summarize application log events and statistics.
  • Consider enabling application monitoring integrated with and Grafana.
  • If you have cluster-admin permissions, use the MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on Linux on Containers during performance, hang, and high-CPU issues.
• You may connect the JConsole monitoring tool built into Java and access Liberty enabled with the monitor-1.0 feature through the localConnector-1.0 and/or restConnector-2.0 features; however, note that JConsole has connection complexities and limited capabilities and Admin Center is often enough for basic statistics visualization.

The following optional sections review general methodology topics.

1. Observe and measure evidence of the problem. For example: "Users are receiving HTTP 500 errors when visiting the website."

2. Create prioritized hypotheses about the causes of the problem. For example: "I found exceptions in the logs. I hypothesize that the exceptions are creating the HTTP 500 errors."

3. Research ways to test the hypotheses using experiments. For example: "I searched the documentation and previous problem reports and the exceptions may be caused by a default setting configuration. I predict that changing this setting will resolve the problem if this hypothesis is true."

4. Run experiments to test hypotheses. For example: "Please change this setting and see if the user errors are resolved."

5. Observe and measure experimental evidence. If the problem is not resolved, repeat the steps above; otherwise, create a theory about the cause of the problem.

Keep track of a summary of the situation, a list of problems, hypotheses, and experiments/tests. Use numbered items so that people can easily reference things in phone calls or emails. The summary should be restricted to a single sentence for problems, resolution criteria, statuses, and next steps. Any details are in the subsequent tables. The summary is a difficult skill to learn, so try to constrain yourself to a single (short!) sentence. For example:

1. Problems: 1) Average website response time of 5000ms and 2) website error rate > 10%.

2. Resolution criteria: 1) Average response time of 300ms and 2) error rate of <= 1%.

3. Statuses: 1) Reduced average response time to 2000ms and 2) error rate to 5%.

4. Next steps: 1) Investigate database response times and 2) gather diagnostic trace.

1. Performance tuning is usually about focusing on a few key variables. We will highlight the most common tuning knobs that can often improve the speed of the average application by 200% or more relative to the default configuration. The first step, however, should be to use and be guided by the tools and methodologies. Gather data, analyze it and create hypotheses: then test your hypotheses. Rinse and repeat. As Donald Knuth says: "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time [...]. Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail." (Donald Knuth, Structured Programming with go to Statements, Stanford University, 1974, Association for Computing Machinery)

2. There is a seemingly daunting number of tuning knobs. Unless you are trying to squeeze out every last drop of performance, we do not recommend a close study of every tuning option.

3. In general, we advocate a bottom-up approach. For example, with a typical WebSphere Application Server application, start with the operating system, then Java, then WAS, then the application, etc. Ideally, investigate these at the same time. The main goal of a performance tuning exercise is to iteratively determine the bottleneck restricting response times and throughput. For example, investigate operating system CPU and memory usage, followed by Java garbage collection usage and/or thread dumps/sampling profilers, followed by WAS PMI, etc.

4. One of the most difficult aspects of performance tuning is understanding whether or not the architecture of the system, or even the test itself, is valid and/or optimal.

5. Meticulously describe and track the problem, each test and its results.

6. Use basic statistics (minimums, maximums, averages, medians, and standard deviations) instead of spot observations.

7. When benchmarking, use a repeatable test that accurately models production behavior, and avoid short term benchmarks which may not have time to warm up.

8. Take the time to automate as much as possible: not just the testing itself, but also data gathering and analysis. This will help you iterate and test more hypotheses.

9. Make sure you are using the latest version of every product because there are often performance or tooling improvements available.

10. When researching problems, you can either analyze or isolate them. Analyzing means taking particular symptoms and generating hypotheses on how to change those symptoms. Isolating means eliminating issues singly until you've discovered important facts. In general, we have found through experience that analysis is preferable to isolation.

11. Review the full end-to-end architecture. Certain internal or external products, devices, content delivery networks, etc. may artificially limit throughput (e.g. Denial of Service protection), periodically mark services down (e.g. network load balancers, WAS plugin, etc.), or become saturated themselves (e.g. CPU on load balancers, etc.).

Review the common tuning of Liberty and underlying components recipe in the IBM WebSphere Performance Tuning Cookbook.

The podman run or docker run command in this lab does not use the -d (daemon) flag which means that it runs in the foreground. To stop such a container, use one of the following methods:

1. Hold down the Control/Ctrl key on your keyboard and press C in the terminal window where the run command is running.
2. In a separate terminal window, find the container ID with podman ps or docker ps command and then run podman stop $ID or docker stop $ID.

If you add -d to podman run or docker run, then to view the standard out logs, find the container ID with podman ps or docker ps and then run podman logs $ID or docker logs $ID. To stop, use podman stop $ID or docker stop $ID.

In a separate command prompt or terminal window, execute one of the following commands depending on whether you're using podman or Docker Desktop:

• podman exec -u was -it $(podman ps -q) bash
• docker exec -u was -it $(podman ps -q) bash

Windows requires extra steps to configure remote desktop to connect to a container:

1. Open PowerShell as Administrator:
   
   

2. Run ipconfig and copy the IPv4 address of the WSL adapter. For example, 172.24.0.1:
   
   

3. Run the following command in PowerShell:

   New-NetFirewallRule -Name "myContainerRDP" -DisplayName "RDP Port for connecting to Container" -Protocol TCP -LocalPort @(3390) -Action Allow

4. Run Remote Desktop
   
   

5. Enter the WSL IP address (for example, 172.24.0.1) followed by :3390 as Computer and click Connect:
   
   

6. You'll see a certificate warning because of the name mismatch. Click Yes to connect:
   
   

7. Type username = was and password = websphere
   
   

8. You should now be remote desktop'ed into the container:
   
   

9. Notes:

   1. In some cases, only the Remote Desktop Connection application worked, and not Remote Desktop:
      
      
   2. Microsoft requires the above steps and the use of port 3390 instead of directly connecting to 3389.

There are many different versions and types of Java in the image. To list them, run:

$ alternatives --display java | grep "^/"
/usr/lib/jvm/java-11-openjdk-11.0.14.1.1-5.fc35.x86_64/bin/java - family java-11-openjdk.x86_64 priority 11001421
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-6.fc35.x86_64/jre/bin/java - family java-1.8.0-openjdk.x86_64 priority 1
/opt/ibm/java/bin/java - family ibmjava priority 99999999
/opt/openjdk8_hotspot/jdk/bin/java - family openjdk priority 89999999
/opt/openjdk11_hotspot/jdk/bin/java - family openjdk priority 89999999
/opt/openjdk17_hotspot/jdk/bin/java - family openjdk priority 89999999
/opt/openjdk8_ibm/jdk/bin/java - family openjdk priority 89999999
/opt/openjdk11_ibm/jdk/bin/java - family openjdk priority 89999999
/opt/openjdk17_ibm/jdk/bin/java - family openjdk priority 89999999

To change the Java that is on the path, run the following command and enter the number of the Java that you wish to change to and press Enter. The directory name tells you the type of Java; for example, OpenJ9 or HotSpot. The directory name also tells you the version of Java (for example, openjdk17 is Java 17).

$ sudo alternatives --config java
sudo: unable to send audit message: Operation not permitted

There are 9 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           java-11-openjdk.x86_64 (/usr/lib/jvm/java-11-openjdk-11.0.14.1.1-5.fc35.x86_64/bin/java)
   2           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-6.fc35.x86_64/jre/bin/java)
*+ 3           ibmjava (/opt/ibm/java/bin/java)
   4           openjdk (/opt/openjdk8_hotspot/jdk/bin/java)
   5           openjdk (/opt/openjdk11_hotspot/jdk/bin/java)
   6           openjdk (/opt/openjdk17_hotspot/jdk/bin/java)
   7           openjdk (/opt/openjdk8_ibm/jdk/bin/java)
   8           openjdk (/opt/openjdk11_ibm/jdk/bin/java)
   9           openjdk (/opt/openjdk17_ibm/jdk/bin/java)

Enter to keep the current selection[+], or type selection number: 9
$ java -version
openjdk version "17.0.2" 2022-01-18
IBM Semeru Runtime Open Edition 17.0.2.0 (build 17.0.2+8)
Eclipse OpenJ9 VM 17.0.2.0 (build openj9-0.30.0, JRE 17 Linux amd64-64-Bit Compressed References 20220128_115 (JIT enabled, AOT enabled)
OpenJ9   - 9dccbe076
OMR      - dac962a28
JCL      - 64cd399ca28 based on jdk-17.0.2+8)

The alternatives command has the concept of groups of commands so when you change Java using the method above, other commands like jar, javac, etc. also change.

Any currently running Java programs will need to be restarted if you want them to use the different version of Java (WAS traditional is an exception because it uses a bundled version of Java).

The clipboard may not be shared when using VNC. There are two workarounds:

1. From within the VNC session, open the PDF of the lab from the desktop and use that instead. Then you can copy/paste within the lab.
2. For command line steps, start a new command prompt or terminal and execute one of the following commands depending on whether you're using podman or Docker Desktop:
   • podman exec -u was -it $(podman ps -q) bash
   • docker exec -u was -it $(podman ps -q) bash

This is a known issue that will be fixed on the next build of the lab. Until then, the workaround is:

1. Start a new command prompt or terminal and execute one of the following commands depending on whether you're using podman or Docker Desktop:
   • podman exec -u root -it $(podman ps -q) bash
   • docker exec -u root -it $(podman ps -q) bash
2. Then execute the following command:

   sed -i '1 i\auth sufficient pam_succeed_if.so user = was' /etc/pam.d/xfce4-screensaver
   

3. Try unlocking the screen again.

Thank you to those that helped build and test this lab:

• Hiroko Takamiya
• Andrea Pichler
• Kazuma Tanabe
• Shinichi Kako

For the full lab, see https://github.com/IBM/webspherelab/blob/master/WAS_Troubleshooting_Perf_Lab.md