Oracle Exalogic Elastic Cloud Administration: D71299GC20 Edition 2.0 May 2011 D73135

Oracle Exalogic Elastic Cloud
Administration
Student Guide
D71299GC20
Edition 2.0
May 2011
D73135
Author Copyright © 2011, Oracle and/or it affiliates. All rights reserved.
TJ Palazzolo Disclaimer
This document contains proprietary information and is protected by copyright and

Technical Contr ibutors other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered
and Reviewer s in any way. Except where your use constitutes "fair use" under copyright law, you
James Bayer may not use, share, download, upload, copy, print, display, perform, reproduce,
publish, license, post, transmit, or distribute this document in whole or in part without
Rick Eppstein the express authorization of Oracle.
Mike Palmeter
The information contained in this document is subject to change without notice. If you
Dave Winter find any problems in the document, please report them in writing to: Oracle University,
500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
Editor s
Restricted Rights Notice
Richard Wallis
Smita Kommini If this documentation is delivered to the United States Government or anyone using
the documentation on behalf of the United States Government, the following notice is
applicable:
Publisher s
U.S. GOVERNMENT RIGHTS
Shaik Basha The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or
Sumesh Koshy disclose these training materials are restricted by the terms of the applicable Oracle
license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names
may be trademarks of their respective owners.
Contents
1 Course Overview
Course Objectives 1-2
Target Audience 1-3
Introductions 1-4
Course Schedule 1-5
Course Practices 1-7
Classroom Guidelines 1-8
For More Information 1-9
Oracle by Example (OBE) 1-10
Related Training 1-11
2 The Exalogic Solution

Objectives 2-2
Why Data Centers? 2-3
Data Center Challenges 2-4
What Is Exalogic? 2-5
Why Exalogic? 2-7
What Is InfiniBand? 2-8
An Engineered Solution 2-9
Java Performance 2-10
High Availability 2-11
Multi-Tenancy 2-12
Software Compatibility 2-13
Selecting an Operating System 2-14
Licensing and Support Overview 2-15
Quiz 2-16
Summary 2-18
3 Hardware Components
Objectives 3-2
Exalogic Machine Topology 3-3
Compute Nodes 3-4
Compute Node: Front View 3-5
Compute Node: Rear View 3-6
Compute Node Memory 3-7
iii
Storage Appliance 3-8
Disk Caching 3-9
Storage Appliance: Front View 3-10
Storage Appliance: Rear View 3-11
InfiniBand (IB) Switches 3-12
Gateway Switch Hardware Views 3-13
IB Host Channel Adapter (HCA) 3-14
Management Switch 3-15
Scaling Your Data Center 3-16
Scaling Your Data Center 3-17
Hardware Package Details 3-18
Power Options 3-19
Hardware Maintenance 3-20
Quiz 3-21
Summary 3-25
4 Initial Machine Configuration

Objectives 4-2
Power-On Procedure 4-3
Hardware Configuration Process 4-4
Exalogic Online Checklists 4-5
Example: Access Route Checklist 4-6
Site Requirements 4-7
Exalogic Networks 4-8
Network Configuration Worksheets 4-9
Default Addresses for a Full Rack 4-10
Default Host Names 4-11
Installation Spreadsheet 4-12
Exalogic Configuration Utility (ECU) 4-13
ECU Internals 4-14
InfiniBand Networking Concepts 4-15
Bonded Interfaces 4-16
Default Network Configuration 4-17
Exalogic Diagnostic Utilities 4-18
Utility Examples 4-19
Distributed Command Line Interface (DCLI) 4-20
DCLI Examples 4-21
Quiz 4-22
Summary 4-25
Practice 4-1: Connect to a Compute Node 4-26
iv
5 Fusion Middleware Concepts
Objectives 5-2
Middleware Concepts 5-3
Oracle Fusion Middleware (FMW) Review 5-4
WebLogic Server (WLS) Overview 5-5
Java EE Deployment: Example 5-6
WLS Architecture Overview 5-7
WLS Domain Contents 5-8
WLS Machines and Compute Nodes 5-9
Node Manager Concepts 5-10
Transaction Recovery 5-11
Message Recovery 5-12
WLS Cluster Architecture 5-13
WLS Exalogic Topology: Example 5-14
WLS Exalogic Capacity Planning 5-15
Quiz 5-16
Summary 5-19
6 Storage Configuration
Objectives 6-2
ZFS Appliance Capabilities 6-3
Administrative Tasks 6-4
Getting Started 6-5
Accessing the Browser User Interface (BUI) 6-6
Viewing Appliance Status 6-7
Viewing Network Settings 6-9
Some Supported Services 6-10
Configuring File System Services 6-11
Identifying Hardware Components 6-12
Replacing Failed Hardware 6-14
Performing Failover Actions 6-15
Creating Administrative Users 6-16
Roles and Authorizations 6-17
Shared Storage Concepts 6-18
Default Storage Configuration 6-19
Creating a Pool 6-20
Pool Data Profiles 6-21
Creating a Project 6-22
Creating a Share 6-24
Overriding Project Defaults 6-25
Setting Quotas 6-26
v
Restricting Share Access 6-27
Configuration Management 6-28
Mounting an NFS Share 6-29
Storage Command-Line Interface (CLI) 6-30
Some CLI Commands 6-31
CLI Examples 6-32
FMW Storage Topology 6-33
FMW Storage: Example 6-34
Quiz 6-35
Summary 6-38
Practice 6-1 Initialize Shared File Systems 6-39
7 Network Configuration
Objectives 7-2
Exalogic Networks: Review 7-3
IB Networking Concepts: Review 7-4
Default Compute Node Network 7-5
Default Storage Appliance Network 7-6
Quarter-Rack Network Topology 7-7
Half-Rack Network Topology 7-8
Full-Rack Network Topology 7-9
Multi-Rack Topology 7-10
Transceivers and Cables 7-11
Administration Tasks 7-12
Status LEDs 7-13
Replacing Failed Hardware 7-14
IB Switch Command-Line Interface (CLI) 7-15
Monitoring the Hardware 7-16
Monitoring Connection Status 7-17
Running IB Diagnostics 7-18
Subnet Manager (SM) Concepts 7-19
Subnet Manager Recommendations 7-20
Configuring the Subnet Manager 7-21
Fabric Monitor Interface 7-22
Fabric Monitor Examples 7-23
Monitoring IB Connections 7-24
Monitoring Gateway Connections 7-25
Multiple Addresses Versus Multiple Ports 7-26
Configuring a Virtual Address 7-27
Network Configuration and Multi-Tenancy 7-28
Virtual LAN (VLAN) 7-29
vi
Configuring a VLAN 7-30
Subnet Partitioning 7-31
Quiz 7-32
Summary 7-35
Practice 7-1 Configure Network Settings 7-36
8 Creating FMW Domains

Objectives 8-2
WebLogic Concepts: Review 8-3
WebLogic Communication Types 8-4
Installation and Configuration Process 8-5
Shared or Dedicated Installations? 8-6
Available WebLogic Installers 8-7
Installation Overview 8-8
Domain Templates 8-9
Configuration Wizard: Overview 8-10
Shared or Copied Domains? 8-13
Migrating Existing Domains 8-14
Pack and Unpack Tools 8-15
Starting the Administration Server 8-16
WebLogic Console: Overview 8-17
Adding a New Server 8-18
Updating Machine Membership 8-19
Updating Cluster Membership 8-20
Quiz 8-21
Summary 8-23
Practice 8-1 Initialize a WebLogic Domain 8-24
9 Optimizing FMW Domains on Exalogic

Objectives 9-2
Additional Configuration Tasks 9-3
Default WebLogic Networking 9-4
Isolating Client Communication 9-5
Isolating Cluster Communication 9-6
Session Replication: Overview 9-7
Network Channels 9-8
Creating a Channel 9-9
Some Channel Protocols 9-11
Creating a Cluster Replication Channel 9-12
Using SDP for Replication 9-13
Using Multiple Ports for Replication 9-14
vii
Other Replication Optimizations 9-15
Other WebLogic Optimizations 9-16
Enabling Other Optimizations 9-17
Licensing Review 9-18
Quiz 9-19
Summary 9-21
Practice 9-1 Configure Network Channels and Optimizations 9-22
10 Starting and Migrating Servers

Objectives 10-2
Node Manager: Review 10-3
Node Manager: Architecture 10-4
Node Manager: Configuration Tasks 10-5
Recommended NM Storage Configuration 10-6
Basic Node Manager Properties 10-7
Enrolling a NM with a Domain 10-8
Starting Servers from the Console 10-9
Starting Servers from the Command Line 10-10
Starting NM on Machine Startup 10-11
WebLogic Logs 10-12
Practice 10-1 Start Servers with the Node Manager 10-13
Whole-Server Migration 10-14
Automatic Server Migration: Example 10-15
Leasing Types 10-16
Migration Configuration Tasks 10-17
Node Manager Network Properties 10-18
Configuring Cluster Leasing 10-19
Database Leasing Schema 10-20
Candidate Machines 10-21
Machine Failback 10-22
Manual Server Migration 10-23
Migrating a Server 10-24
Pinned Server Resources 10-25
Recommended Storage Configuration for JMS and Transaction Recovery 10-26
Configuring a Server's Transaction Store 10-27
Creating a JMS Persistent Store 10-28
Quiz 10-29
Summary 10-32
Practice 10-2 Migrate Failed Servers 10-33
viii
11 Exalogic Database Connectivity
Objectives 11-2
Ethernet Database Connectivity 11-3
Oracle Real Application Clusters (RAC): Overview 11-4
Oracle Exadata: Overview 11-5
Direct InfiniBand Database Connectivity 11-6
Exalogic Quarter Rack and Exadata Quarter Rack 11-7
Exalogic Full Rack and Exadata Full Rack 11-8
JDBC: Overview 11-9
WebLogic Data Source: Overview 11-10
Active GridLink for RAC 11-11
GridLink and FCF 11-12
GridLink and Services 11-13
GridLink and Single Client Access Name (SCAN) 11-14
Creating a GridLink Data Source 11-15
Enabling SDP for Exadata Connectivity 11-16
Multi Data Source: Overview 11-17
Multi Data Source: Architecture 11-18
Licensing Review 11-19
Quiz 11-20
Summary 11-22
Recorded Demonstration: Active GridLink 11-23
12 Additional FMW Infrastructure

Objectives 12-2
Coherence: Overview 12-3
Coherence: Architecture 12-4
Role of Coherence Applications 12-5
Basic Coherence Configuration Tasks 12-6
Creating a Coherence Cluster 12-7
Creating a Coherence Server 12-8
Starting Coherence Servers 12-10
Coherence*Web: Overview 12-11
WLS Cluster: Review 12-12
Proxy Plug-Ins 12-13
Recommended Proxy Topology 12-14
Obtaining and Using Plug-Ins 12-15
Oracle Web Tier: Overview 12-16
OHS Architecture 12-17
Oracle Process Manager and Notification Server (OPMN) 12-18
OPMNCTL: Examples 12-19
ix
Basic Plug-In Parameters 12-20
OHS Plug-In: Example 12-21
Quiz 12-22
Summary 12-24
Practice 12-1: Configure a Cluster Proxy 12-25
13 Managing Hardware with ILOM

Objectives 13-2
Integrated Lights Out Manager (ILOM) 13-3
ILOM Tasks 13-4
Sideband Management 13-5
Accessing the Browser Interface 13-6
Management Settings 13-7
Adding Administrative Users 13-8
ILOM Roles 13-9
Powering the Device On or Off 13-10
Updating Device Firmware 13-11
ILOM Glossary 13-12
Using Indicator LEDs 13-13
Viewing Sensor Readings 13-14
Managing Power Consumption 13-15
Analyzing Power Consumption 13-16
Alert Concepts 13-17
Configuring Alerts 13-18
Command-Line Interface (CLI) Example 13-19
Connecting to the Host OS 13-20
Quiz 13-21
Summary 13-23
Recorded Demonstration: Compute Node ILOM 13-24
x
Course Overview
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Course Objectives
After completing this course, you should be able to:

• Locate and use checklists and worksheets to plan an
installation
• Describe Exalogic's default storage and network setup
• Perform basic storage administration and maintenance
tasks
• Configure a compute node's network to support Fusion
Middleware
• Create and optimize a WebLogic domain on Exalogic
• Start, kill, and migrate servers by using Node Manager
• View device status and health with Oracle Integrated
Lights Out Manager (ILOM)
Oracle Exalogic Elastic Cloud Administration 1 - 2

Target Audience
• This course is intended for data center administrators.

• There are no prerequisite skills, but attendees will benefit
from having a fundamental knowledge of:
– The UNIX command line
– Basic UNIX network and storage concepts
– The Oracle Fusion Middleware product stack
Target Audience
If you are concerned whether your background and experience match that of the target
audience, ask the instructor.

Introductions
• Introduce yourself.
• Tell us about:
– Your company and role
– Any previous experience with Sun hardware
– Any previous Fusion Middleware product experience

Course Schedule
Day Lesson
1 AM 1. Course Overview
2. The Exalogic Solution
3. Hardware Components
PM 4. Initial Machine Configuration
5: Fusion Middleware Concepts
6: Storage Configuration
2 AM 6: Storage Configuration continued
7: Network Configuration
PM 8: Creating FMW Domains
9: Optimizing FMW Domains for Exalogic
Course Schedule
The course schedule might vary depending on the pace of the class. The schedule shown is
an estimate. The instructor may provide updates.

Course Schedule
Day Lesson
3 AM 10: Starting and Migrating Servers
PM 11: Exalogic Database Connectivity

12: Additional FMW Infrastructure
13: Managing Hardware with ILOM
Course Schedule (continued)

The course schedule might vary depending on the pace of the class. The schedule shown is
an estimate. The instructor may provide updates.

Course Practices
• Many topics are reinforced with a hands-on practice.

• Some practices include a scripted solution to aid any
students who encounter problems.
• Each student is assigned a simulated Exalogic
environment that consists of three virtual machines.
“Exalogic”
Compute Node
Compute Node
Storage Appliance

Classroom Guidelines
• The instructor starts each session at the scheduled time.

• Do ask questions, but be respectful of the current topic and
the interests of other students.
• Ensure that cell phones and pagers are silent.
Classroom Guidelines
These guidelines enable you to get maximum benefit from the course.

For More Information
Topic Website
Education and training http://education.oracle.com
Product documentation http://www.oracle.com/technology/documentation
Product downloads http://www.oracle.com/technology/software
Product articles http://www.oracle.com/technology/pub/articles
Product support http://www.oracle.com/support
Product forums http://forums.oracle.com
Product tutorials and demos http://www.oracle.com/oll
For More Information

These materials are not intended to be a complete reference for all hardware and software
products associated with Oracle Exalogic. After you complete the course, Oracle provides a
variety of resources that you can use to obtain additional information.

Oracle by Example (OBE)
OBEs are free online step-by-step tutorials that cover specific

product features.
Oracle by Example (OBE)

The Oracle by Example (OBE) series provides hands-on, step-by-step instructions on how to
implement various technology solutions to business problems. OBE solutions are built for
practical real-world situations, enabling you to gain valuable hands-on experience as well as
use the presented solutions as the foundation for production implementation, dramatically
reducing time to deployment.

Related Training
Course Title
Sun Fire Server Installation and Administration
Sun Storage 7000 Administration
Oracle WebLogic Server: Administration Essentials
Oracle WebLogic Server: Advanced Administration
Oracle WebLogic Server: Monitor and Tune Performance
Oracle WebLogic Server: Diagnostics and Troubleshooting
Oracle Coherence: Administer and Troubleshoot Clusters
Exadata and Database Machine Administration Workshop
Oracle Enterprise Manager Grid Control: Essentials
Sun Ops Center Administration
Related Training
Note that some of the courses listed in the slide are available as traditional or virtual
classroom training, while others are self-paced online offerings (eStudies).

The Exalogic Solution

Objectives
After completing this lesson, you should be able to:

• Describe the challenges faced by data center
administrators
• List the major features of Exalogic
• Discuss the software supported by the Exalogic hardware

Why Data Centers?
• Traditional distributed applications suffer from:

– Heterogeneous and unreliable hardware
– Uneven performance and resource utilization
– High-latency networks
– Complex integration solutions
– High labor costs for deployment and maintenance
• The data center model strives to solve these problems by
using a centralized infrastructure.
Why Data Centers?

For most enterprise IT organizations, years of innovation, expansion, and acquisition have
resulted in sprawling infrastructure that stretches the limits of manageability. Though the
individual IT systems and applications in service are often well considered and expertly
implemented, the sheer scale of the ongoing IT investment itself has emerged as the
dominant concern. Even when the latest technologies, open standards, vendors, and
architectural practices, such as service-oriented architecture (SOA), have been used
pervasively, most enterprises now find themselves with too many platforms, too many
technologies, too many domains of expertise, and too many vendors to coordinate and
manage.
In response, a number of technologies and practices have become staples for large
enterprises, ranging from virtualization and centralized storage to enterprise-wide
standardization of software and hardware. This transformation typically involves adoption of a
more centralized, automated, and elastic infrastructure and regime. Perhaps, most
importantly, these engineered systems must not be monolithic. It must be possible for
enterprises to implement them over time, at a pace determined by real business needs and
investment timelines.

Data Center Challenges
Common concerns include:

• Meeting performance and availability requirements
• The consolidation of resources to minimize floor space and
power costs
• Keeping applications isolated from one another, to address
maintenance and security concerns
• Keeping the system manageable
• Scaling the infrastructure as the business grows
Data Center Challenges

The need for enterprise IT organizations to provide next-generation cloud features such as
elastic capacity while meeting ever more demanding performance and reliability requirements
is driving demand for a new approach to infrastructure. Whether data-intensive or processing-
intensive, whether homogeneous or highly heterogeneous, the key to success is hardware
and software engineered together for performance, reliability, and scale. Building or using
custom, special-purpose systems for different applications is wasteful and expensive.
The most common model for application isolation involves multiple IP subnetting, in which the
most mission-critical applications are assigned their own IP subnets. In this model, some
subnets may also contain applications that have less stringent or otherwise different resource
requirements. Other subnets may contain multiple applications, such as those dedicated to a
given department or line of business, or even used for application testing and development.

What Is Exalogic?
CPUs, storage, Optimized Oracle

network software
What Is Exalogic?
Exalogic is an integrated hardware and software system designed to provide a complete
platform for a wide range of application types and widely varied workloads. Exalogic is
intended for large-scale, performance-sensitive, mission-critical application deployments. It
combines Oracle Fusion Middleware and Sun hardware to enable a high degree of isolation
between concurrently deployed applications, which have varied security, reliability, and
performance requirements. Exalogic enables customers to develop a single environment that
can support end-to-end consolidation of their entire applications portfolio.
Exalogic hardware is preassembled and delivered in standard 19” 42U rack configurations.
The main hardware components of a single Exalogic machine Full Rack are the following:
• 30 Sun Fire X4170 M2 compute nodes
• One dual controller Sun ZFS Storage 7320 appliance with 20 disks
• Four Sun Network QDR InfiniBand Gateway Switches
• One Sun Datacenter InfiniBand Switch 36
• One 48-port Cisco Catalyst 4948 Ethernet management switch
• Two redundant 24 kVA power distribution units

What Is Exalogic?
Oracle Exalogic hardware:

• Combines Sun storage and servers by using a high-speed
InfiniBand network
• Supports up to 360 cores and 2.8 TB of RAM per rack
installation
• Is specifically engineered to host Oracle Fusion
Middleware software
• Results in huge performance gains for WebLogic
applications
• Supports Enterprise Manager for centralized configuration,
monitoring, and diagnostics
What Is Exalogic? (continued)

You can connect up to eight Exalogic machines, or a combination of Exalogic machines and
Oracle Exadata database machines, together without the need for any external switches. If
more than eight racks are required to be connected on the same InfiniBand fabric, Oracle
offers a choice of several high-capacity data center switches, which enable the creation of
Exalogic clouds comprising hundreds of racks and tens of thousands of processors.
Exalogic is designed to fully leverage an internal InfiniBand fabric that connects all of the
processing, storage, memory, and external network interfaces within an Exalogic machine to
form a single, large computing device. Each Exalogic machine is connected to the customer's
data center networks via 10 GB Ethernet (external traffic) and 1 GB Ethernet (management
traffic) interfaces.
Oracle Enterprise Manager Grid Control with Oracle WebLogic Server Management Pack
Enterprise Edition's capabilities include Exalogic specific management tools to monitor the
Oracle software deployed in the Exalogic environment. If using Solaris as your Exalogic
operating system, you can also utilize Oracle Enterprise Manager Ops Center to provide
configuration and management capabilities for the Exalogic hardware components.

Why Exalogic?
The Exalogic solution:

• Is rigorously tested and certified for Fusion Middleware
• Provides the best Java performance for the cost
• Is factory assembled and then configured to your
specifications
• Supports horizontal scalability by linking racks together
through InfiniBand
• Facilitates multiple levels of application isolation
• Proactively works with the Oracle support network
• Can be directly linked to your existing Oracle Exadata
infrastructure
Why Exalogic?
The combination of Oracle Exalogic software and hardware results in substantial performance
gains for Java based applications running on WebLogic Server and other Oracle Fusion
Middleware technologies. In addition to performance, the fact that Exalogic hardware and
software have been engineered together also means that customers are required to do far
less in setting up and running Exalogic than for alternative environments.
InfiniBand is fundamental to the Exalogic Elastic Cloud system. In addition to providing an
extremely fast, high-throughput interconnect between all of the hardware units within a
deployment, it also provides extreme scale, application isolation, and elasticity. Traditional
approaches to growing a data center’s compute capacity involve either vertically scaling
individual computers or networking together many individual computers by using common
technologies such as Ethernet. By contrast, the lossless switched InfiniBand I/O fabric (on
which the Exalogic system is based) connects all configurations together, essentially forming
a single large computer.
An Exalogic cloud is a resource pool that can be dynamically subdivided into secure units of
capacity. InfiniBand supports partitions, in which communication between endpoints on the
I/O fabric is strictly controlled. Individual compute nodes, or even specific I/O devices, may be
grouped into logical partitions within which communication is allowed.

What Is InfiniBand?
• Distributed systems communicate with one another to:

– Separate application tiers (Web, business logic, caching)
– Provide high availability (“clusters”)
• InfiniBand:
– Is a standard network architecture designed for very high
speeds between nodes in close proximity to each other
– Acts as a “fabric” to connect CPUs and other devices
– Is ideal for supercomputers and data centers
– Delivers 40 GB (4x faster than high speed Ethernet)
– Supports direct application connections that bypass the OS
– Is interoperable with Ethernet for external communication
What Is InfiniBand?
Moving data between applications over a traditional network can consume a lot of time and
drain precious server resources. With traditional network technologies, data exchanges
traverse the operating systems on both the source and destination servers, resulting in
excessive application latency due to operating system calls, buffer copies, and interrupts.
InfiniBand, which today delivers 40 GB per second connectivity with application-to-application
latency as low as 1 microsecond, has become a dominant fabric for high-performance
enterprise clusters. Its ultra-low latency and near-zero CPU utilization for remote data
transfers make InfiniBand ideal for high-performance clustered applications.
InfiniBand also provides a direct channel from the source application to the destination
application, bypassing the operating systems on both servers. InfiniBand’s channel
architecture eliminates the need for OS intervention in network and storage communication.
This frees server memory bandwidth and CPU cycles for application processing.
In addition to carrying all InfiniBand traffic, the Sun Network QDR InfiniBand Gateway Switch
enables all InfiniBand attached servers to connect to an Ethernet LAN by using standard
Ethernet semantics. No application modifications are required for applications written to use
standard Ethernet.

An Engineered Solution
• Exalogic hardware and software are specifically

engineered and tested to work together “out of the box”:
– Servers, storage, network, OS, drivers, middleware
– Easy to replicate and scale
• Custom data center solutions:
– Require significant time to plan, implement, and test
– Must be built one component at a time
– Result in compatibility issues
– Require complex ordering and patching procedures
Engineered Solution
An engineered system combines network, storage, processing, and software into a pre-
optimized package. An engineered approach enables IT to cut deployment time from months
to hours, reduces errors, and enables IT personnel and resources to be allocated to higher-
value work. It also reduces the overall time and costs associated with planning, procurement,
installation, and testing. Exalogic provides a repeatable model for building out data center
systems.

Java Performance
• Java is one of the most pervasive technologies in today’s

enterprise applications.
• Oracle’s Java EE platform is now optimized for Exalogic
specific hardware capabilities, including:
– JRockit and Hotspot Java Virtual Machines (JVMs)
– WebLogic Server
• Oracle benchmarks show significant performance gains
against comparable hardware:
– 5 to 10 times faster response times
– 50% more throughput
Java Performance
Exalogic has been designed from the ground up to provide the ideal environment for
enterprise Java applications and Java based infrastructure. Oracle’s entire Fusion Middleware
portfolio is optimized for deployment on Exalogic. Oracle Exalogic Elastic Cloud software
includes a number of optimizations and enhancements made to the core products within
Oracle WebLogic Suite, which includes Oracle WebLogic Server, Oracle Coherence, Oracle
JRockit, and Oracle HotSpot. The combination of Oracle Exalogic software and hardware
results in substantial performance gains for Java based applications running on WebLogic
Server and other Oracle Fusion Middleware technologies.

High Availability
• Hardware redundancy at all levels, including:

– Processors
– Disks
– Power and cooling
– Network switches and cabling
• Software features include:
– Bonded network interfaces
– WebLogic clustering and automatic server migration
– Active GridLink database connections
High Availability
Hardware components within an Exalogic machine use multiple physical InfiniBand (IB) ports
and switches for high availability. Each InfiniBand adapter has two ports, is assigned a single
IP address, and supports active/passive bonding. The active port of the adapter connects to
an IB switch and the passive port connects to another switch. If one fails, traffic is
transparently failed over to the other.
Clustering is configuring a group of WebLogic Servers to work together to provide client
access to the services offered by the servers in the cluster. The cluster appears to a client as
one instance, whether the client is a Web client or a Java application. By replicating the
services provided by one instance, an enterprise system achieves a fail-safe and scalable
environment. Scalability is achieved by balancing the load of incoming requests across the
servers in the cluster.
When a migratable WebLogic Server becomes unavailable for any reason (for example, if it
hangs or loses network connectivity, or if its host machine fails), migration is automatic. Upon
failure, the server is automatically restarted on the same machine if possible. If the server
cannot be restarted on the machine where it failed, it is migrated to another machine. In
addition, an administrator can manually initiate migration of a server instance.

Multi-Tenancy
• Data centers must be able to:

– Host applications from multiple lines of business or even
multiple businesses
– Balance density with security
• Exalogic supports isolation at multiple levels:
– Physical network layer (InfiniBand partitions, VLANs)
– Transport network layer (IP subnets)
– Storage access control
– Solaris zones
• Future Exalogic versions will also support
additional hardware virtualization solutions.
Multi-Tenancy
By default, the Exalogic machine includes a single InfiniBand partition at the rack level. All
Exalogic compute nodes and the storage appliance are full members of this default partition.
The Exalogic network hardware allows you to configure multiple partitions to restrict traffic
between specific IB connections. However, Oracle currently only supports the use of the
default partition to help ensure compatibility between multiple Exalogic and/or Exadata racks.
The Exalogic network hardware does support virtual LANs (VLANs) to achieve a similar result
to partitions. Finally, you can limit access to the network by using multiple subnet masks. The
subnet masks will be enforced at the OS level, not by the network hardware.
If you are using the Solaris OS on Exalogic, you can also utilize the zones feature. Zones act
as completely isolated, virtual operating systems within a single OS instance. Each zone has
its own node name, virtual network interfaces, and storage assigned to it.
You can limit access to an Exalogic shared file system by using both standard UNIX file
permissions and IP- or DNS-based client exceptions. When a client attempts access, its
access will be granted according to the first exception in the list that matches the client (or, if
no such exception exists, according to the global share mode).

Software Compatibility
• On Exalogic hardware, Oracle currently supports:

– Oracle Linux 5.5 and Solaris 11 operating systems
– Any applications already certified on these platforms
• Fusion Middleware 11.1.1.4 (WebLogic Server 10.3.4) is
optimized to take advantage of Exalogic hardware,
including the InfiniBand network.
Software Compatibility
In addition to unique support for Java applications and Oracle Fusion Middleware, Exalogic
also provides users with a choice of Oracle Linux or Oracle Solaris operating systems.
Exalogic is 100% compatible with all standard Oracle Linux 5 and Solaris 11 applications, and
no special certification for Exalogic is required. In other words, all Oracle applications that are
certified for the appropriate releases of Oracle Linux and Solaris are supported on Exalogic.
The Exalogic specific optimizations for Fusion Middleware are certified only against Exalogic.
They are not supported on other non-Exalogic hardware, and regardless of whether the
hardware utilizes InfiniBand. Currently, Exalogic is not certified to run virtualization platforms
such as Oracle VM or VMware, although this is expected to change in the future.

Selecting an Operating System
• At the factory:
– Both Linux and Solaris base images are pre-installed on the
storage appliance
– The compute nodes are configured to run Linux by default
• The latest base images are also available on
edelivery.oracle.com.
• Refer to the Machine Guide for detailed instructions on re-
imaging the compute nodes with Solaris.
Selecting an Operating System

During manufacturing, Exalogic compute nodes are pre-installed with the Oracle Linux
operating system. The Exalogic Machine Guide documentation describes how to re-image
Exalogic compute nodes from Oracle Linux to Oracle Solaris. The imaging process is
performed using a laptop that is connected to your Exalogic machine rack. It utilizes a Preboot
Execution Environment (PXE) server architecture. The overall tasks are:
1. Install Oracle VirtualBox on the laptop.
2. Connect the laptop to the Exalogic management network.
3. Run the Solaris Automated Installer (AI) VM image on the laptop.
4. Start the AI and DHCP servers within the AI VM.
5. Configure a list of compute node MAC addresses that the AI server will accept.
6. Access each compute node’s ILOM interface, configure it to boot using PXE, and reboot
it. During the first boot, the Oracle Solaris operating system is installed on the compute
node. During the second boot, the first run service executes and sets up the Exalogic
machine environment, including network, tuning, and so on. After the third boot, you can
start using the compute node.

Licensing and Support Overview
Hardware:
• Exalogic racks and supporting hardware are sold in one of
three specific configurations, depending on initial capacity
needs.
• To provide quality support, racks can be expanded and
combined only in specific ways.
• Low-level hardware customization is not supported.
Software:
• A license for Exalogic Elastic Cloud Software permits the
use of Exalogic specific-features in WebLogic Suite.
• WebLogic Suite is a separate license and a prerequisite.
Licensing and Support Overview

Exalogic Elastic Cloud Software is not a binary distribution like WebLogic Server. It is simply
the name given to the use of WebLogic Server's features that optimize its performance on
Exalogic. These features are disabled by default. Like most Oracle software products, both
WebLogic Suite and Exalogic Elastic Cloud Software are licensed per processor.

Quiz
What type of network fabric is used to connect Exalogic

hardware components?
a. Ethernet
b. iWARP
c. VIA
d. InfiniBand
Answer: d

Quiz
Which two licenses are required to use Fusion Middleware's

optimizations for Exalogic?
a. WebLogic Suite
b. Enterprise Manager Grid Control
c. Exalogic Elastic Cloud Software
d. E-Business Suite
Answer: a, c

Summary
In this lesson, you should have learned how to:

• Describe the challenges faced by data center
administrators
• List the major features of Exalogic
• Discuss the software supported by the Exalogic hardware

Hardware Components

Objectives

• Describe the components of an Exalogic machine and their
relationships
• Discuss the hardware specifications for an Exalogic
compute node
• Discuss the hardware specifications for the Exalogic
storage appliance
• Compare the available hardware packages
• Discuss opportunities for scaling your Exalogic data center

Exalogic Machine Topology
Clients
Rack
Compute Node
Client Compute Node Mgmt

Network Network
10 Gb 40 Gb
InfiniBand Mgmt 1 Gb
Storage Appliance
Switches Switch
40 Gb
IB
Network
Power Distribution Units (PDUs)
Exalogic Machine Topology

Exalogic hardware is preassembled and delivered in standard 19” 42U rack configurations.
Each Exalogic configuration is a unit of elastic cloud capacity balanced for compute-intensive
workloads. Each Exalogic configuration contains a number of hot-swappable compute nodes
along with a clustered, high-performance disk storage subsystem. The hardware also
includes a high-bandwidth InfiniBand fabric to connect every individual component within the
configuration as well as to externally connect additional Exalogic or Exadata Database
Machine racks. In addition, each configuration includes multiple 10 Gb Ethernet ports for
integration with the data center's service network, along with 1 Gb Ethernet ports used for
integration with the data center’s management network. All Exalogic configurations are fully
redundant at every level and are designed with no single point of failure.
All device management ports are connected to your local data center management network
by using a Cisco Catalyst 4948 switch, which is a built-in component of an Exalogic rack. This
switch offers 48 ports of wire-speed 10/100/1000BASE-T with 4 alternative wired ports that
can accommodate optional 1000BASE-X Small Form-Factor Pluggable (SFP) optics.
Reliability and serviceability are delivered with optional internal AC or DC 1 + 1 hot-
swappable power supplies and a hot-swappable fan tray with redundant fans.

Compute Nodes
• Dual processor
• Redundant power and cooling
• Redundant solid state boot disks
• InfiniBand host channel adapter
Sun Fire X4170 M2

2.93 GHz Intel Xeon Cores 12 (6 per CPU)
1333 MHz RAM 96 Gb
FlashFire Solid State Disk 32 GB (64 Gb raw)
InfiniBand Ports 2
Gigabit (GB) Ethernet Ports 4
Compute Nodes
Data centers often standardize on a server that can run a wide range of business applications
and can be managed easily. The versatility of the Sun Fire X4170 M2 server makes it a
perfect fit for this, because this server balances compute power, memory capacity, and I/O
capability in a compact and energy-efficient, 1-rack unit (RU) enclosure. This is a server that
is ideal for middleware workloads, system administration, and application development.
The Sun Fire X4170 M2 server includes two Intel Xeon Processor 5600 series, each with six
processing cores. This processor includes an energy management feature that reduces
power consumption when workload decreases. In addition, the server has built-in hardware
and software power management capabilities to further enhance power efficiency.
Each compute node contains two solid-state disks (SSDs), which host the operating system
images used to boot the node, act as high-performance local swap space, and also provide
storage for any diagnostic data generated by the system during a failure. Mirroring for the two
solid-state disk drives is accomplished via a RAID controller.
The Sun Fire X4170 M2 server supports up to 18 double data rate (DDR3) memory dual inline
memory module (DIMMs) slots, which can be populated with 4 GB and 8 GB DIMMs.
However, within an Exalogic configuration, Oracle supports only the twelve 8 GB DIMMs that
are installed at the factory.

Compute Node: Front View
32 GB SATA Solid
State Disk (2) USB (2)
Power LED Empty (6)

Fault LEDs
Power
Button
Compute Node: Front View

Although an Exalogic compute node provides bays to support up to 8 SSD or PCI disks, only
two of these are used by default. Each contains a 32 GB SSD disk, which is kept
synchronized by using a RAID controller.
The default contents of the compute node disk include an Oracle Linux image. No separate
OS installation is required. If you prefer to use Solaris instead, a Solaris image is available on
the storage appliance.
Although the compute node's local disk has available space, its sole intended purpose is to be
a high-speed boot disk. Oracle recommends that customers do not use it to store applications
and other data. Instead, they can use the shared storage appliance for all non-OS artifacts.

Compute Node: Rear View
Redundant Power Serial PCI Express InfiniBand

Supply (2) Management Port 2.0 Slot (3) Port (2)
Video
USB (2)
Status LEDs 1 Gb Ethernet 1 Gb Network Interface
Management NIC Cards (NICs) (4)
Compute Node: Rear View

The Sun Fire X4170 M2 server offers hot-swappable and redundant RAID-enabled disks,
cooling fans, and power supply units.
InfiniBand connectivity is accomplished via the dual-port host controller adapter (HCA). This is
a quad data rate (QDR) PCI-E card, which takes advantage of the latest InfiniBand rates.
System management is simplified with Oracle Integrated Lights Out Manager (ILOM), which
comes standard in every Sun Fire X4170 M2 server. ILOM centralizes system management
locally or remotely to ease system configuration, software provisioning, and updates. It
provides a consistent interface across the entire x86 product line.

Compute Node Memory
• Each compute node has 18 DIMM slots and supports up to

144 GB RAM (8 GB per slot).
• However, the memory is fixed at 96 GB to optimize the
clock speed at 1333 MHz.
• Oracle does not support modifying the default hardware
configuration.
Compute Node Memory

DIMM = dual inline memory module
Each CPU supports three channels, each of which supports three DIMMs, making a total of
18 (2 x 3 x 3).

Storage Appliance
• Two redundant nodes (“heads”)

• SAS array of twenty hard disks
(mirrored, two hot spares)
• Solid state cache disks
• InfiniBand connectivity
• Web administration console
Sun ZFS Storage Appliance 7320

2.40 GHz Intel Xeon Cores 4 per head
1333 MHz RAM 12 per head
Serial Attached SCSI (SAS) Disk 18 TB (40 TB raw)
Solid State Read Cache 2 TB per head
Solid State Write Cache 72 GB
Exalogic Storage
The Exalogic ZFS Storage 7320 system consists of two storage controllers or “heads” in a
high-availability cluster configuration, along with one disk shelf. The 7320 controller base
configuration includes two CPUs, 4 Gb Ethernet ports, redundant power supplies, an
InfiniBand network controller, and dual-port SAS HBA for storage expansion. The CPUs are
Intel Xeon 5600 series, 2.40 GHz, and contain four core processors each (a total of eight
cores). The clustered configuration simply uses two servers and a cluster card in each server
for a heartbeat connection between them.
All user-accessible storage is provided by one Sun disk shelf that is external to the two
servers. Solid-state drives are used as a high-performance write cache known as LogZilla or
ZFS intent log (ZIL) devices, and are in place of 4 of the 24 drives in the disk shelf. The
remaining 20 drives are available for storage.
The RAID function is performed by the software. RAID-1 is recommended, but other options
are available. In the default configuration, there are twenty 2 TB disks (40 TB in total space)
but two of them are used as spares. Therefore, there is 36 TB total disk space or 18 TB of
replicated disk space.

Disk Caching
The storage appliance utilizes high-speed solid state disks to:

• Cache frequently read files
• Write files as a background task
Storage Node
Compute File write
Node File read Compute
Read Cache
Node
Write Cache/Buffer
Disk Array
Disk Caching
The Exalogic storage appliance transparently executes writes to low-latency SSD media so
that writes can be quickly acknowledged, allowing the application to continue processing.
Then, it automatically flushes the data to high-capacity drives as a background task.
Another type of SSD media acts as a cache to reduce read latency, and the appliance also
transparently manages the process of copying frequently accessed data into this cache to
seamlessly satisfy read requests from clients.

Storage Appliance: Front View
Nodes (2) 512 GB Solid State 512 GB Boot

(Active and Standby) Read Cache (4 each) Disks (2 each)
18 GB Solid State
Write Cache (4)
2 TB Hard Disk
Drive (20)
Storage Appliance: Front View

Within each of the two Exalogic storage controllers, there are four 512 GB solid state disks,
which are used as a read cache by ZFS. There are also two redundant hard disks that are
used to boot the appliance’s operating system.
On the disk shelf, four of the drive bays are occupied by 18 GB solid state disks, which serve
as a write cache for the storage appliance. The remaining 20 drives are conventional 2 TB
SAS hard disk drives.

Storage Appliance: Rear View
Cluster interconnect SAS Host Bus

InfiniBand ports
for failover Adapter (HBA)
Storage Appliance: Rear View

All inter-head communication consists of one or more messages transmitted over one of the
three cluster I/O links provided by the Clustron hardware. This device offers two low-speed
serial links and one Ethernet link. The use of serial links allows for greater reliability. Ethernet
links may not be serviced quickly enough by a system under extremely heavy load. All three
links are formed by using ordinary straight-through gigabit Ethernet cables. The cables
connecting the serial connections must be used to connect opposing sockets on the two
connectors, effectively providing a crossover.
Clustered heads never communicate using external service or administration network
interfaces, and the interconnects form a secure private network. Messages fall into two
general categories: regular heartbeats used to detect the failure of a remote head, and higher-
level traffic associated with the resource manager and the cluster management subsystem.
The SAS HBA and the JBOD designs both include two four-channel mini-SAS connectors for
the A path and B path. Both the A and B paths can access all of the disk drives which
provides connectivity to all of the disks from either path.

InfiniBand (IB) Switches
• InfiniBand Quad Data Rate (QDR) ports

• 10 Gb Ethernet gateway ports
• Redundant power and cooling
• Embedded management module
• LEDs for box status and for each
connection status
NM2-GW NM2-36P
Spine (used only to
Type Gateway (“leaf”)
connect multiple racks)
40 Gb IB Ports 32 36
10 Gb Ethernet
8 0
Ports
InfiniBand Switches
Within Exalogic, the InfiniBand gateway switches provide a shared, high-speed network for
application clusters that comprise compute nodes and the shared storage appliance. The
gateway switch features a standard 1U data center form factor, thirty-two 40 Gb InfiniBand
ports, eight 10 Gb Ethernet ports, and the Sun ILOM management interface.
Oracle's converged fabric leverages the properties of the InfiniBand architecture to enable
high performance in demanding, clustered data center environments. The fabric supports the
creation of logically isolated network partitions as well as advanced features for traffic
isolation and quality of service (QoS) management. QDR InfiniBand delivers 40 Gb of raw
bandwidth. The InfiniBand base rate is 10 Gb per second. QDR-capable products deliver four
times the base rate.
Deploying a converged fabric minimizes the cost and complexity of building and operating
scalable systems by reducing the number of adapters, cables, and switches to install,
manage, and maintain. LAN traffic carried over Oracle's converged data center fabric uses
familiar Ethernet semantics. Furthermore, the LAN ports on the Sun Network QDR InfiniBand
Gateway Switch appear as endpoints to the LAN, thus avoiding potential interoperability
issues with network management tools and methods.

Gateway Switch Hardware Views
Ethernet Port (8)
Ethernet Four over each IB port
Management Port (2)
REAR
InfiniBand Port (32) Unused Port (2)
Redundant Power
Supply (2)
FRONT
Redundant Fans
(3; 5 max.)
Switch Hardware Views

Within the Sun Network QDR InfiniBand Gateway Switch, the embedded fabric manager is
enabled to support active/standby dual manager configurations, ensuring a seamless
migration of the fabric management services in the event of a module failure. The switch is
also provisioned with redundant power and cooling for high availability.
The front panel of the Sun Network QDR InfiniBand Gateway Switch has 36 QSFP
connectors. The connectors are each dedicated to InfiniBand or Gateway functions as follows:
• The “leftmost” 32 ports support QDR InfiniBand connectivity.
• The two “upper rightmost” ports support Ethernet connectivity.
• The two “lower rightmost” ports are covered with a plug marked “Do Not Remove.” No
connections should be made to these ports.
The switch has two physical ports dedicated to Ethernet functionality. These ports can support
up to four 10 GB Ethernet connections each. Oracle provides “splitter” cables, which fan out a
single Quad Small Form Pluggable (QSFP) switch port to four optical fiber LC connectors. A
QSFP transceiver is available to connect the “splitter” cable to the switch.

IB Host Channel Adapter (HCA)
• Compute and storage nodes use two physical IB ports and

switches for redundancy (active/passive).
• A special “bond” network adapter on the OS makes these
two interfaces appear as a single interface to applications.
IB Host Channel Adapter (HCA)

Oracle's family of Sun PCIe Quad Data Rate (QDR) InfiniBand Host Channel Adapters
(HCAs) gives you better price/performance for both throughput and low latency. They are
designed for servers supporting the PCI Express 2.0 specification with either the low-profile or
modular hot-pluggable PCI Express (PCIe) ExpressModule form factors. The adapters
support two IB ports for connecting IB traffic (QSFP connectors); each port supports 10
Gb/sec traffic on SDR, 20 Gb/sec traffic on DDR, and 40 Gb/sec on QDR.

Management Switch
• All device management ports are

connected to a dedicated 1 Gb switch.
– NET0 on compute nodes and storage
appliance
• Isolate management and monitoring
tasks from client traffic.
• The switch includes redundant power
and cooling.
Cisco Catalyst 4948

1 Gb Ethernet Ports 48
Management Switch
The Cisco Catalyst 4948 offers 48 ports of wire-speed 10/100/1000BASE-T with four
alternative wired ports that can accommodate optional 1000BASE-X Small Form-Factor
Pluggable (SFP) optics. Exceptional reliability and serviceability are delivered with optional
internal AC or DC 1+1 hot-swappable power supplies and a hot-swappable fan tray with
redundant fans.
The Cisco Catalyst 4948 includes a single, dedicated 10/100 console port and a single,
dedicated 10/100 management port for offline disaster recovery. Remote in-band
management is available with the Simple Network Management Protocol (SNMP), telnet
client, Bootstrap Protocol (BOOTP), and Trivial File Transfer Protocol (TFTP). Support for
local or remote out-of-band management is delivered through a terminal or modem attached
to the console interface. The management port helps enable the Cisco Catalyst 4948 to
reload a new image from a TFTP server.

Scaling Your Data Center
Multi-Rack
Full Rack
Quarter Half Rack
Rack

InfiniBand is fundamental to the Exalogic Elastic Cloud system. In addition to providing an
extremely fast, high-throughput interconnect between all of the hardware units within a
deployment, it also provides extreme scale, application isolation, and elasticity. Traditional
approaches to growing a data center’s compute capacity involve either vertically scaling
individual computers or networking many individual computers by using common technologies
such as Ethernet. By contrast, the lossless switched InfiniBand I/O fabric (on which the
Exalogic system is based) connects all configurations, essentially forming a single large
computer.
Exalogic systems scale horizontally, meaning that there is no degradation of system
performance as the size of the cloud increases. It is equally important that an Exalogic cloud
is a resource pool that can be dynamically subdivided into secure units of capacity. The
underpinnings of this capability are in the design of InfiniBand itself, which was designed for
precisely this use. InfiniBand supports partitions, in which communication between endpoints
on the I/O fabric is strictly controlled within the fabric switches. Individual compute nodes, or
even specific I/O devices, may be grouped into logical partitions, within which communication
is allowed.

• Grow your data center based on these three supported

configurations (quarter rack, half rack, and full rack).
• Half and full racks include unused “spine” InfiniBand
switches to link racks together.
• Connect from two to eight full racks without any external
InfiniBand switches.
Scaling Your Data Center (continued)

It is possible to connect as many as eight full racks of Exalogic machines (or any combination
of Exalogic and Exadata configurations) without the need for any external switches. In cases
where more than eight racks of Exalogic or Exadata hardware are required, Oracle offers a
choice of several high-capacity data center switches. This approach supports the creation of
Exalogic clouds that can comprise hundreds of racks and tens of thousands of processors.

Hardware Package Details
Metric Quarter Rack Half Rack Full Rack

Compute Nodes 8 16 30
Node Cores 96 192 360
Node RAM 768 GB 1.5 TB 2.8 TB
Storage Appliance 1 1 1
Disk Storage 40 TB 40 TB 40 TB
IB Gateway Switches 2 2 4
IB Spine Switches 0 1 1
Approx. Weight (lbs.) 1000 1500 2100
Hardware Package Details

Quarter-rack configurations contain eight compute nodes, one storage appliance that contains
an active-passive pair of servers along with disks, two InfiniBand (IB) gateways, and a Cisco
management switch.
Half-rack configurations contain 16 compute nodes, one storage appliance, two IB gateways,
one IB spine switch, and a management switch.
Full-rack configurations contain 30 compute nodes, one storage appliance, four IB gateways,
one IB spine switch, and a management switch.
Note that the Exalogic shipping pallet, ramps, and corrugated packaging add approximately
150 pounds to the values mentioned here.

Power Options
• A full rack includes (and requires) two redundant 24-kVA

power distribution units (PDUs).
– Three-phase high and low options
– No single-phase option
• Half racks and quarter racks are available with 15-kVA or
24-kVA PDUs.
• PDU power consumption can be monitored from an LCD
display or from the rack’s management network.
40A
30A
20A
10A
0A
Phase 1 Phase 2 Phase 3
16.1 14.5 14.1
Power Options
To protect your Exalogic machine from such disturbances, you should have a dedicated
power distribution system, power-conditioning equipment, as well as lightning arresters or
power cables to protect from electrical storms. The cabinets for the Exalogic machine are
shipped with grounding-type power cords (three-wire). Always connect the cords to grounded
power outlets. For additional grounding, attach a chassis earth ground cable to the Exalogic
machine. The additional ground point enables electrical current leakage to dissipate more
efficiently.
Each rack has two pre-installed power distribution units (PDUs). Different types of PDUs
accept different types of power sources and use different numbers of power outlets. You must
specify the type of PDU that is correct for your data center.
If you upgrade a quarter rack or half rack that was originally ordered with a 15-kVA PDU to a
full rack, you must also upgrade to a 24-kVA PDU.

Hardware Maintenance
• The Exalogic Machine Guide includes a “Replacement

Units” section that lists part numbers for:
– Compute node SSDs, fans, power supplies, memory, and
PCI-E cards
– Storage appliance SSDs, HDDs, fans, power supplies, and
PCI-E cards
– Switch fans and power supplies
– IB and Ethernet cables of various lengths
• Refer to the service manuals of individual products for
detailed component replacement steps.

Quiz
Name three supported Exalogic rack configurations.

a. Admin
b. Half
c. Basic
d. Quarter
e. Full
Answer: b, d, e

Quiz
Which two types of storage are available to an Exalogic

compute node?
a. Shared disks
b. Gateway disk
c. Solid State disk
d. QDR disk
Answer: a, c

Quiz
How many hard disks does the Exalogic storage appliance

contain?
a. 10
b. 15
c. 18
d. 20
e. 40
Answer: d

Quiz
What type of InfiniBand switch is used to connect multiple

racks?
a. Subnet
b. Gateway
c. Spine
d. Management
e. Quarter
Answer: c

Summary

• Describe the components of an Exalogic machine and their
relationships
• Discuss the hardware specifications for an Exalogic
compute node
• Discuss the hardware specifications for the Exalogic
storage appliance
• Compare the available hardware packages
• Discuss opportunities for scaling your Exalogic data center

Initial Machine Configuration

Objectives

• Locate and use checklists and worksheets to plan an
installation
• Explain how Oracle initializes a rack based on your input
• Describe the Exalogic default network setup
• List some basic diagnostic utilities

Power-On Procedure
1. Ensure that all power cables are connected:

– Single phase: three cables from each PDU
– Three phase: one cable from each PDU
2. Ensure that all six PDU circuit breakers are turned on.
3. Wait for all components to start in standby mode (power
LEDs blink).
4. Press the power button on both storage appliance heads
and verify that power LEDs no longer blink.
5. Press the power button on each compute node
and verify that power LEDs no longer blink.
6. Verify that power LEDs are lit on the switches.
7. Remotely reboot components by using ILOM.
Power-On Procedure
For a newly installed Exalogic rack, unfasten the power cord cable ties. The ties are for
shipping only and are no longer needed. Then route the power cords to the facility
receptacles either above the rack or below the flooring, and secure the power cords in
bundles. Lastly, plug the PDU power cord connectors into the facility receptacles. The 24
kVA PDU has four plugs, and the 15 kVA PDU has two plugs
Switch on the six power distribution unit (PDU) circuit breakers located on the rear of the
main PDU (PDU A) inside the Exalogic machine. The circuit breakers are on the rear of the
Exalogic machine cabinet. Press the ON (|) side of the toggle switch. Wait three to five
minutes for all Oracle Integrated Lights Out Manager (ILOM) service processors to boot.
Next, open the front cabinet door. Verify that server standby power is on for compute
nodes, InfiniBand gateway switches, and storage appliance in the Exalogic machine. When
standby power is distributed to the chassis, the service processor (SP) powers on and
starts up to manage the system. The main power is supplied for the remainder of the
chassis and fans when the power button on the front of the server chassis is pressed. The
power button is disabled while the SP is starting. After the SP has started, the power button
is enabled and the system is placed in standby power mode.

Hardware Configuration Process
1. The customer completes checklists and worksheets.

2. Oracle delivers, installs, connects, and tests the rack(s).
3. Oracle runs the Exalogic Configuration Utility (ECU) to
initialize network and other settings.
4. The customer uses ILOM and OS tools to further
customize the default network configuration, if needed.
5. The customer configures projects and shares on the
storage appliance.
1 2 3
Hardware Configuration Process

Initial installation and startup (step 2 in the slide) includes the following substeps:
1. Ensure that each of the three main power cords is connected.
2. Switch on the six power distribution unit (PDU) circuit breakers located on the rear of
the main PDU (PDU A) inside the Exalogic machine. The circuit breakers are on the
rear of the Exalogic machine cabinet.
3. Wait three to five minutes for all ILOM service processors to boot.
4. Open the front cabinet door. Verify that server standby power is on for compute
nodes, InfiniBand gateway switches, and the Sun ZFS Storage 7320 appliance in the
Exalogic machine. The Power/OK LED on the front panel blinks green in a pattern of
0.1 seconds on, 2.9 seconds off.
5. Verify that the main power is on for each compute node. In main power mode, the
Power/OK LED on the front panel blinks in a pattern of one second on, one second off
while the system BIOS starts. After the BIOS initialization is completed and the
operating system begins to start, the Power/OK LED is illuminated and remains a
steady green.
6. Press the soft switches located on the front of the two storage heads in the Sun ZFS
Storage 7320 appliance. Wait three to five minutes for it to initiate NFS services.

Exalogic Online Checklists
The documentation includes a very thorough set of checklists

to ensure that the site is prepared for an Exalogic machine:
• System Components Checklist
• Data Center Room Checklist
• Data Center Environmental Checklist
• Access Route Checklist
• Facility Power Checklist
• Power Checklist
• Safety Checklist
• Logistics Checklist
• Network Specification Checklist
Exalogic Online Checklists

The System Components Checklist contains questions pertaining to the number of racks
being installed; if they are full, half, or quarter racks; and whether this system will be
connected to another Exalogic machine and, if so, whether the racks are adjacent to each
other.
The Data Center Room Checklist includes questions pertaining to the location of the
Exalogic machine, maintenance access requirements, position of rack in regard to air flow,
raised floor support, and floor-to-ceiling height.
Inside the Data Center Environmental Checklist expect to see questions pertaining to
meeting temperature and humidity requirements, floor layouts and ventilation requirements,
and general air conditioning and airflow requirements.
The Access Route Checklist contains questions that involve whether the doors and
entryways meet height and width requirements (including the unpacked unit), if the ramps
or inclined angles are in the permitted range, whether the pallet jack supports the device
weight, and whether the elevator’s width, height, and weight are supported.

Example: Access Route Checklist
• Has the access route been checked for clearances of the

packaged equipment?
• Do all the doors and entryways conform to the width and
height requirements for transportation, including the width
of the unpacked unit?
• Do all the doors meet the height requirement of 87 inches
for packaged delivery?
• Are there any stairs or ramps in the moving path for the
new hardware?
• If a pallet jack is to be used, have you confirmed whether
the pallet jack supports the device weight?
Example: Access Route Checklist

The Facility Power Checklist contains questions involving power configurations, single-
phase or three-phase power, the quantity of power outlets, confirming proper socket
receptacles, deciding on optional ground cables, confirming that suitable circuit breakers
are installed, and determining if a UPS is going to be used.
The Power Checklist contains questions pertaining to the minimum required power
sources, the location of power outlets, and the power frequency.
The Safety Checklist includes questions involving the location of emergency power shutoff,
the fire protection system, and antistatic flooring.
The Logistics Checklist contains questions involving data center personnel; their security or
access control; where the delivery, unpacking, and staging area is going to be; the moving
personnel; whether there are any restrictions on the delivery truck; and the delivery dock
access.
The Network Specification Checklist contains questions that pertain to reserving unused IP
addresses, registering the IP addresses with DNS, and running and labeling network
cables.

Site Requirements
The documentation includes a comprehensive list of site

requirements.
Component Full Rack Half Rack Quarter Rack
17.575 kW 10.897 kW 7.206 kW

Power
(18.5 kVA) (11.47 kVA) (7.585 kVA)
63124 BTU/hour 39137 BTU/hr 25881 BTU/hr
Cooling
(66533 kJ/hr) (41250 kJ/hr) (27278 kJ/hr)
Physical 1998 mm (h) x 600 mm (w) x 1200 mm (d)
Dimensions 78.66 in (h) x 23.62 in (w) x 47.24 in (d)
966.6 kg 679.5 kg 491.2 kg
Weight
(2131 lbs.) (1498 lbs.) (1083 lbs.)
IP Addresses 98 54 30
Site Requirements
The operating temperature range is 15 degrees Celsius to 32 degrees Celsius (59 degrees
Fahrenheit to 89.6 degrees Fahrenheit). The humidity range is 10 percent to 90 percent
relative humidity, noncondensing.
The minimum ceiling height for the cabinet is 2300 mm (90 inches), measured from the true
floor or raised floor, whichever is higher. An additional 914 mm (36 inches) is for top
clearance. The space above the cabinet and its surroundings must not restrict the
movement of cool air between the air conditioner and the cabinet, or the movement of hot
air coming out of the top of the cabinet. Oracle recommends that the Exalogic machine be
installed on raised flooring. The site floor and the raised flooring must be able to support
the total weight of the Exalogic machine.
The cabinets for the Exalogic machine are shipped with grounding-type power cords (three-
wire). Always connect the cords to grounded power outlets. Because different grounding
methods are used, depending on location, check the grounding type, and refer to
documentation such as IEC documents for the correct grounding method.
Always provide adequate space in front of and behind the rack to allow for proper
ventilation. Do not obstruct the front or rear of the rack with other equipment or objects.

Exalogic Networks
Network Speed Description

• Provides access to compute nodes via
Client 10 Gb
intranet/Internet
• Is used to monitor and administer components
Management 1 Gb • Provides access to ILOM and other device
management interfaces
• Is a private, nonroutable network
• Connects racks, along with all rack components
Private
40 Gb • Is used by compute nodes to access shared
InfiniBand
storage
• Is used for internal cluster communication
Exalogic Networks
Exalogic machine includes compute nodes and the Sun ZFS Storage 7320 appliance, as
well as equipment to connect the compute nodes to your network. The network connections
allow the servers to be administered remotely, enable clients to connect to the compute
nodes, and enable client access to the storage appliance. Additional configuration, such as
defining multiple virtual local area networks (VLANs) or enabling routing, may be required
for the switches to operate properly in your environment. This type of configuration is
beyond the scope of the installation service.
There are up to five networks for Exalogic machine. Each network must be on a distinct
and separate subnet from the others. The Exalogic management network connects to your
existing management network, and is used for administrative work for all components of
Exalogic machine. It connects ILOM, compute nodes, server heads in the storage
appliance, and switches connected to the Ethernet switch in the Exalogic machine rack.
This management network is in a single subnet. Do not use the management network
interface (ETH0/NET0) on compute nodes for client or application network traffic. Cabling
or configuration changes to these interfaces on Exalogic compute nodes is not permitted.

Network Configuration Worksheets
• Oracle can initialize the network settings of Exalogic

machine components based on customer input.
• Complete the worksheets found in the documentation.
Worksheet Inputs
• Domain name
General • Domain Name Service (DNS) server location
• Network Time Protocol (NTP) server location
• Starting IP address
Client
• Subnet mask, gateway address
Management
Private
Network Configuration Worksheets

The Exalogic machine requires a large number of host names and IP addresses during
initial configuration. The number of IP addresses required for a particular network, such as
the management network, depends on the type of system. The network configuration (such
as host names and IP addresses) used during installation is generated from information
you supply to your Oracle technical representative in the completed configuration
worksheet.
Default bonded IP addresses for the private InfiniBand network are assigned to Exalogic
compute nodes during manufacturing. Because these addresses are nonroutable and not
accessible externally, no customer worksheets are required. However, your Oracle
representative can reconfigure these IP addresses by using the Oracle ECU tool, if desired.

Default Addresses for a Full Rack
Network Component Addresses

Client Compute Node 192.168.1.1-14, 17-32
Storage Appliance 192.168.1.15-16
Management Compute Node 192.168.1.101-114, 117-132
IB Switch 192.168.1.201-205
Management Switch 192.168.1.200
PDU 192.168.1.210-211
Private Compute Node 192.168.10.1-14, 17-32
Default Addresses for a Full Rack

Refer to the documentation for the default addresses assigned to full, half, and quarter
Exalogic racks during manufacturing.

Default Host Names
Host names are derived from the assigned rack name (el01,
el02, el03, by default).
Network Component Sample Hostname

Client Compute Node el01cn01
Storage Appliance el01sn01
Management Compute Node el01cn01-c
Storage Appliance el01sn01-c
IB Switch el01sw-ib01
InfiniBand Compute Node el01cn01-priv
Storage Appliance el01sn-priv
Default Hostnames
The Exalogic machine name is used to derive the host names for the network interfaces of
all components. For example, an Exalogic machine name of el01 will result in compute
node host names of el01cn01, el01cn02, el01cn03, and so on. In a multi-rack configuration,
each Exalogic machine name should be unique and identify the rack. Oracle recommends
using el01 for the first Exalogic rack, el02 for the second, el03 for the third, and so on.

Installation Spreadsheet
Oracle produces a spreadsheet with your desired network

configuration.
Management IPs
Private IPs
Client IPs
Installation Spreadsheet
The Exalogic installation spreadsheet is divided into a Configuration tab and a Preview tab.
Entries made on the Configuration tab prepopulate entries on the Preview tab. Entries on
the Preview tab are used as the basis for the configuration that will be generated and used
with ECU. You can also manually edit the Preview values for any fine tuning, although this
is not recommended.
Enter configuration parameters in the General section of the spreadsheet, including the
machine name, and base names for storage and compute nodes. These values are used
as prefixes for host names that will be generated. Enter the domain name, region, and time
zone, as applicable. The Customer Name field determines the location to which the
configuration file will be saved.
Enter configuration parameters in the Network section of the spreadsheet. In this section,
you enter the list of DNS Name Servers, NTP Servers, Search Domains, and Default
Gateway. The start IP Address (Network), Subnet Mask, and Gateway will be the same
across all NET0, ILOM, BOND0, and BOND1 interfaces. As you enter values in these
fields, the corresponding fields on the Preview tab are populated with values.

Exalogic Configuration Utility (ECU)
Oracle ECU (formerly OneCommand):

• Includes a set of scripts and utilities that perform the initial
configuration of the Exalogic machine
• Is intended to be run by an Oracle representative from a
compute node
• Uses data from the spreadsheet so that all components
match your specification
• Can be used only on a machine in its original factory
configuration
Spreadsheet Macro Input File Script
Exalogic Configuration Utility (ECU)

After the Exalogic machine is installed in your data center and powered on, your Oracle
representative will optionally collect information about your specific network configuration
requirements, IP address ranges, host names to be assigned to each compute node, and
so on. This information is entered in a spreadsheet. The data entered in the spreadsheet
provides input to the Oracle Exalogic machine configuration tools, such as the Oracle ECU
tool. This tool includes a set of scripts and utilities that perform the initial configuration of
the Exalogic machine and is run by your Oracle representative on-site. For ECU to work,
the machine must be in a state as configured in the factory. It is not intended to be a tool for
maintaining or re-imaging a rack.
The spreadsheet is an Oracle OpenOffice document with macros that detect events to
prepopulate fields, perform validation, and finally generate an ECU configuration file. The
ECU tools and spreadsheet are available for download from the Oracle E-Delivery site. The
package is named “Configuration Utilities for Exalogic.”

ECU Internals
1. Ping all components by using their factory network

settings.
2. Set up SSH on all compute nodes and distribute keys.
3. Push files to compute nodes.
4. Verify hardware, firmware, and operating system.
5. Test IB topology and connectivity.
6. Update the network settings on all compute nodes.
7. Update the network settings on all IB switches and storage
nodes.
8. Ping all components by using their new network settings.
9. Retest IB topology and connectivity.
10. Configure NTP on all nodes.
ECU Internals
If you are running ECU for the first time, it generates a file representing the factory
configuration of the rack. Review the exalogic_current.conf and
exalogic_pending.conf files to ensure that they are accurate. Verify that the current
configuration has the correct number of nodes and switches and the numbers match with
the pending configuration. Verify IP addresses, host names, and such network parameters
in both files.
ECU verifies ILOM for all compute nodes listed in the exalogic_current.conf file. The
master node is omitted from this process because it cannot ping its own ILOM when
sideband management is used. ECU also verifies entries corresponding to the storage
heads in the exalogic_current.conf file and validates that their corresponding ILOMs
can be pinged as well.
ECU verifies that each Sun Network QDR InfiniBand Gateway Switch (NM2-GW) is running
a Subnet Manager (SM). If a switch is not running an SM, an SM is dynamically started on
that switch. It also runs the verify-topology utility, which validates the network cabling.
ECU distributes keys to each compute node and allows subsequent steps to operate more
efficiently by not prompting for a password. After this step runs, accessing remote compute
nodes via SSH will not require a password.

InfiniBand Networking Concepts
Term Definition Description

IPoIB IP over InfiniBand Applications connected by an IB fabric communicate
by using standard IP address semantics.
SDP Socket Direct Protocol Applications communicate directly with the IB fabric,
bypassing the operating system's TCP/IP stack.
EoIB Ethernet over Applications within an IB fabric communicate with

InfiniBand external Ethernet networks.
vNIC Virtual Network Software that emulates an Ethernet NIC on the IB
Interface Card network
InfiniBand Networking Concepts

IP over InfiniBand (IPoIB) enables applications on separate devices to communicate with
each other over a private InfiniBand fabric by using native IB protocols and without the
overhead of Ethernet. For example, a compute node on one Exalogic rack may
communicate with a database on an Exadata rack. However, applications must support the
SDP protocol to utilize IPoIB instead of the default TCP/IP stack of the host operating
system.
The InfiniBand switches also act as gateways to connect to external Ethernet networks.
They support eight 10 Gb Ethernet ports. Exalogic compute nodes can communicate
through these ports by using Ethernet over InfiniBand (EoIB). Each port is represented on
the compute nodes as a vNIC. This allows that node's IB connection to appear like any
other Ethernet NIC to both the operating system and to the external Ethernet network.

Bonded Interfaces
• Compute and storage nodes use two physical IB ports and

switches for redundancy (active/passive).
• A special network adapter makes these two interfaces
appear as a single interface to applications:
– Linux: Bonded interface
– Solaris: IP multi-path (IPMP) group
• Different bonded interfaces are used for IPoIB and EoIB.
IPoIB, SDP
bond0
Switch1 IB0
Apps
EoIB vNIC0
Switch2 IB1
bond1
vNIC1
Compute Node
Bonded Interfaces
The Exalogic compute nodes and storage appliance include a Host Channel Adapter
(HCA) to communicate with an InfiniBand network. The HCA consists of two ports, IB0 and
IB1. These raw IB connections are represented on the operating system as the interfaces
ib0 and ib1. The operating system and applications are generally not intended to use these
interfaces directly.
InfiniBand switches (both leaf and spine switches) are configured to automatically separate
the IPoIB traffic and the EoIB traffic. After the InfiniBand gateway switches are connected
to the Exalogic compute nodes, the following bonded interfaces are defined:
• bond0 for IPoIB traffic and for using SDP
• bond1 for EoIB traffic (uses two vNICs to communicate through ib0 and ib1)
Each HCA has an IP address, and active/passive bonding is configured. The active port of
the HCA connects to an IB switch and the passive port of the HCA connects to another
switch. If one connection fails, traffic is transparently failed over to the other.
Solaris IPMP provides the same functionality as bonded Interfaces on Oracle Linux. Oracle
recommends that you name your IPMP groups bond0 and bond1 to keep the terminology
consistent with Oracle Linux.

Default Network Configuration
Private IB
Network
IPoIB, Compute Node

SDP
IB IB0 bond0
Switches IB1 bond1 eth0
EoIB
NET3 NET2 NET1 NET0 MGMT
Client Ethernet
Network
Mgmt Mgmt
Network Switch
Default Network Configuration

By default, each Exalogic compute node is configured with one bonded EoIB interface for
one external LAN (client network), and is named bond1.
The Cisco Ethernet management switch is connected to the NET0 port of compute nodes,
the NET0 port of the storage appliance, and also the management ports of the InfiniBand
gateway switches. On the compute nodes, this connection is represented on the operating
system by an “eth” network interface, such as eth0. The compute nodes are configured at
the time of manufacturing to use sideband management only. Therefore, the MGMT (or
ILOM) port is not connected, but ILOM is accessible from NET0.
Two physical IB ports (0A and 1A) on each switch are used to host the eight virtual
Ethernet ports. For example, port 0A comprises 0A-ETH-0, 0A-ETH-1, 0A-ETH-2, and 0A-
ETH-3. Splitter cables are used to route each of these two IB ports to existing Ethernet
networks.
The number of Ethernet ports used in Exalogic deployment depends on your specific
bandwidth requirements (how many 10 Gb ports can be shared per compute node) and on
your specific LAN/VLAN connection requirements. You can configure additional EoIB
network interfaces for connecting to additional LANs, as required.

Exalogic Diagnostic Utilities
• The base compute node OS image includes

/opt/exalogic.tools.
• You can use these tools to:
– Verify IB configuration for full, half, and quarter racks
– Verify IB network connectivity to other components
– Determine which IB ports are currently active
– Verify compute node hardware and firmware
– Verify BIOS settings and OS packages
• Additional third-party IB tools are found at /usr/sbin.

Utility Examples
# check_ibports
...
Port 1 is in the ACTIVE state
# verify-topology –t quarterrack
Quarter rack expected to have 2 internal switches but has 1
... [FAILURE]
Are any hosts connected to spine switch... [SUCCESS]
[ERROR] node05 appears to be disconnected from the fabric
# CheckSWProfile
[SUCCESS]....Has supported operating system
[SUCCCES]....Has supported processor
[SUCCESS]....Kernel is at the supported version
[SUCCESS]....Has supported kernel architecture
[SUCCESS]....Software is at the supported profile
Utility Examples
The check_ibports tool determines which of the two ports on the local IB Host Channel
Adapter (HCA) is active.
The verify-topology tool validates the expected cabling and connectivity for a full, half, or
quarter rack configuration.
The CheckSWProfile tool validates the current compute node’s operating system, kernel,
software packages, and hardware architecture for supported levels.

Distributed Command Line Interface (DCLI)
The dcli utility:

• Enables you to execute a command on multiple nodes
from a single location
• Can be used to create folders, change permissions, mount
file systems, run diagnostics, and so on
• Can also upload the file to execute onto each node
• Prompts for each node's credentials unless SSH user
equivalence is configured or the SSH key is exchanged
Distributed Command Line Interface (DCLI)

The dcli utility is part of the base OS image and is found at /opt/exalogic.tools. You
can use the DCLI utility to execute commands or scripts in parallel across a defined set of
host names. It simplifies any operations that must be run across a subset or all compute
nodes. Configuration of SSH user equivalency across all cells is an important prerequisite
for optimizing the use of DCLI. For convenience, the tool also provides the -k option to
automate the distribution of SSH private keys. Internally the Oracle ECU tool utilizes DCLI
to perform several of its tasks.
DCLI also captures any output from each compute node and prints it to the shell from which
DCLI was run. The output from each individual compute node is prefixed by that particular
compute node's host name.

DCLI Examples
Configure SSH user equivalence
setup-ssh –H el01cn02 –P mypassword

setup-ssh –H el01cn03 –P mypassword
...
Execute a command on the specified nodes
dcli –c el01cn02,el01cn03 "mount –a –t nfs"
Upload and execute a script on nodes defined in an external file
dcli –g nodes.txt –x configMailServer.sh
DCLI Examples
To configure SSH user equivalence between the host from which DCLI will be run and a
target host, use the setup-ssh utility, which is found in the same location as DCLI.
To give DCLI a list of hosts on which to run the specified command, supply a comma-
separated list with the -c option or supply the location of a file that lists the host names
with the -g option. Then either give a command to run, surrounded by quotation marks, or
provide the name of a script file with the –x argument. The script file will be uploaded to the
user’s home folder on each target host before being executed.

Quiz
What tool is used by Oracle at a customer site to initialize an

Exalogic machine?
a. EME
b. ECU
c. ERU
d. ECE
Answer: b

Quiz
A full rack weighs twice as much as a half rack.

a. True
b. False
Answer: b

Quiz
Which of the following is not a default network associated with

the Exalogic machine?
a. Management network
b. Client Access Network
c. Template network
d. InfiniBand private network
Answer: c

Summary

• Explain how Oracle initializes an Exalogic machine based
on your input
• Describe the Exalogic default network and storage setup
• List some of the capabilities of ILOM
• Customize the Exalogic storage configuration

Practice 4-1:
Connect to a Compute Node
This practice covers the following topics:
• Accessing a compute node's operating system
• Inspecting the default network and file system settings
• Creating a user account for FMW installations
• Executing a command on multiple compute nodes

Fusion Middleware Concepts

Objectives

• Describe a typical Java EE deployment
• Discuss the WebLogic domain architecture
• Explain the purpose of the WebLogic Node Manager
• Describe the WebLogic recovery services

Middleware Concepts
Middleware:
• Extends the traditional client/server software architecture
• Is the software that sits between clients, business
applications, and business data
• Should be invisible to users
• Is a key component of distributed enterprise systems
• Enables dynamic, service-oriented architectures (SOA)
• Enables you to focus more on your custom business
requirements and less on your infrastructure
Middleware Concepts
Middleware is the software that connects software components or enterprise applications.
Middleware is the software layer that lies between the operating system and the applications
on each side of a distributed computer network. Typically, it supports complex, distributed
business software applications. It is especially integral to information technology that is based
on web services and service-oriented architecture (SOA). Middleware makes application
development easier by providing common programming abstractions, by masking application
heterogeneity and the distribution of the underlying hardware and operating systems, and by
hiding low-level programming details.
Because of the continued growth and use of network-based applications by businesses,
middleware technologies have become increasingly important. Companies and organizations
are now building enterprise-wide information systems by integrating previously independent
applications with new software developments. Middleware can provide uniform, standard, and
high-level interfaces to application developers and integrators so that applications can easily
be composed, reused, ported, and made to interoperate.

Oracle Fusion Middleware (FMW) Review
Most components of FMW:

• Are written in Java and require a JVM
• Require a Java EE application server such as WebLogic
Server
Web Identity Content

SOA ...
Center Mgmt Mgmt
WebLogic Server Coherence
Java Virtual Machine (JVM) WebTier
OS/Hardware
Oracle Fusion Middleware (FMW) Review

FMW products are organized into two general categories.
• Java components: Deployed as one or more Java EE applications and a set of
resources. Java components are deployed to an Oracle WebLogic Server domain as
part of a domain template. Examples of Java components are the Oracle SOA Suite and
Oracle WebCenter components.
• System components: Not deployed as Java applications. Instead, a system
component is managed by the Oracle Process Manager and Notification server
(OPMN). A Java component and a system component are peers. Examples of FMW
system components are Oracle HTTP Server, Oracle Web Cache, Oracle Internet
Directory, and Oracle Business Intelligence.

WebLogic Server (WLS) Overview
WebLogic Server:
• Is a Java application
• Hosts Java EE applications
• Can be administered from a web-based console or a
scripting interface
• Provides clustering services for load distribution and high
availability
• Includes a robust and high-performance enterprise
messaging infrastructure
• Offers an extensible security realm for authentication,
authorization, credential mapping, and so on
WebLogic Server (WLS) Overview

Oracle WebLogic Server is a scalable, enterprise-ready Java Platform, Enterprise Edition
(Java EE) application server. The WebLogic Server infrastructure supports the deployment of
many types of distributed applications and is an ideal foundation for building applications
based on SOA. In addition to the Java EE implementation, WebLogic Server enables
enterprises to deploy mission-critical applications in a robust, secure, highly available, and
scalable environment. These features enable enterprises to configure clusters of WebLogic
Server instances to distribute load, and provide extra capacity in case of hardware or other
failures. New diagnostic tools enable system administrators to monitor and tune the
performance of deployed applications and the WebLogic Server environment itself. Extensive
security features protect access to services, keep enterprise data secure, and prevent
malicious attacks.
WebLogic Server provides many tools for system administrators to help with these tasks,
including a browser-based Administration Console, the WebLogic Scripting Tool (WLST), a
scripting language for automation of WebLogic system administration tasks based on Jython.
WLS also supports SNMP.

Java EE Deployment: Example
Web
Application
Server
EJB
Application
Persistence
Server
Web Service
XML
Application
Server EJB
Message Queue
Application
Server Server
Java EE Deployment: Example

Users interact with a web application by using a browser. The web application is responsible
for rendering the website and for capturing user input through buttons, forms, links, and so on.
The web application itself may contain all of the necessary business logic to perform the tasks
that users request, such as retrieving the details of a product, processing an order, or filing an
insurance claim. But instead the web application may access Enterprise JavaBeans (EJB)
applications to perform the business logic. These EJB applications may be located on the
same server as the web application or on a different server.
Java EE containers like WebLogic Server provide additional services for EJB, including
persistence. For example, WebLogic could be responsible for writing a newly placed order to
a relational database. Similarly, WLS could update multiple databases within a single
transaction. WebLogic also supports the Java Message Service (JMS) to enable reliable,
asynchronous communication between applications.
If you want to expose business logic in a standard way to other applications, both within your
organization and beyond, use a web service application. XML-based web services can be
accessed by both Java and non-Java applications and are also a cornerstone for SOA.

WLS Architecture Overview
• Server processes are grouped into logical domains.

• Each domain has an administration server that is used to
start, configure, manage, and monitor other servers.
Machine Machine Machine
Domain B
Domain A Domain A Domain B
Admin
Server Server Server
Admin
Server Server
Server
Product Product
Installation Product Installation Installation
WLS Architecture Overview

An Oracle WebLogic Server administration domain is a logically related group of Oracle
WebLogic Server resources. Domains include a special Oracle WebLogic Server instance
called the Administration Server, which is the central point from which you configure and
manage all resources in the domain. Usually, you configure a domain to include additional
Oracle WebLogic Server instances called managed servers. You deploy web applications,
EJB, web services, and other resources onto the managed servers and use the
Administration Server for configuration and management purposes only.
You can use a single Oracle WebLogic Server installation to create and run multiple domains,
or you can use multiple installations to run a single domain. How you organize your Oracle
WebLogic Server installations into domains depends on your business needs. You can define
multiple domains based on different system administrators’ responsibilities, application
boundaries, or geographical locations of the machines on which servers run. Conversely, you
might decide to use a single domain to centralize all Oracle WebLogic Server administration
activities.
For development or test environments, you can create a simple domain that consists of a
single server instance. This single instance acts as an Administration Server and hosts the
applications that you are developing.

WLS Domain Contents
• A domain is a directory structure that contains:

– Configuration files managed by the admin server
– Server start scripts
– LDAP files
• Each server in the domain creates a separate subdirectory
that contains:
– Log files
MyDomain
– Recovery files
bin
config
servers
WLS Domain Contents

The domain directory must have Write privileges, as well as Read privileges, for the domain to
function properly, even if no changes are made to the configuration after it is created.
The bin directory contains scripts that are used in the process of starting and stopping the
Administration Server and the managed servers in the domain. These scripts are generally
provided as .sh files for UNIX and .cmd files for Windows.
Each domain describes its configuration in an XML document that is located in the domain's
configuration directory. At run time, each Oracle WebLogic Server instance in a given domain
creates an in-memory representation of the configuration described in this document. The
central configuration file for a domain is DOMAIN_NAME/config/config.xml. This file
specifies the name of the domain and the configuration of each server instance, cluster,
resource, and service in the domain. The file includes references to additional XML files that
are stored in subdirectories of the DOMAIN_NAME/config directory.
The servers/SERVER_NAME/logs directory holds logs and diagnostic information. The
servers/SERVER_NAME/store directory holds WebLogic persistent stores, which are used
for failover and recovery of transactions and messages. For each persistent store, there is a
separate subdirectory.

WLS Machines and Compute Nodes
WLS allows you to define logical machines that:

• Denote hardware boundaries
• Are assigned one or more servers
• Help WLS make failover and migration decisions
• Typically correspond to compute nodes on Exalogic
Compute Node 1 Compute Node 2
Domain
Machine 1 Machine 2
Server Server
Server Server
WLS Machines and Compute Nodes

WLS machine definitions identify a particular physical piece of hardware. A machine is used
to indicate which servers are colocated on the same hardware. WLS cannot simply use host
names or IP addresses to accurately determine this, so machines are explicitly defined. This
information is used by the WLS Node Manager when restarting or migrating a failed server,
and also by clustered servers when selecting the best location for storing replicated data.
Note that the WLS machine name does not need to be a valid host name or listen address. It
is just a unique logical identifier. However, to avoid confusion, Oracle recommends that you
give a machine the same name as the compute node it is intended to represent. Also avoid
confusing the concept of a WLS machine with an Exalogic machine, which refers to the
physical rack containing multiple compute nodes.

Node Manager Concepts
Node Manager:
• Is a process that accepts remote commands to start, stop,
or suspend servers on the same machine
• Monitors server availability and restarts failed ones
• Can be used to migrate servers on a failed machine to
another machine
Machine
Server
Node Start/stop
Manager Monitor/
restart Server
Node Manager Concepts

Server instances in a WebLogic Server production environment are often distributed across
multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server
utility that enables you to start, shut down, and restart Administration Server and Managed
Server instances from a remote location. Although Node Manager is optional, it is
recommended if your WebLogic Server environment hosts applications with high-availability
requirements. A Node Manager process is not associated with a specific WebLogic domain
but with a machine. You can use the same Node Manager process to control server instances
in any WebLogic Server domain, as long as the server instances reside on the same machine
as the Node Manager process.
If a server instance that was started by using Node Manager fails, Node Manager
automatically restarts it. If Node Manager fails or is explicitly shut down, upon restart, it
determines the server instances that were under its control when it exited. Node Manager can
restart any failed server instances as needed.
The Administration Console can be used to issue commands to node managers running on
remote machines. WLST (in offline mode) also serves as a Node Manager command-line
interface that can run in the absence of a running Administration Server.

Transaction Recovery
• Applications can use WebLogic to orchestrate distributed

transactions that involve multiple databases, messages,
and so on.
• Transactions ensure that all operations either succeed or
fail.
• WebLogic logs in-progress transactions to the file system.
• WebLogic uses these logs to recover from system crashes
or network failures.
Server
TLOGs
Transaction Recovery
WebLogic Server supports distributed transactions and the two-phase commit protocol for
enterprise applications. A distributed transaction is a transaction that updates multiple
resource managers (such as databases) in a coordinated manner. In contrast, a local
transaction begins and commits the transaction to a single resource manager that internally
coordinates API calls; there is no transaction manager. The two-phase commit protocol is a
method of coordinating a single transaction across two or more resource managers. It
guarantees data integrity by ensuring that transactional updates are committed in all of the
participating databases, or are fully rolled back out of all the databases, reverting to the state
prior to the start of the transaction. In other words, either all the participating databases are
updated, or none are updated.
WLS executes a recovery procedure when failures occur. It determines which transactions
were active in the machine at the time of the crash, and then determines whether the
transaction should be rolled back or committed. Because this service is designed to gracefully
handle transaction recovery after a crash, Oracle recommends that you attempt to restart a
crashed server and allow the service to handle incomplete transactions. When moving
transaction log records after a server failure, make all transaction log records available on the
new machine before starting the server there.

Message Recovery
• WebLogic records in-progress JMS messages to either:

– The file system (default)
– A database
• If WebLogic and/or a client fails, the server can recover
and not lose messages.
• Only messages that the application or administrator flags
as “persistent” are saved.
Server
Producer Message Server Consumer
Store
Message Recovery
An enterprise messaging system enables applications to communicate with one another
through the exchange of messages. A message is a request, report, and/or event that
contains information needed to coordinate communication between different applications.
Java Message Service (JMS) servers reside on a WebLogic Server instance, host a defined
set of JMS modules, and also define any associated persistent storage for recovery.
A persistent message is guaranteed to be delivered once-and-only-once. The message
cannot be lost due to a JMS provider failure and it must not be delivered twice. It is not
considered sent until it has been safely written to a file or database. Non-persistent messages
are not stored. They are guaranteed to be delivered at-most-once, unless there is a JMS
provider failure, in which case messages may be lost, and must not be delivered twice.
For storing persistent messages, you can simply use the host server's default persistent file
store, which requires no configuration on your part. However, you can also create a dedicated
file-based store or JDBC store for JMS. A file store maintains subsystem data, such as
persistent JMS messages and durable subscribers, in a group of files in a directory. A JDBC
store maintains subsystem data in a relational database.

WLS Cluster Architecture
A WLS cluster supports additional features:

• To provide high availability for applications and services
• To perform load balancing and failover
• That are transparent to both applications and clients
Domain
Hardware or
software Cluster
Server 1
Web client Proxy
Server 2
EJB client
JMS client Server n
WLS Cluster Architecture

one instance, whether the client is a web client or a Java application. By replicating the
WebLogic Server provides clustering support for web applications by replicating the HTTP
session state of clients. You can balance the web application load across a cluster by using a
WebLogic Server proxy plug-in or the external load-balancing hardware.
Failover for EJB objects is accomplished by using the object’s replica-aware stub. When a
client makes a call through a replica-aware stub to a service that fails, the stub detects the
failure and retries the call on another replica.
WLS also supports clustering for JMS applications. Applications that use a distributed
destination are more highly available than applications that use stand-alone destinations
because WebLogic JMS provides load balancing and failover for the members of a distributed
destination in a cluster.

WLS Exalogic Topology: Example
Compute Node A
Admin Server Domain
Server 1 Server 2 Server 3

Node
Manager
Compute Node B
Server10 Server11 Server12
Admin
Cluster Server Node
Manager
WLS Exalogic Topology: Example

The example in the slide shows a single WebLogic domain that spans two Exalogic compute
nodes. Each compute node or “machine” has a single running node manager to support all
WebLogic (and optionally Coherence) processes on that compute node. The domain defines
a WebLogic administration server and 12 managed servers. The managed servers are all
members of a single WebLogic cluster and are distributed across the two compute nodes to
provide high availability. However, remember that to completely utilize the capacity of a
compute node, you should try running one server for each core, or 24 servers for two compute
nodes. For the example system in the slide, however, 12 servers provided sufficient
performance.

WLS Exalogic Capacity Planning
• Exalogic was designed to host one WLS process on each

CPU core:
– A full rack can support up to 360 WLS processes.
– A half rack can support up to 180 WLS processes.
– Ignore admin servers and node managers, which do not
process significant loads.
– If CPU usage is low and RAM is available, add more WLS
processes.
• Most Java EE applications are I/O driven, but very CPU-
intensive WLS processes may benefit from multiple cores.
• As always, you should determine optimal performance
settings for your WLS cluster experimentally.
WLS Exalogic Capacity Planning

An instance of WLS running a typical Web application on a 64-bit platform will utilize about 4
to 6 GB of RAM under peak load. However, a single Exalogic compute node has 96 GB of
RAM. Therefore, it is likely that even if you run one WLS instance per compute node core, you
will still have ample RAM to run additional WLS instances if CPU utilization is not high.
The most commonly tuned performance characteristic of any Java process is the heap size.
The heap represents the amount of RAM that is available to the JVM for storing objects. The
default heap size for WLS is 512 MB, but larger settings may increase performance and
throughput. You can override the default setting by editing setDomainEnv.sh or by defining
an environment variable named USER_MEM_ARGS.

Quiz
Which is not a component of a WebLogic domain?

a. Machine
b. Proxy
c. Cluster
d. Server
Answer: b

Quiz
Which WLS feature enables you to start servers remotely?

a. Cluster Manager
b. Process Manager
c. Build Manager
d. Node Manager
Answer: d

Quiz
Name two types of WebLogic services that are logged for

failover and recovery.
a. Beans
b. Requests
c. Messages
d. Applications
e. Transactions
Answer: c, e

Summary

• Describe a typical Java EE deployment
• Discuss the WebLogic domain architecture
• Explain the purpose of the WebLogic Node Manager
• Describe the WebLogic recovery services

Storage Configuration

Objectives

• List the capabilities of the storage appliance
• Perform basic administration and maintenance tasks
• Configure appliance access services
• Describe Exalogic’s default storage setup
• Create pools, projects, and shares
• Mount a shared file system by using NFS
• Administer the appliance from the command line
• Describe the recommended configuration for FMW

ZFS Appliance Capabilities
Key features of the Exalogic storage appliance include:

• Browser, command-line, and scripting interfaces
• Redundancy at all levels and for all paths
• Low-cost SAS and SATA hard disks
• Integrated SSDs for read and write caching
• Low-impact data compression and virus scanning
• Support for a variety of client protocols
• Template-driven configuration
• Customizable alerts and analytics
• Generation of support tickets
ZFS Appliance Capabilities

The Sun ZFS Storage 7000 family of products provides efficient file and block data services to
clients over a network, and a rich set of data services that can be applied to the data stored
on the system. For example, the appliance can optionally compress data before writing to the
storage pool. This allows for much greater storage utilization at the expense of increased
CPU utilization.
The ZFS Hybrid Storage Pool is composed of optional Flash-memory devices for acceleration
of reads and writes; low-power, high-capacity disks; and DRAM memory, all managed
transparently as a single data hierarchy.
To maximize the availability of your data in production, Sun ZFS Storage appliances include a
complete end-to-end architecture for data integrity, including redundancies at every level of
the stack. Key features include:
• Predictive self-healing and diagnosis of all system hardware failures: CPUs, DRAM, I/O
cards, disks, fans, power supplies
• Multiple I/O paths between the controller and disk shelves
• Phone Home service to report telemetry to Oracle support during all software and
hardware issues

Administrative Tasks
Core administrative tasks for the appliance include:

• Determine system status and health
• Identify and replace failed hardware
• Configure access protocols
• Configure storage pools and shares
• Configure users and access rights
• Back up or restore your configuration settings
• Monitor system load and performance
Administrative Tasks
Analytics provides real-time graphs of various statistics, which can be saved for later viewing.
About a dozen high-level statistics are provided, such as NFSv3 operations/sec, which can
then be customized to provide lower-level details. Groups of viewed statistics can be saved as
worksheets for future reference.

Getting Started
1. Power on the appliance.

2. (Optional) Configure initial IP addresses, DNS locations,
and root credentials by using OneCommand.
3. Connect by using SSH and enter start /SP/console.
4. Verify or update the configuration settings.
5. Continue the initial setup steps by using the browser or
command-line interfaces.
6. Verify or update the configuration settings a final time.
Getting Started
All stand-alone controllers should have at least one NIC port configured as a management
interface. Select the Allow Admin option in the browser user interface (BUI) to enable BUI
connections on port 215 and CLI connections on SSH port 22. Gather the following
information in preparation for configuring an Ethernet interface on the storage controller: IP
address, IP netmask, host name, DNS domain name, DNS server IP address, and gateway IP
address.
You can access the system by connecting an administrative client to the serial management
port on the controller. Alternatively, if a Dynamic Host Configuration Protocol (DHCP) server is
on the network, you can access the system by connecting the network management port to
your network. For a serial connection, use telnet. For an Ethernet connection, use SSH. At the
command prompt, type start /SP/console. After you configure the primary interface,
configure the remaining system parameters by using the BUI from any client on the same
network.
Note that if you are installing a cluster, configure only one controller initially. The software
propagates the configuration to the peer controller during cluster initialization. After the cluster
is initialized, you can administer the system from either storage controller. However, do not
attempt initial configuration on both controllers independently.

Accessing the Browser User Interface (BUI)
• The default port is 215 and the default account is root.

• A help menu is also available at /wiki.
Main menu
Power off or
reboot appliance.
Accessing the Browser User Interface (BUI)

The BUI is the graphical tool for administration of the appliance. The BUI provides an intuitive
environment for administration tasks, visualizing concepts, and analyzing performance data.
Direct your browser to the system by using either the IP address or host name that you
assigned to the NET-0 port during initial configuration as follows: https://ipaddress:215 or
https://hostname:215. The login screen appears. Internet Explorer 6 and earlier versions have
known issues and are unsupported, and login will not complete.
The online help linked in the top right of the BUI is context-sensitive. For every top-level and
second-level screen in the BUI, the associated help page appears when you click the Help
button.
The masthead contains several interface elements for navigation and notification, as well as
primary functionality. At the left, from top to bottom, are the company logo, a hardware model
badge, and hardware power-off and restart buttons. Across the right, again from top to
bottom, are login identification, logout, help, and main navigation. The title bar appears below
the masthead and provides local navigation and functions that vary depending on the current
view.
System alerts appear in the masthead as they are triggered. If multiple alerts are triggered
sequentially, refer to the list of recent alerts found on the Dashboard screen or the full log
available on the Maintenance > Logs screen.

Viewing Appliance Status
Select a pool. 1
2

The Status > Dashboard screen provides a view of storage, memory, services, hardware,
activity, and recent alerts. The Status > Settings screen enables you to change the graphs
that appear on the dashboard and to customize the threshold settings associated with the
weather icons shown for each graph on the dashboard.
The activity area of the dashboard shows graphs of eight performance statistics by default.
The statistical average is plotted in blue and the maximum appears in light gray. To go to the
Analytics worksheet for an activity, click one of the four graphs (day, hour, minute, second) for
the statistic that you want to evaluate. To view the average for each graph, move the cursor
over a graph and the average appears in the tooltip. The weather icon in the upper left
provides a report of activity according to thresholds that you can customize for each statistic.
The weather icon is intended to grab your attention when something is unusually busy or idle.
The vertical scale of all graphs is printed at the top right, and all graphs are scaled to this
same height. The height is calculated from the selected graph (plus a margin). The height will
rescale based on activity in the selected graph, with the exception of utilization graphs, which
have a fixed height of 100%.

Service
status
Disk usage
Click for more details
Memory
usage
Hardware
availability
Click to
view full log.
Viewing Appliance Status (continued)

The Usage area of the dashboard provides a summary of your storage pool and main
memory usage. The pool name may be clicked to change the pool that is displayed on the
status screen, should multiple pools be configured on the appliance. The name of the pool
appears at the top right of the Usage area. To the left is a pie chart of used and available
space. To go to the Shares screen for the pool, click the Storage pie chart. To the left is a pie
chart that shows memory usage by component. To go to the Analytics worksheet for dynamic
memory usage broken down by application name, click the Memory pie chart.
The Services area of the dashboard shows the status of services on the appliance, with a light
icon to indicate the status of each service. Most services will be green, indicating that the
service is online, or gray, indicating that the service is disabled.
The Hardware area of the dashboard shows a status overview of the hardware components
on the appliance, including CPUs, disks, and fans. If there is a known fault, an amber icon is
displayed. To go to the Hardware Maintenance screen for a detailed look at hardware state,
click the name of a hardware component.
The Recent Alerts section shows the last four appliance alerts. Click the box to go to the
Maintenance Logs screen to examine all recent alerts in detail.

Viewing Network Settings
1
2
3
Ethernet
port status
View/edit addresses
IB port status
Viewing Network Settings

The networking configuration features enable you to create a variety of advanced networking
setups from your physical network ports, including link aggregations, virtual LANs (VLANs), IB
partitions, and IP multipathing (IPMP) groups. You can then define any number of IPv4 and
IPv6 addresses for these abstractions, for use in connecting to the various data services on
the system.
Devices are created by the system to represent the available network or InfiniBand ports.
They have no configuration settings of their own. Data links are required to complete the
network configuration, whether they apply specific settings to the network devices or not.
Interfaces assign IP addresses to the data links.
The Configuration page is shown by default, and lists Devices, Datalinks, and Interfaces,
along with buttons for administration. The Devices list shows link status on the right, as well
as an icon to reflect the state of the network port. To configure an IP address on a network
device, first create a data link, and then create an interface to use that data link. You can use
the ‘+’ icon for this task, but the interface also supports drag-and-drop. For example, you can
configure a data link and drag it onto the Interfaces table.
The Addresses page shows a summary table of the current network configuration. The
Routing page provides configuration of the IP routing table and associated properties.

Some Supported Services
Category Service Description

Access SSH Command-line admin interface
Data NFS File system access via NFS v3 or v4 protocols
FTP File system access via FTP or FTPS protocols
SFTP File system access via SFTP protocol
HTTP File system access via WebDAV over HTTP/HTTPS
SMB File system access via SMB protocol
Replication Replicate data to another remote storage appliance.
Migration Move new and existing NFS clients to another server.
Virus Scan Scan updated files and quarantine any infected ones.
Directory LDAP Authenticate users via an external LDAP directory.
AD Authenticate users via an MS Active Directory server.
System SNMP System access by using SNMP
Some Supported Services

The Secure Shell (SSH ) service allows users to log in to the appliance CLI and perform most
of the same administrative actions that can be performed in the BUI. The SSH service can
also be used as means of executing automated scripts from a remote host.
Network file system (NFS) is an industry-standard protocol to share files over a network. NFS
versions 2, 3, and 4 are supported. Setting the NFS minimum and maximum versions to the
same value will cause the appliance to communicate only with clients who are using that
version. This may be useful if you find an issue with one NFS version or the other (such as
the performance characteristics of that NFS version with your workload), and want to force
clients to use only the version that works best.
The FTP service allows file system access from FTP clients. Anonymous logins are not
allowed, and users must authenticate. The SSH File Transfer Protocol (SFTP) service allows
file system access from SFTP clients. The SFTP service uses a nonstandard port number for
connections to the appliance. This is to avoid conflicts with administrative SSH connections to
port 22. By default, the SFTP port is 218 and must be specified on the SFTP client before
connecting.
The HTTP service provides access to file systems by using the HTTP and HTTPS protocols
and the HTTP extension, Web-based Distributed Authoring and Versioning (WebDAV).

Configuring File System Services
1
2
Enable, disable, or
restart service.
4
Configure
service settings.
Configuring File System Services

1. Click Configuration.
2. Click Services.
3. Locate the list of Data Services. Select a service to edit its configuration. You can also
enable and disable the service at the appliance level.
4. Edit the available settings for the selected service.
NFS settings include:
- Minimum/maximum supported version: Controls which versions of NFS are
supported
- Grace period: Seconds that all clients have to reclaim locks after an appliance
reboot. During this period, the NFS service only processes reclaims of old locks.
All other requests for service must wait until the grace period is over.
FTP settings include:
- Maximum # of connections: Set this to cover the anticipated number of
concurrent users. By default this is 30, and 0 implies unlimited.
- Default login root: The default is “/” and points to the top of the shares hierarchy.

Identifying Hardware Components
1
2 Power off
or restart.
3
el01stor01
Turn on
locator LED.

The Maintenance > Hardware screen of the BUI provides visual representations of the
physical system components, enabling you to identify and locate hardware components
visually and verify their status. A green icon indicates a good status, amber indicates a fault,
and gray indicates a component that is not present.
The main hardware page lists the system chassis, a summary of its contents, and any
attached disk shelves (on supported systems). This provides an overview of the hardware
present on the system, as well as controls to locate, reset, or power off the system.
A thumbnail of the controller is presented at the left. Clicking the thumbnail or the Show
Details link takes you to a detailed view of the chassis and is identical to clicking the right-
pointing arrow at the top left of the view.

Location of selected
component
Change
perspective.
View specifications.
Take disk offline. Turn on disk

Component status locator LEDs.
Identifying Hardware Components (continued)

To view the chassis details, click the arrow icon. At the left is a set of images describing the
chassis. If there are multiple views, you can switch between them by clicking the name of the
view above the image. For each view, faulted components will be highlighted in red. In
addition, the currently selected component will be highlighted in the image. Clicking a
component within the image selects the corresponding component in the list to the right.
A tab is present for each component type in the following list. Each component type has a
state icon, which will be orange if there is a faulted component of the given type. Clicking the
information icon while over a row or double-clicking a row opens a dialog box with detailed
information about the component. The information displayed in the list depends on the
component type, but typically includes the manufacturer, model, part number, serial number,
revision number, and size/speed.
Disk components also provide additional icons to toggle the locate indicator for the disk or to
take the disk offline. This option is available only for disks that are part of a configured storage
pool (including the system pool). Taking a disk offline prevents the system from reading or
writing to it. Faulted devices are automatically avoided.

Replacing Failed Hardware
• See the Sun ZFS Storage Appliance Customer Service

Manual for detailed procedures about disks, power
supplies, fans, and so on.
• For example, to replace a drive on the disk shelf:
1. Take the disk offline.
2. Press the release button on the front of the disk.
3. Pull the ejection lever and remove the old disk.
4. Insert the new disk and verify that the LED indicator turns
green.

The shelf drives are hot swappable and can be replaced without removing power from the
shelf. The replacement drive must be of the same capacity and type as the drive to be
replaced. To avoid possible data loss when removing nonfaulted drives, label each drive with
the number of the slot from which it was removed and reinstall each drive into the same slot.
Faulted drives are indicated by an amber LED.
The power supplies are hot-swappable and can be replaced without removing power from the
system. The power supplies can produce a high-energy hazard and should be replaced only
by instructed individuals with authorized access to the equipment. Failed power supplies are
indicated by an amber LED on the rear panel. To replace a power supply:
1. Ensure that the power supply on/off switch is in the off position.
2. Disconnect the power cord tie strap from the power cord and unplug the power cord
from the power supply.
3. Using your thumb and forefinger, unscrew both ejection screws until loose and swing
the ejection arms out until they are fully open.
4. Pull the power supply out of the chassis, being careful not to damage the circuit board
connector extending from the back of the power supply.
5. With the ejection arms fully open, slide the new power supply into the chassis slot.

Performing Failover Actions
• In Exalogic, storage heads are in an active/passive state

by default.
• You can manually fail over to the other storage head for
maintenance purposes.
1
2
3
Active head Passive head
Cluster ports
are connected
el01stor01 el01stor02
Performing Failover Actions

All communication between storage heads is transmitted over one of the three cluster I/O
links provided by the Clustron hardware. This device offers two lower-speed serial links and
one Ethernet link. The use of serial links allows for greater reliability. Ethernet links may not
be serviced quickly enough by a system under extremely heavy load.
Normally, heartbeat messages are sent by each head on all three cluster I/O links at 50 ms
intervals. Failure to receive any message is considered link failure after 200 ms (serial links)
or 500 ms (Ethernet links). If all three links have failed, the peer is assumed to have failed.
Takeover arbitration will be performed.
The Configuration > Cluster view provides a graphical overview of the status of the cluster
card, the cluster head node states, and all of the resources. Thumbnail pictures are displayed
for each head and for each cluster card connection. A solid line connects a link when that link
is connected and active, and the line disappears if that connection is broken or while the other
system is restarting/rebooting.
The Setup and Unconfig buttons are used to initialize the cluster configuration or to revert a
cluster back to stand-alone operation. The Failback button makes the currently selected
storage head the new owner (active) within the cluster. The Takeover button is similar to
Failback but also forces the other storage head to reboot.

Creating Administrative Users
1
2
3
4
Creating Administrative Users

1. Click Configuration.
2. Click Users.
3. Add a new user.
4. Users can either be Directory or Local. Local users have all account and role information
saved on the appliance. Directory users exist in an external LDAP or NIS repository,
while their role information is still saved on the appliance. In addition to a username and
password, these other optional user fields are also available:
- Kiosk User: A kiosk user will not be able to access the appliance via the CLI.
- Kiosk Screen: A specific screen (such as Dashboard) that this kiosk user is
restricted to
5. Users are granted privileges by assigning custom roles to them. A role is a collection of
privileges that can be assigned to users. It may be desirable to create administrator and
operator roles with different authorization levels. Staff members may be assigned any
role that is suitable for their needs without assigning unnecessary privileges. An
Exceptions screen is also available. These authorizations are excluded from those
normally available due to the selected roles.

Roles and Authorizations
Select
2 authorization type
Add authorizations
to role
Roles and Authorizations

Authorizations allow users to perform specific tasks, such as creating shares, rebooting the
appliance, and updating the system software. Authorizations are grouped into Scopes, and
each scope may have a set of optional filters to narrow the scope of the authorization. For
example, rather than an authorization to restart all services, a filter can be used so that this
authorization can restart the HTTP service only.
1. From the same Configuration > Users screen, add a new Role.
2. Browse the scopes in the BUI to see which other authorizations exist. Available scopes
include Alerts, Analytics, Clustering, Hardware, Networking, Projects/Shares, Roles,
Services, and Users.

Shared Storage Concepts
Term Definition
Pool High availability and replication characteristics for a collection of disks
Project Default administrative and file system settings for a collection of shares
Share A file system mount point, access protocols, access rights, and other
settings
Pool
Project Project
Share Share Share Share
Shared Storage Concepts

The appliance is based on the ZFS file system. ZFS groups the underlying storage devices
into pools. Shared file systems then allocate disk space from these pools. Before creating file
systems, you must first configure storage on the appliance. After a storage pool is configured,
you do not have to statically size file systems, although this behavior can be achieved by
using quotas and reservations.
Each storage node can have any number of pools, and each pool can be assigned ownership
independently in a cluster. While an arbitrary number of pools is supported, creating multiple
pools with the same redundancy characteristics owned by the same cluster head is not
advised. Doing so results in poor performance, suboptimal allocation of resources, artificial
partitioning of storage, and additional administrative complexity.
The storage appliance exports file systems as shares, which are managed in this section of
the appliance. All file systems are grouped into projects. A project defines a common
administrative control point for managing shares. All shares within a project can share
common settings, and quotas can be enforced at the project level in addition to the share
level. Projects can also be used solely for grouping logically related shares, so their common
attributes (such as accumulated space) can be accessed from a single point.

Default Storage Configuration
The OneCommand tool:

• Defines a single pool that spans 18 of the 20 disks
• Defines common shares as well as shares dedicated for
each compute node
• Enables NFS access on all shares
exalogic
common
node1 ... node30

general patches
general dumps general dumps
Default Storage Configuration

The default storage configuration is performed at the time of manufacture. A single storage
pool is configured. Active-passive clustering for the server heads is configured. Data is
mirrored, which yields a highly reliable and high-performing system.
Two exclusive NFS shares are available for each of the Exalogic compute nodes: one for
crash dumps, and another for general purposes. You can implement access control for these
shares based on your requirements. Two common NFS shares are also available and are
intended to be accessed by all compute nodes. One of the shares is intended for patches and
another for general purposes.

Creating a Pool
Multiple pools are recommended only when you require

drastically different levels of redundancy or performance.
1
2
el01stor01
Data profile for the

default pool
Disk allocation for

the default pool
Creating a Pool
Each node can have any number of pools, and each pool can be assigned ownership
independently in a cluster. While an arbitrary number of pools is supported, creating multiple
pools with the same redundancy characteristics owned by the same cluster head is not
advised. Doing so will result in poor performance, suboptimal allocation of resources, artificial
partitioning of storage, and additional administrative complexity.
When allocating raw storage to pools, keep in mind that filling pools completely results in
significantly reduced performance, especially when writing to shares. These effects typically
become noticeable after the pool exceeds 80% full, and can be significant when the pool
exceeds 90% full. Therefore, best results are obtained by over-provisioning by approximately
20%.
After the task of creating a new pool is started, the setup falls into two different phases:
verification and configuration. The verification phase enables you to verify that all storage is
attached and functioning, and allocate disks within chassis. In a stand-alone system, this
presents a list of all available storage and drive types, with the ability to change the number of
disks to allocate to the new pool. By default, the maximum number of disks is allocated, but
this number can be reduced in anticipation of creating multiple pools.

Pool Data Profiles
Profile Description
Striped No redundancy
Mirrored Data is always copied to another disk.

Reduces usable space by 1/2
Triple-Mirrored Data is always copied to 2 additional disks.

Single-Parity RAID For every 3 disks, there is a single parity disk for recovery.
Double-Parity RAID For every 3 disks, there are 2 parity disks for recovery.
Triple-Parity RAID For every n disks, there are 3 parity disks for recovery.
Pool Data Profiles

After verification is completed, the next step involves choosing a storage profile that reflects
the RAS and performance goals of your setup. The set of possible profiles presented depends
on your available storage.
• Striped: Data is striped across disks, with no redundancy whatsoever. Although this
maximizes both performance and capacity, it comes at great cost: A single disk failure
results in data loss.
• Mirrored: Data is mirrored, reducing capacity by half but yielding a highly reliable and
high-performing system. This is recommended when space is considered ample but
performance is at a premium.
• Triple-Mirrored: Data is triply mirrored, reducing capacity by one-third, but yielding a
very highly reliable and high-performing system. This configuration is intended for
situations in which maximum performance, and availability are required while capacity is
much less important. Compared with a two-way mirror, a three-way mirror adds
additional protection against disk failures and latent disk failures, in particular during
reconstruction for a previous failure.
• Single-Parity RAID: Each stripe is kept to three data disks and a single parity disk.
Narrow stripe widths offer better random read performance than the wider stripe,
double-parity configuration, but they do not have quite the capacity cost of the mirrored
option.
Creating a Project
1 5
2
6
6
Base path for share
3 mount points
Default settings for

this project's shares
Creating a Project
1. Click Shares in the top menu.
2. Click Projects on the submenu.
3. A list of all projects is displayed. Click the plus (+) button to create a new project.
4. Supply the project with a name. By default, this name is used as the base mount point
for this project.
5. Click the General tab for this new project.
6. Configure the default settings for any shares that will be members of this project
including:
- Quota: Sets a maximum limit on the total amount of space consumed by all file
systems and within the project.
- Read-only: The contents of a read-only file system cannot be modified, regardless
of any protocol settings. This setting does not affect the ability to rename, destroy,
or change properties of the file system.

Creating a Project
Default owner and

UNIX permissions
Service is
enabled.
Default settings for

each protocol
Creating a Project (continued)

• User: The owner of the root directory. This can be specified as a user ID or username.
For UNIX-based NFS access, this can be changed from the client by using the chown
command.
• Group: The group of the root directory. This can be specified as a group ID or group
name. For UNIX-based NFS access, this can be changed from the client by using the
chgrp command.
• Permissions: Standard UNIX permissions (owner/group/everyone, read/write/execute)
for the root directory. For UNIX-based NFS access, this can be changed from the client
by using the chmod command. Selecting permissions is done by clicking individual
boxes. Alternatively, clicking the label (“user,” “group,” or “other”) selects (or deselects)
all permissions within the label.
• Share Mode: Determines whether shares in this project are available for reading only,
for reading and writing, or neither.
• Anonymous User Mapping: Unless the “root” option is in effect for a particular client,
the root user on that client is treated as an unknown user, and all attempts by that user
to access the share's files will be treated as attempts by a user with this UID.

Creating a Share
1 2 4
Default mount point:

/export/CService1/<name>
Creating a Share
Shares that are part of a project can either have local settings for properties, or they can
inherit their settings from the parent project. By default, shares inherit all properties from the
project. If a property is changed on a project, all shares that inherit that property are updated
to reflect the new value. When inherited, all properties have the same value as the parent
project, with the exception of the mount point. When inherited, the mount point is the
concatenation of the project setting and the share name.
1. Access a project.
2. Click the Shares menu option.
3. Click the plus (+) button to define a new shared file system.
4. Give the share a name. By default, this name is used as part of the mount point. Edit
other share settings such as:
- Data Migration Source: If set, this file system is actively shadowing an existing
file system, either locally or over NFS.
- Case sensitivity: It controls whether directory lookups are case-sensitive or not
case-sensitive.

Overriding Project Defaults
1 2
Clear to override
project values.
Overriding Project Defaults

Every file system on the appliance must be given a unique mount point which serves as the
access point for the file system data. Projects can be given mount points, but these serve only
as a tool to manage the namespace by using inherited properties. Projects are never mounted
and do not export data over any protocol.
The project/share combination is a unique identifier for a share within a pool. Multiple projects
can contain shares with the same name, but a single project cannot contain shares with the
same name. A single project can contain both file systems, and they share the same
namespace.
Shares that are part of a project can either have local settings for properties, or they can
inherit their settings from the parent project. By default, shares inherit all properties from the
project. If a property is changed on a project, all shares that inherit that property are updated
to reflect the new value. When inherited, all properties have the same value as the parent
project, with the exception of the mount-point and SMB properties. When inherited, these
properties concatenate the project setting with their own share name.
To override the default properties defined in the parent project, deselect the “Inherit from
project” check box.

Setting Quotas
You can restrict the maximum size of:

• All shares within a project
• A specific share
Setting Quotas
The appliance leverages a pooled storage model where all file systems share common space.
By default, file systems never have an explicit size assigned to them, and only take up as
much space as they need. This approach provides maximum flexibility and simplicity of
management in an environment when users are generally trusted to do the right thing. A
stricter environment, where user's data usage is monitored and/or restricted, requires more
careful management. A quota represents a limit on the amount of space that can be
consumed by any particular entity. This can be based on file system, project, user, or group,
and is independent of any current space usage. Clients attempting to write new data will get
an error message when the file system is full, either because of a quota or because the
storage pool is out of space.
A reservation, on the other hand, represents a guarantee of space for a particular project or
file system. This takes available space away from the rest of the pool without increasing the
actual space consumed by the file system. This setting cannot be applied to users and
groups. The traditional notion of a statically sized file system can be created by setting a
quota and reservation to the same value. By default, a reservation includes all snapshots of a
file system. If, instead, the Include Snapshots property is not set, then the reservation only
applies to the immediate data of the file system.

Restricting Share Access
You can limit access to a share by configuring:

• Standard UNIX file permissions
• NFS exceptions based on client IP address or DNS
Starting IP and
CIDR bit mask
Restricting Share Access

Exceptions to the overall sharing modes may be defined for clients or collections of clients.
When a client attempts access, its access will be granted according to the first exception in
the list that matches the client (or, if no such exception exists, according to the global share
modes defined previously). These client collections may be defined by using one of three
types:
• Host(FQDN) or Netgroup: A single client whose IP address resolves to the specified
fully qualified name, or a netgroup that contains fully qualified names to which a client's
IP address resolves. An example is exl01cn01.mydomain.com.
• DNS Domain: All clients whose IP addresses resolve to a fully qualified name ending in
this suffix. An example is mydomain.com.
• Network: All clients whose IP addresses are within the specified IP subnet, expressed
in CIDR notation. An example is 192.168.20.0/22.
For each specified client or collection of clients, you will then express two parameters:
whether the client will be permitted read-only or read-write access to the share, and whether
the root user on the client will be treated as the root user (if selected) or the unknown user.

Configuration Management
Use this menu to:

• Update or roll back the appliance software
• Back up the current appliance configuration
• Restore the initial configuration or a previously saved one
• Generate a diagnostic bundle and upload it to Oracle
support
1
2
Configuration Management
Each update may be supplied with new firmware or updates to external resources. In general,
these updates are backward-compatible and applied automatically without user intervention.
There are exceptions, however, for nonreversible updates. These updates involve updating a
resource external to the system software in a way that is incompatible with older software
releases. After the update is applied, rolling back to previous versions will result in undefined
behavior. For these updates, you will always be given an explicit option to apply them
automatically during upgrade or apply them after the fact.
Following the application of a software upgrade, any hardware for which the upgrade includes
newer versions of firmware will be upgraded. There are several types of devices for which
firmware upgrades may be made available; each has distinct characteristics.
The appliance can generate support bundles containing system configuration information and
core files for use by remote support in debugging system failures. Support bundles are
generated automatically in response to faults if the Phone Home service is enabled.
Administrators can also manually generate and upload a support bundle. To facilitate this, the
appliance must be connected to the Internet, either directly or through the web proxy
configured on the Phone Home service screen.

Mounting an NFS Share
/etc/fstab (Linux):
1
el01stor01:/export/CService1/apps /u01/apps nfs

rw,bg,hard,nointr,rsize=131072,wsize=131072,tcp,vers=3
/etc/vfstab (Solaris):
el01stor01:/export/CService1/apps /u01/apps nfs -yes

rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,
vers=3
Recommended settings
2
> mount /u01/apps
Mounting an NFS Share

UNIX NFS mount options include:
• bg: If the first NFS mount attempt times out, retry the mount in the background. After a
mount operation is moved to the background, all subsequent mounts on the same NFS
server will be run in the background immediately, without first attempting the mount. A
missing mount point is treated as a timeout, to allow for nested NFS mounts.
• hard/soft: Specifies whether the program that is using a file via an NFS connection
should stop and wait (hard) for the server to come back online (if the host serving the
exported file system is unavailable) or whether it should report an error (soft). Using soft
mounts is not recommended, as they can generate I/O errors in very congested
networks or when using a very busy server.
• intr: Allows NFS requests to be interrupted if the server fails or cannot be reached
• rsize/wsize: These settings speed up NFS communication for reads (rsize) and writes
(wsize) by setting a larger data block size, in bytes, to be transferred at one time. For
NFSv2 or NFSv3, the default values for both parameters is set to 8192. For NFSv4, the
default values for both parameters is set to 32768.
• tcp/udp: Specifies that the NFS mount uses the TCP or UDP protocol

Storage Command-Line Interface (CLI)
• Use SSH to connect to the appliance.

• Tab completion is supported.
• Supply a batch of commands by redirecting input to a file.
• Configuration entities and commands are organized into
groups or “contexts,” similar to the BUI.
configuration maintenance shares status analytics
storage system Project1 storage
services hardware Project2 services
users activity
ShareA
roles alerts
Storage Command-Line Interface (CLI)

The CLI is designed to mirror the capabilities of the BUI, while also providing a powerful
scripting environment for performing repetitive tasks. The appliance presents a command-line
interface available through either the serial console or SSH. To administer the appliance, you
must log in as root. When you log in, the CLI presents you with a prompt that consists of the
host name, followed by a colon, followed by a greater-than sign.
Tab completion is used extensively. If you are not sure what to enter in a particular context,
pressing the Tab key provides you with possible options. Throughout the documentation,
pressing Tab is presented as the word “tab” in bold italic. Help is always available: The help
command provides context-specific help. Help on a particular topic is available by specifying
the topic as an argument to help. An example is help commands. Available topics are
displayed by tab-completing the help command, or by typing help topics.
A central principle in the CLI is the context in which commands are executed. The context
dictates which elements of the system can be managed, and which commands are available.
Contexts have a tree structure in which contexts may themselves contain nested contexts and
the structure generally mirrors that of the views in the BUI. Note that the prompt changes to
reflect the context, with the context provided between the colon (:) and the greater-than sign
(>) in the prompt.

Some CLI Commands
Command Description
help Print a list of commands that can be used in current context.

<context> Navigate to the given context.
select <entity> Navigate to the given entity (project, file system, and so on).
destroy <entity> Delete the given entity.
<entity> <name> Create a new entity (for example, project Project1).
show Print a list of properties, configuration entities, and child
contexts in the current context.
get Print all properties of the current entity, or a specific property.
set <prop>=<value> Update the given property of the current entity.
commit/revert Save or undo your modifications.
done Return to the parent context.
Some CLI Commands

Some child contexts are dynamic in that they correspond not to fixed views in the browser, but
rather to dynamic entities that have been created by either the user or the system. To
navigate to these contexts, use the select command followed by the name of the dynamic
context. The names of the dynamic contexts contained within a given context are shown by
using the list command.
Once in a context, execute context-specific commands. For example, to obtain the current
user's preferences, execute the get command from the configuration preferences context. The
set command will set a property to a specified value, with the property name and its value
separated by an equal sign (=).
The text “uncommitted” in the prompt denotes that this an uncommitted context. An
uncommitted entity is committed via the commit command. Any attempt to navigate away from
the uncommitted context will prompt for confirmation. When committing an uncommitted
entity, the properties associated with the new entity will be validated, and an error will be
generated if the entity cannot be created. For example, the creation of a new threshold alert
requires the specification of a statistic name. Failure to set this results in an error.

CLI Examples
Create a new project and share:
shares
project CService1
set mountpoint=/export/CService1
set compression=gzip
commit
select CService1
filesystem apps
set readonly=true
commit
Monitor appliance health:
status
storage
show
hardware
show
CLI Examples
To return to the previous context, use the done command. Similarly, to navigate to a parent
context, use the cd command. Inspired by the classic UNIX command, cd takes an argument
of .. to denote moving to the parent context.

FMW Storage Topology
Oracle recommends that you:

• Configure storage projects:
– For each separate FMW installation
– To isolate data for different business units or domains
• Create project shares for:
– Product installations and binaries
– WebLogic domains
– WebLogic Node Manager configurations
– WebLogic persistent stores
FMW Storage Topology

If your Exalogic machine will host applications that require different versions of FMW
products, Oracle recommends that you create a separate project on the storage appliance for
each version and then make them available to all or a subset of the compute nodes. For most
FMW products, including WLS, compute nodes must be granted only read access to these
projects. Each product installation will be a separately configured share within the project.
Oracle recommends that, whenever possible, you use the shared storage appliance to store
products, configuration files, logs, and applications, and that you not use the compute node's
local flash storage device. A node's local device is intended primarily as a boot device.
The WLS persistent store service provides a built-in, high-performance storage solution for
WLS subsystems that require persistence for recovery and migration purposes. For example,
it can store persistent JMS messages or transaction logs. The persistent store uses a file-
based store by default, but database stores are also supported. Both types can survive a
process crash or hardware power failure without losing any committed changes. To support
automatic failover, these stores must be available to all candidate servers (usually all
members of a cluster) in a domain.

FMW Storage: Example
exalogic
FMW11gR1 HR
domains
Middleware
/hrdomain
/wlserver_1034
/jrockit_160
nodemanager
recovery
/jms
/tlog
apps

Quiz
Which of the following is not a task that can be performed with

the appliance BUI?
a. Illuminate an LED.
b. Monitor disk performance.
c. Back up the configuration.
d. Order replacement cables.
e. Assign an authorization to a role.
Answer: d

Quiz
Which of the following represents the entity hierarchy used by

the appliance?
a. Pool, Project, Share
b. Port, Pool, Share
c. Share, Port, Pool
d. Share, Pool, Project
e. Pool, Share, Project
Answer: a

Quiz
Name three services that clients can use to access a shared

file system.
a. FTP
b. NFS
c. SOAP
d. SCSI
e. WebDAV
Answer: a, b, e

Summary

• List the capabilities of the storage appliance
• Perform basic administration and maintenance tasks
• Configure appliance access services
• Describe Exalogic's default storage setup
• Create pools, projects, and shares
• Mount a shared file system by using NFS
• Administer the appliance from the command line
• Describe the recommended configuration for FMW

Practice 6-1
Initialize Shared File Systems
• Accessing a storage appliance from a browser and
command line
• Creating storage projects and shares
• Configuring NFS access for a share
• Mounting shares on a local file system

Network Configuration

Objectives

• Describe the network topology for various rack
configurations
• Identify switch LEDs and replace failed hardware
• Access an InfiniBand switch from the browser or command
line
• Describe the role of the InfiniBand subnet manager
• Configure a compute node's network to support FMW

Exalogic Networks: Review
Network Speed Description

• Provides access to compute nodes via
Client 10 Gb
intranet/Internet
• Is used to monitor and administer components
Management 1 Gb • Provides access to ILOM and other device
management interfaces
• Is a private, nonroutable network
• Connects racks, along with all rack components
Private
40 Gb • Is used by compute nodes to access shared
InfiniBand
storage
• Is used for internal cluster communication
Exalogic Networks: Review

An Exalogic machine includes compute nodes, a Sun ZFS Storage 7320 appliance, as well as
equipment to connect the compute nodes to your network. The network connections allow the
servers to be administered remotely, enable clients to connect to the compute nodes, and
enable client access to the storage appliance. Additional configuration, such as defining
multiple virtual local area networks (VLANs) or enabling routing, may be required for the
switches to operate properly in your environment and is beyond the scope of the installation
service.
There are up to five networks for an Exalogic machine. Each network must be on a distinct
and separate subnet from the others. The Exalogic management network connects to your
existing management network and is used for administrative work for all components of the
Exalogic machine. It connects ILOM, compute nodes, server heads in the storage appliance,
and switches connected to the Ethernet switch in the Exalogic machine rack. This
management network is in a single subnet. Do not use the management network interface
(ETH0/NET0) on compute nodes for client or application network traffic. Cabling or
configuration changes to these interfaces on Exalogic compute nodes is not permitted.

IB Networking Concepts: Review
Term Definition Description

IPoIB IP over InfiniBand Applications connected by an IB fabric communicate
using standard IP address semantics.
SDP Socket Direct Protocol Applications communicate directly with the IB fabric,
bypassing the operating system's TCP/IP stack.
EoIB Ethernet over Applications within an IB fabric communicate with

InfiniBand external Ethernet networks.
vNIC Virtual Network Software that emulates an Ethernet NIC on the IB
Interface Card network
IB Networking Concepts: Review

IP over InfiniBand (IPoIB) enables applications on separate devices to communicate with
each other over a private InfiniBand fabric by using native IB protocols and without the
overhead of Ethernet. For example, a compute node on one Exalogic rack may communicate
with a database on an Exadata rack. However, applications must support the SDP protocol to
utilize IPoIB instead of the default TCP/IP stack of the host operating system.
The InfiniBand switches also act as gateways to connect to external Ethernet networks. They
support eight 10-GB Ethernet ports. Exalogic compute nodes can communicate through these
ports by using Ethernet over InfiniBand (EoIB). Each port is represented on the compute
nodes as a vNIC. This representation enables that node's IB connection to appear like any
other Ethernet NIC to both the operating system and the external Ethernet network.
Up to eight external physical Ethernet networks can be connected to the IB switches. But you
can also define multiple VLANs for each of these Ethernet ports. By default, no VLANs are
configured.

Default Compute Node Network
Private IB
Network
IPoIB, Compute Node

SDP
IB IB0 bond0
Switches IB1 bond1 eth0
EoIB
ILOM Ethernet
Client
Network
Mgmt Mgmt
Network Switch
Default Compute Node Network

By default, each Exalogic compute node is configured with one bonded EoIB interface for one
external LAN (client network), and is named bond1. The Cisco Ethernet management switch
is connected to the NET0 port of compute nodes, the NET0 port of the storage appliance, and
also the management ports of the InfiniBand gateway switches. On the compute nodes, this
connection is represented on the operating system by an “eth” network interface, such as
eth0. The compute nodes are configured at the time of manufacturing to use sideband
management only. Therefore, the MGMT (or ILOM) port is not connected, but ILOM is
accessible from NET0.

Default Storage Appliance Network
Private IB
Network
NFS
Storage Heads
IPoIB IB0
IB
Switches IB1
BUI, ILOM Ethernet
Mgmt Mgmt
Network Switch
Default Storage Appliance Network

The Cisco Ethernet management switch is connected to the NET0 port of the storage
appliance. Each storage node’s ILOM interface is accessible from NET0, along with the
storage appliance browser user interface (BUI). Note that the storage appliance is not
configured as part of the Exalogic client network, and is only intended to be accessed by other
nodes within the same data center.

Quarter-Rack Network Topology
10 Gb Network
1-8 1-8
Redundant links 16 unused
10 Gb IB ports
Redundant
switches
6
NM2-GW #1 NM2-GW #2
40 Gb
2 Storage 2
Heads 1–2
IB0 (active) IB1 (passive)
8 Compute 8
Nodes 1–8
Not shown: mgmt network
Quarter-Rack Network Topology

NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). Half of
the compute nodes are connected with their active ports to the first gateway switch and their
passive ports to the second gateway switch. The remaining half of the compute nodes are
connected with their active ports to the second gateway switch and their passive ports to the
first gateway switch. This connection is for high availability and load distribution purposes.
Exalogic machine quarter racks do not contain the Sun Datacenter InfiniBand Switch 36.
Applications running on compute nodes communicate with applications on other compute
nodes using this InfiniBand network. Exalogic machines can be connected to an external
network, including a standard database hosted on a machine outside of the Exalogic
machine, via the InfiniBand-to-10 Gb Ethernet gateways using Ethernet over InfiniBand
(EoIB). Each Exalogic machine configuration includes at least two such gateways, which also
act as InfiniBand switches connecting all compute nodes and the Sun ZFS Storage 7320
appliance within the Exalogic machine.

Half-Rack Network Topology
10 Gb Network
1-8 1-8
8 unused
IB ports
6
NM2-GW #1 NM2-GW #2
2 Storage 2
Heads 1–2
IB0 IB1
8 Compute 8
Nodes 1–8
IB1 IB0
8 Compute 8
Nodes 9–16
Half-Rack Network Topology

The Sun Datacenter InfiniBand Switch 36, referred to as a spine switch, is present in a half
rack but not connected in a single rack configuration.

Full-Rack Network Topology
10 Gb Network
1-8 1-8 1-8 1-8
4 6 4
NM2-GW #3 NM2-GW #1 NM2-GW #2 NM2-GW #4
2 Storage 2
Heads 1-2
All four switches No unused
are interconnected. 8 Compute 8 IB ports
Nodes 1–8
8 Compute 8
Nodes 9–16
7 Compute 7
Nodes 17–23
7 Compute 7
Nodes 24–30
Full-Rack Network Topology

The spine switch is also present in a full rack but not connected in a single rack configuration.

Multi-Rack Topology
Refer to the Exalogic Multirack Cabling Guide for details.
Six unused
NM2-36 Full Rack IB ports
4
Four unused
IB ports
NM2-GW NM2-GW
NM2-GW NM2-GW 6 6
6 6 NM2-GW NM2-GW
NM2-GW NM2-GW
4
NM2-36
Full Rack
Multi-Rack Topology
Sun Network QDR InfiniBand Gateway Switch (NM2-GW) 4x4 (four switches in each Exalogic
machine full rack) cross-connects are re-cabled to their respective local spine switches (Sun
Datacenter InfiniBand Switch 36 (NM2-36). In addition, each leaf switch is connected to the
other rack’s spine switch. For example, leaf switches in rack 1 are connected to the spine
switch in rack 2.
Before you can set up the multi-rack cabling, you must first shut down the affected Exalogic
machines and Oracle Exadata Database Machines. Consider the operational impact of
shutting down the systems, and take appropriate action. Also be careful when you handle
InfiniBand cables. InfiniBand cables are very fragile. The cable ends might break off if the
ends are dropped or mishandled in any way.
A spare cable bundle is provided with Exalogic machines. Extract the spare InfiniBand cables
from the bundle. You need to use the InfiniBand cables only. Restore spare Ethernet cables
to the original spares location. You do not use the Ethernet cables.
Re-label all inter-rack cables from leaf switches to spine switches. Re-label each cable to
identify the correct locations at both cable ends, so that cables can be connected correctly
during future maintenance.
Refer to the Exalogic Machine Multirack Cabling Guide for more detailed instructions.

Transceivers and Cables
• IB switches utilize transceivers to support different types of

cables and connectors.
• Gateway switches also use special splitter cables:
– One IB QSFP connector
– Four LC SFP+ connectors (fiber-optic)
– Multiple 10 Gb connections required to match IB bandwidth
• Refer to the Exalogic Machine Guide for a list of available
transceivers and cables of various lengths.
Transceivers and Cables

Generally speaking, IB switches support both pass-through and splitter cables, and both
optical and copper. Copper IB cables are typically limited to 5 meters in length, while optical
cables can reach 30 meters or more.
Cable paths should be as short as possible. When the length of a cable path has been
calculated, select the shortest cable to satisfy the length requirement. Also keep in mind that
bending cables and bundling cables together both increase the required lengths of cables.
The gateway switches support the following types of transceivers:
• 10 Gb SFP+SR (LC MMF connector)
• QSFP (MTP MMF connector)
The gateway also supports QSFP optical splitter cables in lengths of 10, 20, and 30 meters.
The QSFP transceivers are installed by default in the gateway switches of your Exalogic
machine. The splitter cables and SFP+ transceivers for 10 Gb data center connectivity should
be ordered separately.

Administration Tasks
Core administrative tasks for the Exalogic network include the

following:
• Determine switch status and health.
• Identify and replace failed switch hardware.
• Verify the desired IB network topology.
• Identify and troubleshoot connectivity issues.
• Configure the IB subnet manager.
• Configure virtual IP addresses on compute nodes.

Status LEDs
Front
DC power OK
Power supply fault
AC power OK
Fan fault
Link status
Locator
Switch fault
Switch OK
Rear
Status LEDs
Status LEDs are used on many components of the switch chassis as a means of indicating
the component’s state. The power supply status LEDs and fan status LEDs are located on the
front of the switch chassis:
• The power OK LED is green. When it is on, a 12V DC voltage is present.
• The power attention/fault LED is amber. When it is on, a fault has been detected on the
power supply and power has been shut down.
• The power AC OK LED is green. When it is on, AC power is present.
• The fan attention/fault LED is amber. When it is on, there is a problem with the fan.
The chassis status LEDs, network management status LEDs, and link status LEDs are
located on the rear of the switch chassis:
• The switch locator LED is white. When it is flashing, the switch is indentifying itself. The
locator LED can be enabled remotely using the browser or command-line management
interfaces.
• The switch attention/fault LED is amber. When it is on, a networking fault has been
detected on the switch.
• The switch OK LED is green. When it is off, the switch is off or is initializing. When it is
on, the gateway is functional and no faults are present.

• See the Sun Network QDR InfiniBand Gateway Switch:

Service Manual for detailed procedures about power
supplies, fans, and so on.
• For example, to replace a fan on the switch:
1. Loosen the screw on the top-right side.
2. Use the handle to remove the old fan.
3. Insert the new fan and verify that the fault LED turns off.

To replace a switch power supply:
1. Locate the power supply to be removed, using alert or locator LEDs.
2. Press and hold the release tab to the left and pull the handle of the power supply.
Continue to pull the handle of the power supply to remove it from the chassis.
3. Verify that the slot where the power supply installs is clean and free of debris.
4. Orient the power supply with the status LEDs on the left and the release tab on the right.
Slide the power supply into the open slot, pushing on the handle.
5. When the power supply seats, push firmly so that the release tab clicks to secure the
power supply into the chassis.
6. Power on the power supply.

IB Switch Command-Line Interface (CLI)
• Use SSH to connect to a switch via the management

network.
• Log in as root/welcome1 and run the password command
to modify the default.
• Use the help command to see a list of available
commands.
• The interface also supports tab completion.
View all commands:
# help all
View hardware diagnostic commands:
# help diag
IB Switch Command-Line Interface (CLI)

The management controller uses a simplified Linux OS and file system. After you log in to the
root account, the shell prompt (#) appears, and you can enter shell commands.
The hardware commands are a means of monitoring the switch hardware and working with
gateway ports. Only the root user of the management controller can run all of the hardware
commands
The InfiniBand commands are a means of monitoring and controlling many aspects of the
InfiniBand fabric. These commands are also run from the management controller, which is
also the host of a Subnet Manager internal to the gateway. Only the root user of the
management controller can run the InfiniBand commands.
With the help command, you can either supply the specific name of a command or a class of
commands. The available classes include general, diag, ibdiag, sm, and other.

Monitoring the Hardware
# version
SUN DCS gw version: 1.1.2-2
Build time: Sep 24 2010 10:32:29
FPGA version: 0x33
SP board info:
Manufacturing Date: 2010.06.03
Serial Number: "NCD5A0025"
...
# showunhealthy
OK - No unhealthy sensors
# env_test
...
Voltage test returned OK
...
Temperature test returned OK
...
FAN test returned OK
...
Monitoring the Hardware

In time, newer or updated gateway firmware might become available. The version hardware
command shows the hardware and software versions, as well as date information for the
gateway and management controller.
The showunhealthy hardware command shows a list of switch components that appear to
have a problem. Unlike the env_test command, this command displays messages for only
those components that have failed testing.
The env_test hardware command performs a series of hardware and environmental tests of
the gateway. Its output provides voltage and temperature values, pass-fail results, and error
messages. This command is essentially a combination of the following other commands:
• checkpower
• checkvoltages
• showtemps
• getfanspeed
• connector
• checkboot

Monitoring Connection Status
# listlinkup Cable present

Connector 0A Not present No cable but no link
...
Connector 5A Present <-> Switch Port 30 down (Enabled)
Connector 6A Present <-> Switch Port 35 down (Enabled)
Connector 7A Present <-> Switch Port 32 up (Enabled)
...
Cable present
Connector 0A-ETH Present
and link is up
Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled)
Hosts found on Gateway

this IB network splitter cable
# ibhosts
Ca: ... ports 2 "el01cn01 EL-C 192.168.10.8 HCA-1"
Ca: ... ports 2 "el01cn02 EL-C 192.168.10.7 HCA-1"
...
Monitoring Connection Status

The listlinkup hardware command lists the presence of links and the up-down state of the
associated ports on the switch chip.
The ibhosts IB command is a script that discovers the InfiniBand fabric topology or uses the
existing topology file to extract the channel adapter nodes.
A related IB command is ibportstate, which queries the logical and physical state of a
specific InfiniBand port. The command can return the link width and speed of a switch port, as
well as enabling, disabling, or resetting the port. The command can also set the link speed of
any InfiniBand port.

Running IB Diagnostics
# ibdiagnet
Discovering ... 13 nodes (1 Switches & 12 CA-s) discovered.
...
No bad Links (with logical state = INIT) were found
...
Subnet: IPv4 PKey:0x0001 QKey:0x00000b1b MTU:2048Byte ...
No members found for group
...
# ibnetdiscover
Topology file: generated on Sat Feb 5 02:31:53 2011
...
Ca2 ... "el01cn0101 EL-C 192.168.10.8 HCA-1"
[1](21280001a0a4b9) "S-002128547ca2c0a0"[18]
# lid 15 lmc 0 "SUN IB QDR GW switch 100.140.50.5" lid 6 4xQDR
...
# ibcheckerrors
...
Summary: 13 nodes checked, 0 bad nodes found
28 ports checked, 0 ports have errors beyond threshold
Running IB Diagnostics
The ibdiagnet command performs a collection of tests on the InfiniBand fabric by using
directed route packets and extracts all the available information regarding connectivity. The
command also checks for duplicate node/port GUIDs in the InfiniBand fabric along with
suspected bad links, and displays any issues to standard output. It also generates several
output files, including:
• ibdiagnet.lst: List of all the nodes, ports, and links in the fabric
• ibdiagnet.fdbs: Dump of the unicast forwarding tables of the fabric switches
• ibdiagnet.sm: List of all the Subnet Manager (state and priority) in the fabric
The ibnetdiscover command discovers the InfiniBand network topology and generates a
human-readable topology file. It displays all nodes, node types, node descriptions, links, port
numbers, port LIDs, and GUIDs.
The ibcheckerrors command uses the topology file that was previously generated by
ibnetdiscover to scan the InfiniBand fabric, validate the connectivity as described in the
topology file, and report errors as indicated by the port counters.

Subnet Manager (SM) Concepts
• The IB subnet manager process has these responsibilities:

– Discovering and monitoring the network topology
– Assigning IDs to each endpoint on the network
– Calculating and programming route and partition tables
• When multiple switches run the SM, the following occurs:
– One switch is designated as the master and the others as
standby for failover.
– Each switch can be assigned a numeric priority (0–15).
Switch A Switch B
SM Master SM
Switch A
SM
Subnet Manager (SM) Concepts

The Exalogic subnet manager (SM) manages all operational characteristics of the InfiniBand
network, such as discovering the network topology and calculating forwarding tables. The
InfiniBand network typically has more than one SM, but only one SM is active at a time. The
active SM is the master SM; others are standby SMs. If the master SM shuts down or fails, a
standby SM automatically becomes the master SM.
If you are connecting a single Exalogic machine to a single Exadata Database Machine,
ensure that an SM runs on all of the InfiniBand switches (leaf switches and spine switches).
You must configure the master SM to run on a spine switch only. If you are connecting
multiple Exalogic machines to one or more Exadata Database Machines, only spine switches
and Top of Rack (ToR) switches should run SM. You must configure the master SM to run on
a ToR switch.

Subnet Manager Recommendations
Topology Spine Switches Gateway Switches
Stand-alone Rack (default) Not used • Enable SM

• Set to priority 5
Two Full or Half Racks • Enable master SM Disable SM
One Full/Half + One Quarter Enable SM Enable master SM
Three or More Racks • Enable master SM Disable SM

Subnet Manager Recommendations

For a single Exalogic machine, gateway (leaf) switches should run the subnet manager (SM).
You must set the Subnet Manager priority for each of the gateway switches to 5.
For two Exalogic machines, the master SM should run on a spine switch for half-rack/full-rack
configurations and on a leaf switch for configurations involving quarter racks. For
configurations not including a quarter rack, you must set the SM priority on the spine switch to
8. In addition, you must disable the SM on the gateway switches in multi-rack configurations.
For three or more Exalogic machines, the SM should run on the spine switch only. You should
disable SM on other switches. You must set the SM priority on the spine switch to 8.

Configuring the Subnet Manager
(Re)start the subnet manager:
# disablesm
# enablesm
View the current master subnet manager:
# getmaster
Local SM enabled and running ...
Master SubnetManager on sm lid 6 sm guid ...:
SUN IB QDR GW switch 100.150.50.5
View the current priority:
# setsmpriority list
smpriority 5 ...
Update the priority:
# setsmpriority 8
Configuring the Subnet Manager

If you want to quickly determine the local Subnet Manager's priority and state, you can use
the sminfo command. A state value of 3 indicates a master SM.
The getmaster command displays the host name or IP address of the switch in the network
where the master SM is running.
You are required to relocate the master SM from a leaf switch to the spine switch when you
are connecting more than one Exalogic machine. Relocating the master SM does not affect
the availability of the InfiniBand network. You can perform this task while normal workload is
running. When you use the disablesm command on a switch that is currently hosting the
master SM, the master SM is automatically relocated to another switch in the network.

Fabric Monitor Interface
1 Access the switch

from a browser.
Main menu
3
Hardware and firmware

specifications
Similar functionality to the CLI
Fabric Monitor Interface

By using the Fabric Monitor, you can visually monitor the status of the gateway, the switch
hardware, and the connectors through a web-based interface. The Fabric Monitor is
accessible from the ILOM web interface of the switch.
1. Access the ILOM browser interface on the management network. Then log in as the root
account of the switch (root/welcome1, by default).
2. Click the Switch/Fabric Monitoring Tools tab and then click Launch Fabric Monitor.
3. The top portion of the interface displays an interactive graphic representing the rear
LEDs and ports on the switch. The bottom portion of the interface gives detailed
hardware and IB information about the switch. This information is organized into a series
of tabs.
Overall, the browser interface provides equivalent functionality to that of the switch command-
line interface, although it is generally read-only. Hidden on the right side of the Fabric Monitor
is the control panel. Clicking the More link opens the control panel and legend panels. The
control panel has five check boxes with which you can select the types of data that you want
the interface to poll. By default, all data is collected.

Fabric Monitor Examples
Monitor the
hardware.
View the subnet

manager settings.
Fabric Monitor Examples

The System Info tab displays status information regarding the switch hardware, including
basic information about the management controller, firmware version and build date, FRU IDs
for the chassis and power supplies, and status of the power supplies and fans.
The Sensor Info tab displays the latest hardware sensor readings for the switch’s power
supplies and fans, including the current voltage and temperature values.
The IB Performance tab displays the status and available bandwidth of the switch ports by
using a table format. By clicking on a column heading, the information in the table is sorted
according to that column heading, either in ascending or descending order.
The IB Port Map tab displays information about peer devices attached to the switch by using a
table format.
The Subnet Manager tab displays the current SM settings for this switch, including whether or
not it is the master SM, along with the priority of the SM.

Monitoring IB Connections
1
Active link LED status
Not connected with errors
2
Monitoring IB Connections
The rear panel diagram displays the presence of connectors and their status by using a
graphic. The diagram displays the management controller’s IP address, the connector
receptacles, and their respective connector names. When a cable is attached to a receptacle,
a connection is made. That connection is displayed in the diagram as a gray rectangle, with
three or four smaller indicators. Moving the mouse cursor over an indicator, clicking an
indicator, or clicking a connection opens a window that provides additional information about
that indicator or connection.
In the rear panel diagram, there are 32 InfiniBand receptacles displayed, labeled 0A to 15A
and 0B to 15B. When a connector is physically present in an InfiniBand receptacle, the
receptacle changes from a black rectangle to a gray rectangle with three indicators. Moving
the mouse cursor over an indicator that is orange or red opens a small window that provides
the reason for the respective state. A center indicator is orange when the link is at a speed
slower than QDR, such as SDR or DDR. A right indicator is red when there are significant
errors (symbol, recovery, and so on) on the link.
Clicking a gray InfiniBand connector opens a window that displays connector FRU, port state,
error, and statistical information for that connection.

Monitoring Gateway Connections
Monitoring Gateway Connections

The rear panel diagram displays four gateway receptacles, labeled 0A, 1A, 0B, and 1B. When
a connector is physically present in a gateway receptacle, the receptacle changes from a
black rectangle to a gray rectangle with four indicators. Each indicator represents one of the
four possible ports available at the connection. The rectangles left of the gateway connection
are the BX indicators, which display the status of the internal switch hardware connections.
Moving the mouse cursor over a BX indicator opens a small window that provides information
about the BridgeX port. If the indicator is red, then the window displays a reason for the
respective state.
Clicking a gray gateway connector opens a window that displays connector FRU and port
information for that connection. At the top of the window is the connector name. There are two
parts of the window, the cable FRU ID information on the left, and a smaller status pane for
the ports on the right. Clicking a tab displays that port’s information, including the physical
state, logical state, protocol, and any configured vNICs.

Multiple Addresses Versus Multiple Ports
• To run multiple WebLogic Servers on a single compute

node, each type of communication must have a unique
combination of IP/port.
• Oracle recommends assigning each server:
– Unique “floating/virtual” IP addresses
– The same port numbers
• This approach is also required when using the server
migration feature.
Machine Machine
192.168.1.1:7003 ServerA OR
192.168.1.1:7003 ServerA
192.168.1.1:7005 ServerB 192.168.1.2:7003 ServerB
Multiple Addresses Versus Multiple Ports

Using Oracle’s recommended enterprise deployment configuration, create and assign
individual floating IP addresses for each of the WebLogic Managed Servers and for the
Administration Server. Floating IP addresses are IP addresses that may be re-assigned
between compute nodes in the cluster. For example, if a compute node fails or goes down,
then WebLogic Managed Servers running on one compute node can be migrated to another
compute node. When WebLogic Node Manager is set up, it manages the registration and
removal of the floating IP addresses. This overall process is referred to as WebLogic whole
server migration.

Configuring a Virtual Address
Add an IP address to the bond0 (IPoIB) interface:
ifconfig bond0:1 192.168.100.100 netmask 255.255.255.0 up
(or ip addr add 192.168.100.100/24 dev bond0)
Update the routing tables of other hosts on this network:
arping –U -I bond0 192.168.100.100
Configuring a Virtual Address

ifconfig [interface] [address] netmask [mask] up/down
arping [ -AbDfhqUV] [ -c count] [ -w deadline] [ -s source] [-I
interface] [destination]
Common uses for ifconfig include setting an interface's IP address and subnet mask, and
disabling or enabling a given interface. At boot time, the operating system initializes its
network interfaces with shell scripts that call ifconfig. Other floating IP addresses can be
dynamically added and removed after the OS has booted, but these will no longer be present
after the server is restarted.
With logical network interfaces, you can configure a physical interface such as eth0 to have
many different IP addresses, even IP addresses that are in different IP classes. Logical
interfaces consist of the physical unit’s name as well as a logical name, separated by a colon.
For example, “eth0:1”, “eth0:2”, and so on.
The arping command is used to discover hosts on a network. The program tests whether a
given IP address is in use on the local network, and can get additional information about the
device using that address. It is analogous in function to ping but operates at layer 2 (Link)
instead of layer 3 (Internet). Include the -U option to send a broadcast message by using the
interface specified by the -I option. The -q option disables all command output (quiet). The -
c option indicates the number of packets to attempt to send and wait for a reply.
Network Configuration and Multi-Tenancy
Optionally configure the Exalogic network to:

• Allow a single LAN to appear to be multiple LANs
• Isolate traffic for different groups of compute nodes
• Restrict the databases and other external systems that
specific compute nodes can access
Network Layer Technique
Physical Virtual LAN (VLAN)
Internet IP Subnet Partitioning
Network Configuration and Multi-Tenancy

VLANs are essentially layer 2 entities, compared with IP subnets, which are layer 3. In an
environment employing VLANs, a one-to-one relationship often exists between VLANs and IP
subnets, although it is possible to have multiple subnets on one VLAN or have one subnet
spread across multiple VLANs.
Creating a VLAN might be necessary on an Exalogic switch if you wish to connect to a
database whose network is already configured with a VLAN. In addition, VLANs might be a
useful technique for isolating database traffic within the data center or across data centers.

Virtual LAN (VLAN)
A VLAN:
• Groups together physical network connections that are
allowed to communicate with one another
• Enables remote hosts to communicate as if they were on a
single LAN
• Is registered on a specific port of a switch
• Is assigned a numeric ID
Switch
Port1: VLAN10
Port2: VLAN10
Port3: VLAN20
Port4: VLAN30
Virtual LAN (VLAN)

Static VLANs are essentially port-based VLANs. They are created by assigning ports to a
VLAN. As a device enters the network, the device automatically assumes the VLAN of the
port. If the device later changes physical ports and needs access to the same VLAN, the
network administrator must manually make a port-to-VLAN assignment for the new
connection. No traffic is permitted across different VLANs.

Configuring a VLAN
Display all VLANs on this switch:
# showvlan
Connector VLN PKEY

-----------------------
1A-ETH-2 3 1234
Assign a VLAN to a switch port:
# createvlan 1A-ETH-3 -VLAN 3 –PKEY default
Switch VLAN ID IB partition ID

connector ID
Configuring a VLAN
You can use the showvlan command to identify which VLANs are associated with the IB
gateway 10 Gb Ethernet connectors. The command lists the connectors on the left, and the
associated VLANs and respective partition keys on the right.
When you create a VLAN, you create a mapping between a gateway connector, a VLAN
identifier, and a partition key (0000-7fff). By default on Exalogic only one partition is available,
so only one partition key is valid. You can create VLANs with the createvlan command. If a
VNIC will be associated to a VLAN, you must create the VLAN before the VNIC.

Subnet Partitioning
IP subnets:
• Enable hosts on the same local network to be isolated
from one another
• Are configured and enforced at the OS level by using
subnet masks
Exalogic
NodeA NodeB NodeC

192.168.1.1 192.168.1.2 192.168.1.10
255.255.255.248 255.255.255.248 255.255.255.248
Subnet Partitioning
When planning an Exalogic deployment across a collection of compute nodes, use a subnet
mask that provides a suitable IP address range to cover all of these servers on the target
group of compute nodes. In some scenarios, the application from one department may require
communication with the application of another department, each of which is hosted on
separate compute nodes. To enable the applications to communicate, you must set up
another IP subnet in which all target compute nodes are members.

Quiz
How many InfiniBand switches are included in a full rack?

a. Two
b. Three
c. Five
d. Six
Answer: c

Quiz
Which of the following is not a task that can be performed with

the switch CLI?
a. Check LED status.
b. Configure a VLAN.
c. Create a virtual IP.
d. Set the SM priority.
e. View the network topology.
Answer: c

Quiz
How many physical Ethernet connections does a single IB

gateway switch support?
a. Two
b. Four
c. Six
d. Eight
Answer: d

Summary

• Describe the network topology for various rack
configurations
• Identify switch LEDs and replace failed hardware
• Access an InfiniBand switch from the browser or command
line
• Describe the role of the InfiniBand subnet manager
• Configure a compute node's network to support FMW

Practice 7-1
Configure Network Settings
This practice covers configuring virtual IP addresses on a
compute node to support a future FMW installation.

Creating FMW Domains

Objectives

• List the tasks involved in creating a domain
• Install WebLogic Server on Exalogic
• Create a WebLogic domain on Exalogic
• Create and share domains from the command line

WebLogic Concepts: Review
Machine Machine Machine
/domainB /domainB
/domainA /domainA
Admin
Server
Server Server
Admin
Server Server Server Server
Cluster Cluster
Node Node
Node Manager
Manager Manager
Product Installation
WebLogic Concepts: Review

An Oracle WebLogic Server administration domain is a logically related group of Oracle
WebLogic Server resources. Domains include a special Oracle WebLogic Server instance
called the Administration Server, which is the central point from which you configure and
manage all resources in the domain. Usually, you configure a domain to include additional
Oracle WebLogic Server instances called managed servers. You deploy web applications,
EJBs, web services, and other resources onto the managed servers and use the
Administration Server for configuration and management purposes only.
You can use a single Oracle WebLogic Server installation to create and run multiple domains,
or you can use multiple installations to run a single domain. How you organize your Oracle
WebLogic Server installations into domains depends on your business needs. You can define
multiple domains based on different system administrators’ responsibilities, application
boundaries, or geographical locations of the machines on which servers run. Conversely, you
might decide to use a single domain to centralize all Oracle WebLogic Server administration
activities.
For development or test environments, you can create a simple domain that consists of a
single server instance. This single instance acts as an Administration Server and hosts the
applications that you are developing.

WebLogic Communication Types
Type Description Preferred Network

Web (HTTP) Client Access to web or web service Client or Private
applications (internal web service)
Java (T3) Client Access to JMS or EJB applications Client or Private
Admin Interactions between admin and Private
managed servers in a domain
Admin Console Access to the WebLogic Client
administration console
Cluster Heartbeat Periodic messages between cluster Private
members to determine availability
Cluster Replication Exchange of user data between Private
cluster members for high availability
Database Outgoing JDBC communication Client or Private
(Exadata)
WebLogic Communication Types

If you want to expose business logic in a standard way to other applications, both within your
organization and beyond, use a web service application. XML-based web services can be
accessed by both Java and non-Java applications and are also a cornerstone for service-
oriented applications (SOA). If the web service is accessible by other remote systems and
external partners, it must be made available on the Exalogic client network. If instead the web
service is simply a reusable business logic component that is accessed by other applications
within the same data center, the Exalogic private IB network may be a better candidate.
WebLogic applications often communicate with external database systems. In Java EE, this is
accomplished using Java Database Connectivity (JDBC). Most databases are hosted outside
of Exalogic and, therefore, outside of the private IB network. In these cases, WebLogic’s
JDBC configuration settings would specify the Exalogic client network. However, if your
database is hosted on an Oracle Exadata machine and is connected to the same IB network
as your Exalogic machine, JDBC communication should use that network instead.

Installation and Configuration Process
1. Run the WLS installer and place it onto shared storage.

2. Run other FMW installers if necessary.
3. Run the FMW Configuration Wizard to define a new
domain:
a) Select the appropriate template or templates.
b) Place the domain onto shared storage.
c) Define initial servers, clusters, and machines.
d) Use bond0 (private) addresses for default server
communication.
4. Start the administration server for the domain.
5. Use the WLS console or scripting tool to make additional
modifications.
Installation and Configuration Process

Install the Oracle WebLogic product binaries on one of the shares in the Sun ZFS Storage
7320 appliance. The share must be accessible by all compute nodes that are going to run this
version of WebLogic.
Ensure that you do not run the Oracle WebLogic installer as the root user. Instead, log in as a
non-root user who has write privileges to the appropriate shares mounted on the Sun ZFS
Storage 7320 appliance. Similarly, do not create domains or run server processes as root.
The product installation includes a set of predefined domain and extension templates. This
set includes the base WebLogic Server domain template and various extension templates that
enable you to add component features and samples to the base domain.
The WebLogic web-based administration console runs on a domain’s administration server.
Using this tool, you can dynamically create or delete servers, modify the settings of existing
servers, add servers to a cluster, or remove servers from a cluster. You can also use the
console to create JDBC database connections and deploy Java EE applications.

Shared or Dedicated Installations?
• Sharing a single FMW installation across domains:

– Requires fewer product binaries/patches to maintain
– Enforces consistency throughout the organization
• Dedicating an installation for each domain:
– Provides greater management flexibility
– Enables you to maintain product binaries/patches
independently and on different schedules
Domain 1
FMW Installation 1
Domain 2
FMW Installation 2 Domain 3
Shared or Dedicated Installations?

Exalogic uses a shared storage appliance that allows all WebLogic instances in Exalogic,
including instances running in different WebLogic domains, to share the same product
installations. Topologies that use shared installations across domains and physical servers
offer some advantages over topologies that use dedicated installations per domain or physical
server. Shared installation topologies create fewer sets of product binaries to be managed,
simplify the mapping of WLS instances to the installation being used, and enforce maximum
consistency of versions and maintenance levels in the Exalogic system. Therefore, shared
installation topologies may result in management efficiencies.
However, in some scenarios, you may require multiple installations, each dedicated to
specific WebLogic domains or to compute nodes. Topologies with multiple, dedicated
installations provide more management flexibility, particularly when maintenance
considerations are important. Applications running in different domains may have different
maintenance requirements. The frequency of their updates may vary, and the update
requirements may affect different functional areas of the product, resulting in diverse patch
requirements. They may also host applications from different departments or business units,
and require that their applications and systems be isolated from other applications to minimize
cross-application dependencies.

Available WebLogic Installers
Installer Type Description
Platform-Specific • Includes a JVM for the target platform (for example,

x64 Linux or Solaris)
• May also include Oracle Coherence and/or Eclipse
IDE plug-ins
Generic (JAR) • Requires a JVM to run
• Can be used on any supported platform
ZIP • For development purposes only
• Very small footprint
• Cannot be used with other FMW products
./wls1034_linux64.bin
java –jar wls1034_generic.jar
Available WebLogic Installers

• Oracle WebLogic Server + Coherence + OEPE Package Installer:This installer is an
installation program that enables you to install Oracle WebLogic Server components
and includes Oracle Coherence and Oracle Enterprise Pack for Eclipse.
• Oracle WebLogic Server + Coherence Package Installer: This installer is an
installation program that allows you to install Oracle WebLogic Server components and
includes Oracle Coherence.
• Oracle WebLogic Server ZIP Installer: This is a zip distribution for Oracle WebLogic
Server intended for WebLogic Server development only. It contains the necessary
artifacts for installing and running WebLogic Server and cannot be used as an
installation target for other FMW products. It does not contain the following optional
components: samples, Derby Evaluation Database, or web server plug-ins. It does not
include Oracle Coherence or Oracle Enterprise pack for Eclipse. The zip distribution is
supported on Windows, Linux and Mac OS X platforms

Installation Overview
Installation Overview
1. Specify the Oracle Middleware Home directory that will serve as the central directory for
all the Oracle products installed on the target system. You can optionally also enter your
email and account information for online support.
2. A typical Oracle WebLogic Server installation includes all server components, along with
Oracle Coherence if it is part of the installer. It does not include sample code and scripts
for WebLogic and Coherence, however. This option is recommended for production
deployments.
3. In most instances, you can simply accept the default product installation paths, which
are found under the Middleware Home directory you provided earlier.
Oracle also recommends that you make a backup copy of your installation at this stage, and
before the installation of any additional FMW products. However, keep in mind that, with the
exception of product patches, you should not have to modify any of the installation files.
Modifications should be made at the domain level.

Domain Templates
• The FMW Configuration Wizard creates WebLogic

domains based on templates.
• WLS and FMW products include base templates. You can
also build your own.
Configuration
Wizard
Read templates Create domain
cd <WL_HOME>/common/bin
./config.sh
Domain Templates
A domain is the basic administration unit of WebLogic Server. It consists of one or more
WebLogic Server instances, and logically related resources and services that are managed,
collectively, as one unit. In addition to infrastructure components such as servers and
clusters, a domain defines application deployments, supported application services (such as
database and messaging services), security options, and physical host machines.
The Configuration Wizard guides you through the process of creating a domain for your target
environment by selecting the product components that you want to include in your domain, or
by using domain templates. If required, you can also customize the domain to suit your
environment by adding and configuring managed servers, clusters, and machine definitions,
or customizing predefined JDBC data sources, and JMS file store directories.
A domain template defines the full set of resources within a domain, including infrastructure
components, applications, services, security options, and general environment and operating
system options. You can create this type of template from an existing domain by using the
Domain Template Builder, WLST, or the pack command-line tool. Subsequently, you can
create a domain based on the template by using the Configuration Wizard or WLST.

Configuration Wizard: Overview
Select template(s)
to start from.
4
3
5

1. If you want to create a new domain, select the “Create a new WebLogic domain” option.
If you want to add another template to an existing domain, select the other option.
2. Depending on which FMW products you have installed, you are presented with a list of
template options. For basic WebLogic domains, use the default template. For other
products, refer to the documentation regarding which template or templates to choose.
3. Provide the name of the domain home folder along with a location on the file system.
This location should be on the storage appliance to make it accessible to all compute
nodes participating in this domain.
4. Enter the default administrative credentials for this domain. The name and password are
stored in an internal Lightweight Directory Access Protocol (LDAP) directory by default.
You can later move to an external identity store if you want.
5. Select the Production Mode option, which requires administrators to authenticate when
starting servers or changing the domain's configuration.

6
Admin server
settings
Initial list of
managed servers
7
Use virtual IP
addresses.
Configuration Wizard: Overview (continued)

6. Define the administration sever for this domain. Supply a logical name, a listen address,
and a port number. Use a virtual IP address if you plan to use the server migration
feature.
7. Define a list of servers that will be participating in this domain. Each server is given a
logical name, a listen address, and a port number. Use virtual IP addresses if you plan
to run multiple servers on the same compute node, or alternatively use separate port
numbers. When started, these servers will contact the domain's administration server to
obtain all of their configuration parameters.

8
9
10
11
Exalogic
compute nodes
Configuration Wizard: Overview (continued)

8. Define one or more clusters. Simply provide a logical name for each.
9. Select which servers will be members of each cluster.
10. Define a machine for each compute node that will be used to run servers in this domain.
For each machine, supply a logical name along with the listen address and port number
that the WLS Node Manager process will be bound to on that machine.
11. Specify which servers will initially run on each machine.

Shared or Copied Domains?
• Multiple servers can share a single domain folder, but NFS

data integrity and file lock issues can appear if you use:
– File-based persistence for transaction/message recovery
– The transaction/message migration features
• Alternatively, create a new domain folder for each compute
node on shared storage.
Node 1 Node 1
NFS
Servers /node1/mydomain Servers
OR NFS
/mydomain Node 2 Node 2
Servers /node2/mydomain Servers
Shared or Copied Domains?

Oracle strongly recommends verifying the behavior of a server restart after abrupt machine
failures when the JMS messages and transaction logs are stored on an NFS-mounted
directory. Depending on the NFS implementation, different issues can arise after a failover or
restart. WebLogic Server maintains locks on files used for storing JMS data and transaction
logs to protect from potential data corruption if two instances of the same WebLogic Server
are accidentally started.
NFS storage may not fully protect transactional data, because it may be configured to silently
buffer synchronous write requests in volatile memory. If a WebLogic file store directory is
located on an NFS mount, and the file store's Synchronous Write Policy is not disabled, check
your NFS implementation and configuration to make sure that it is configured to support
synchronous writes. A disabled synchronous write policy does not perform synchronous
writes but, as a consequence, is generally not transactionally safe. You might detect
undesirable buffering of synchronous write requests by observing high persistent message or
transaction throughput that exceeds the physical capabilities of your storage device. On the
NFS server, check the synchronous write setting of the exported NFS directory that hosts
your File Store. A SAN-based file store or a JDBC store may provide an easier solution for
safe centralized storage.
Refer to http://download.oracle.com/docs/cd/E17904_01/web.1111/e13814/storetune.htm.

Migrating Existing Domains
• A simple file system operation is often not sufficient to

move or copy a domain folder.
• Create an intermediary template by using one of these
tools:
– Pack/unpack command-line tools
– WebLogic Scripting Tool (WLST)
• Old domains created with WLS 9.x or 10.x can run on the
latest 10.3 version without modification.
• Older versions require that you run the Upgrade Wizard
tool.
Migrating Existing Domains

The unpack command provides a one-step method for creating a domain quickly from an
existing template by using the default settings defined in the template. When you create a
domain, the unpack command does not provide the same customization options as the
Configuration Wizard.
The 10.3.4 WebLogic Upgrade Wizard enables you to upgrade domains created in WebLogic
Server 7.0 or 8.1. You can also use the WebLogic Upgrade Wizard to upgrade a WebLogic
domain created in WebLogic Server 9.x or 10.0 to 10.3.4, but this is optional. This type of
domain runs under WebLogic Server 10.3.4 without modification. However, note that if you do
not package the domain as a template, you will need to manually update the script files in the
domain to point to the new installation of WebLogic Server. For example, set MW_HOME,
MW_HOME, BEA_HOME, BEA_JAVA_HOME, JAVA_HOME, and WL_HOME to the appropriate
values.
Oracle does not recommend upgrading an application environment that is currently deployed
in production. Instead, you should upgrade your application environment while it is under
development or test and execute standard procedures for quality assurance and performance
tuning before promoting the upgraded environment to production.

Pack and Unpack Tools
These utilities are found at <WL_HOME>/common/bin.
Create a template from an existing domain:
pack.sh –domain=domains/test/mydomain
–template=templates/mydomain.jar
-template_name="My Custom Template"
Create a domain folder by using a custom template:
unpack.sh –template=templates/mydomain.jar
–domain=domains/prod/mydomain
Create a template to support managed servers only (no admin server):
pack.sh –managed=true –domain=...
Pack and Unpack Tools

The pack command provides an alternative method for creating a template from the
command line in one simple step. Although the pack command does not enable you to
customize the contents of your template in the same way as the Domain Template Builder, it
is useful for performing the following tasks quickly:
• Creating a domain template that contains a snapshot of an entire working domain. You
can then use this template as the basis for a new domain.
• Creating a managed server template that contains a subset of the files in a domain that
are required to create a managed server domain directory hierarchy on a remote
machine
When you use the unpack command with a managed server template, it creates a managed
server domain directory. An entry for the managed server domain directory is also created in
the NM_HOME/nodemanager.domains file, where NM_HOME designates the Node Manager
installation directory for the product installation on the remote machine. By default, this
directory is located in WL_HOME/common/nodemanager.

Starting the Administration Server
• Domains include a startWebLogic script to launch the

administration server.
• Create a file to avoid supplying credentials:
<domain>/servers/<server>/security/boot.properties
• The file is encrypted during the initial startup.
boot.properties: startWebLogic
username=weblogic
password=mypassword setDomainEnv
commEnv
Starting the Administration Server

To start a server within a domain, you can use the generated startWebLogic script or
develop your own custom scripts. The default startWebLogic script executes your
domain’s setDomainEnv script. This script in turn calls a script named commEnv, which is
included with your product installation. The commEnv script uses the setPatchEnv script,
which is responsible for initializing variables that point to your currently installed patches.
Another script that makes use of commEnv is setWLSEnv, which is not directly used to start
servers. Instead, it provides a convenient way of initializing your environment to support
WebLogic developer and administrator tools including Ant and WLST.
A boot identity file is a text file that contains user credentials for automatically starting and
stopping an instance of Oracle WebLogic Server. Technically you are not “logging in” as
anybody, but simply validating that you have the credentials (authority) to start the instance.
An administration server can refer to this file for user credentials instead of prompting you to
provide them. Because the credentials are encrypted, using a boot identity file is much more
secure than storing unencrypted credentials in a startup or shutdown script. If there is no boot
identity file when starting a server, the server instance prompts you to enter a username and
password.

WebLogic Console: Overview
1 http://<admin_ip>:7001/console
2
4
WebLogic Console: Overview

1. Access the console from your domain's administration server.
2. To modify the configuration for a production domain, click the Lock & Edit button in the
Change Center.
3. With the help of the Domain Structure panel, navigate to a specific resource to edit.
4. Parameters are organized into one or more levels of tabs.
5. Edit one or more parameters and click Save.
6. Repeat the process to modify additional resources. Click Activate in the Change Center
to apply your changes to any running servers.

Adding a New Server
Adding a New Server

1. From the Domain Structure panel, click Environment > Servers.
2. In the right panel, click the New button. Alternatively, copy an existing server
configuration by using the Clone button.
3. Enter a name, listen address, and listen port for the new server. To also add the server
to your cluster, select “Yes, make this server a member of an existing cluster.”

Updating Machine Membership
1
4
Updating Machine Membership

1. From the Domain Structure panel, click Environment > Machines.
2. In the right panel, select an existing machine or click New to create a new one.
3. Click the Configuration > Servers tab.
4. Use the Add and Remove buttons to modify the list of servers associated with this
machine.

Updating Cluster Membership
Updating Cluster Membership

1. From the Domain Structure panel, click Environment > Clusters.
2. In the right panel, select an existing cluster or click New to create a new one.
3. Click the Configuration > Servers tab.
4. Use the Add and Remove buttons to modify the list of servers associated with this
cluster.

Quiz
In a WebLogic configuration, a server can be associated with

which of the following? (Choose three.)
a. Desktop
b. Cluster
c. Domain
d. Machine
e. Node Manager
Answer: b, c, d

Quiz
Name two tools that can create a domain folder.

a. Pack
b. Unpack
c. Console
d. Template Builder
e. Config Wizard
Answer: b, e

Summary

• List the tasks involved in creating a domain
• Install WebLogic Server on Exalogic
• Create a WebLogic domain on Exalogic
• Create and share domains from the command line

Practice 8-1
Initialize a WebLogic Domain
• Installing WebLogic Server on shared storage
• Creating a new WebLogic domain on shared storage
• Defining machines, servers, and clusters
• Customizing server memory settings
• Starting a domain's administration server
• Editing a domain by using the WebLogic administration
console

Optimizing FMW Domains on Exalogic

Objectives

• Describe the WebLogic default network behavior
• Configure network channels for a WebLogic Server
• Discuss some optimizations that WebLogic Server
performs on Exalogic

Additional Configuration Tasks
1. Configure WebLogic network channels to use:

– bond0 for default admin communication
– bond0 for cluster replication traffic
– bond1 for client communication
2. Enable these features on replication channels:
– SDP
– Multiple ports
– Lazy deserialization
3. Enable other domain-wide Exalogic optimizations.
Additional Configuration Tasks

Most Exalogic specific WLS optimizations are not enabled by default. Some require a simple
configuration check box. But others require multiple configuration steps. For example, WLS
clusters can be configured to further improve server-to-server communication. First, you can
enable multiple replication channels, which improve network throughput among cluster
members. Second, you can enable InfiniBand support via the Sockets Direct Protocol, which
reduces CPU utilization because network traffic bypasses the TCP stack.

Default WebLogic Networking
The default server port:

• Accepts all protocols (HTTP, T3, IIOP, SNMP)
• Supports various security and tuning parameters
• Is used for client/server communication
• Is used for remote server management (console, WLST)
• Is used for internal server/server communication
– Server startup and management messages
– Cluster messages
Default WebLogic Networking

You can configure each WebLogic Server instance to communicate over a variety of
protocols, such as HTTP, Hypertext Transmission Protocol, Secure (HTTPS), and Internet
Inter-ORB Protocol (IIOP). In addition, you can configure general server network settings that
apply to all protocols. The default listen address and port accept all types of incoming server
communications, including:
• Web application HTTP requests
• Remote access to the server Java Naming and Directory Interface (JNDI) tree
• Remote Enterprise JavaBeans (EJB) application Remote Method Invocations (RMI)
• Simple Network Management Protocol (SNMP) polling requests
• Configuration and monitoring requests from remote management tools, such as the
console or WLST
• Configuration and monitoring requests sent from the Administration Server to the
Managed Server
• Initial startup messages sent from a managed server to the Administration Server
• Messages sent between cluster members, such as for object replication

Isolating Client Communication
• Use separate addresses and/or ports for client and admin

communication.
• Disable client access points during server maintenance or
troubleshooting.
IP1:7001
Admin
Server
IP2:8001
Console Startup
Update
Monitor
IP3:8001
Managed
Server
IP4:7001
Isolating Client Communication

While maintaining or troubleshooting a production server, it is often desirable to disable all
incoming application requests. However, a server’s default network configuration implies that
all traffic runs on the same port. Therefore, if the port were closed, all remote administration
tools, such as the console or WLST, would also not be able to connect to the server.
WebLogic Server supports a domain in which all servers use a separate SSL port that
accepts only administration requests. The use of dedicated admin ports enables you to:
• Start a server in standby state: This allows you to administer a server, whereas its
other network connections are unavailable to accept client connections.
• Separate administration traffic from application traffic in your domain: In
production environments, separating the two forms of traffic ensures that critical
administration operations (starting and stopping servers, changing a server’s
configuration, and deploying applications) do not compete with high-volume application
traffic on the same network connection.
• Administer a deadlocked server instance: If you do not configure an administration
port, administrative commands such as THREAD_DUMP and SHUTDOWN will not work on
deadlocked server instances.

Isolating Cluster Communication
• Use dedicated addresses and/or ports for peer-to-peer and

broadcast cluster messaging:
– Server heartbeats
– Session replication
IP3:7001
ServerB
IP4:9001
IP1:9001
ServerA
IP2:7001 IP5:9001
ServerC
IP6:8001
Isolating Cluster Communication (continued)

Similar to administration ports, the servers within a cluster can also use separate ports
dedicated to internal cluster communication. Administrators have the option to configure these
clusters or “replication” ports to use either a standard or a secure (SSL) protocol.
In general, ports on a server that can be used to send internal messages to other servers in
the same domain are called “outgoing” ports. Ports that are not enabled for “outgoing” are
used solely to process incoming client requests.

Session Replication: Overview
• Each user’s session data always exists on two servers:

– Primary
– Secondary
• Every update to the primary session is automatically
replicated on the secondary server.
• If either fails, a new server is automatically selected.
Cluster
Server Server Server C
Primary Secondary
Session Replication: Overview

Using in-memory replication, Oracle WebLogic Server copies a session state from one server
instance to another. The primary server creates a primary session state on the server to which
the client first connects and a secondary replica on another Oracle WebLogic Server instance
in the cluster. The replica is kept up-to-date so that it can be used if the server that hosts the
web application fails.

Network Channels
• A network channel consists of a:

– Listen address and port
– Single supported protocol along with its service
characteristics
• Each server:
– Has an implicit default channel, which can be disabled
– Has a default SSL channel if configured
– Supports all protocols by default
– Can be assigned additional channels
• Channels can be created, enabled, or disabled
dynamically without restarting the server.
Network Channels
A network channel is a configurable resource that defines the attributes of a network
connection for a specific WebLogic Server instance. A network channel definition includes a
listen address, port number, supported protocol, and whether it can be used for internal
server-to-server communication. You can use network channels to manage quality of service,
meet varying connection requirements, and improve the utilization of your systems and
network resources.
Administrators create a channel for a specific server instance. Channels are not defined
globally and applied to one or more servers. You can assign multiple channels to a server
instance, but each channel must have a unique combination of listen address, listen port, and
protocol. Similarly, if you assign non-SSL and SSL channels to the same server instance,
make sure that they do not use the same port number.
If you want to disable the non-SSL listen port so that the server listens only on the SSL listen
port, deselect Listen Port Enabled in the Configuration > General settings for the server.
Similarly, if you want to disable the SSL listen port so that the server listens only on the non-
SSL listen port, deselect SSL Listen Port Enabled. Note that unless you define custom
network channels, you cannot disable both the default non-SSL listen port and the SSL listen
port. At least one port must be active on a server.

Creating a Channel
Configure channels for

4
each individual server.
1
2
Creating a Channel
To configure a network channel for an existing server, perform the following steps:
1. Select the server, and then click its Protocols tab.
2. Click the Channels subtab.
3. Click New.
4. Enter a name for the channel, select the protocol it will accept or use, and click Next.
For administrative channels, select the admin protocol. For cluster channels, select the
“cluster-broadcast” or “cluster-broadcast-secure” protocols.

Creating a Channel
Does this channel support

5 HTTP, in addition to the
selected protocol?
7
Can this channel be used for
internal communication with
other servers in this domain?
Creating a Channel (continued)

5. Enter a listen address and listen port that this channel will bind to, and click Next. If an
address is not supplied, the address of the default channel will be used.
6. Click Next. By default, the new channel will be enabled and automatically bind to its
address and port. If instead you want to enable it manually at a later time, deselect the
Enabled check box. Other options include:
- HTTP Enabled for This Protocol: Specifies whether HTTP traffic should be allowed
over this network channel. HTTP is generally required for other binary protocols for
downloading stubs and other resources (only applicable if selected protocol is not
HTTP or HTTPS).
- Outbound Enabled: Specifies whether new server-to-server connections may
consider this network channel when initiating a connection. Leave this field
deselected for client channels.
7. For secure protocols, optionally enable two-way SSL. Click Finish.

Some Channel Protocols
Value Description
http Accepts requests for web applications

https Accepts SSL requests for web applications
t3 • Accepts requests for EJB/JMS applications, or
• Used for internal replication messages within a
cluster
t3s Accepts SSL requests for EJB/JMS applications
snmp Accepts SNMP requests
admin Used for internal administrative communication
between servers
cluster-broadcast Used for internal heartbeat messages within a cluster
Some Channel Protocols

Each channel requires a listen address and port. If your server is being accessed through a
proxy server on a separate listen address and/or port, you may be required to supply an
external listen address and/or external listen port for each channel. These values will be used
in cases where the server must publish its location to external clients, such as a web server
plug-in or a hyperlink in a web browser.

Creating a Cluster Replication Channel
1. Create an internal T3 channel for each member of a

cluster.
2. Assign the channel to the cluster (default name is
“ReplicationChannel”).
Creating a Cluster Replication Channel

When creating a network channel, there is not a specific protocol option available for internal
cluster replication traffic. Instead, you must configure a channel that supports the T3 protocol
and specify the name of that channel as part of a cluster’s replication settings.
1. Click Clusters and then select a cluster.
2. Click the Configuration > Replication tab.
3. Set the Replication Channel to the name of the custom network channel that you
created on each member of the cluster. As a result, these channels will be used for
replication traffic instead of the default channels on each server.

Using SDP for Replication
Using SDP for Replication

WLS can use SDP to bypass the operating system and communicate directly with the
InfiniBand fabric for session replication.

Using Multiple Ports for Replication
• A single replication channel cannot use the available

bandwidth on the IB network.
• For convenience, simply specify a port range and multiple
channels will be created and used automatically.
Using Multiple Ports for Replication

When members of a cluster need to communicate and replicate HTTP session data for high
availability, they use an internal Java protocol called T3. Because Exalogic uses InfiniBand,
establishing individual T3 connections does not take full advantage of the available
bandwidth. Instead, WLS uses parallel or “multiplexed” connections for this inter-cluster
communication.
Multiple replication channels do not need to be configured on each clustered server instance.
Only one replication channel with explicit IP-Address needs to be configured for each server
and replication. Port range can be specified for each server. For example, the range 7001-
7010 will create 10 replication channels with ports 7001 to 7010 for the given server. These
channels inherit all the properties of the configured replication channel except the listen port.
Names of these channels will be derived from the name of the configured replication channel.
A numeric suffix (1,2,3) will be added to each name. If you specify a range of 10 ports, 10
additional channels will be created. Public ports are the same as the listen port for these
additional channels.

Other Replication Optimizations
startWebLogic.sh:
...
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true"
Other Replication Optimizations

As replicated Java objects are passed across the InfiniBand network, they must be serialized
and deserialized by WLS. To avoid unnecessary processing, WLS deserializes a replicated
object only if the server from which it originated has failed.

Other WebLogic Optimizations
Subsystem Description
Threading Create more threads to take advantage of the compute

node's processing power.
Network I/O • Tighter integration with native IB libraries
• Use larger packet sizes to take advantage of the IB
throughput.
File I/O Use larger buffers to take advantage of the IB connection
to the storage appliance.
Other WebLogic Optimizations

Traditional I/O employs the use of buffers to read and write data, and as this data is
transferred between operating system memory, JVM memory, and WLS memory (heap), it
must be copied from buffer to buffer. On Exalogic, this file I/O buffering is significantly reduced
or eliminated.
InfiniBand supports much higher throughput rates compared to standard Ethernet, so
WebLogic automatically uses larger packet sizes to communicate with the InfiniBand fabric on
Exalogic. This includes external network communication as well as communication with other
servers on other compute nodes.

Enabling Other Optimizations
Optimizations can be turned on and off:

• Collectively by using a domain-wide flag
• Individually by using command-line server arguments
Enabled for all servers

in this domain
Enabling Other Optimizations

To enable all general optimizations on all servers in a domain, perform the following steps:
1. Access the WebLogic Server console.
2. In the Domain Structure panel, click the name of the domain.
3. In the right panel, select the Enable Exalogic Optimizations check box.
Refer to the Deployment Guide for a complete list of server startup arguments that individually
enable or disable specific enhancements. Here are some examples:
• Dweblogic.ScatteredReadsEnabled=true/false
• Dweblogic.GatheredWritesEnabled=true/false
• Dweblogic.replication.enableLazyDeserialization=true/false

Licensing Review
Only a license for Exalogic Elastic Cloud Software permits the

use of these WebLogic features:
• Exalogic optimizations flag
• Replication SDP flag

Quiz
Name three available network channel protocols.

a. t3
b. http
c. admin
d. machine
e. replication
Answer: a, b, c

Quiz
Which of the following is not an available optimization for

session replication on Exalogic?
a. Socket Direct Protocol
b. Node Manager
c. Lazy deserialization
d. Multiple ports
Answer: b

Summary

• Describe the WebLogic default network behavior
• Configure network channels for a WebLogic Server
• Discuss some optimizations that WebLogic Server
performs on Exalogic

Practice 9-1
Configure Network Channels and Optimizations
• Enabling WebLogic domain optimizations for Exalogic
• Configuring WebLogic network channels to support
Exalogic
• Configuring multiple cluster replication ports
• Optimizing replication communication

Starting and Migrating Servers

Objectives

• List the capabilities of Node Manager
• Configure a node manager for a compute node
• Start and kill servers by using Node Manager
• Explain the process of server migration
• Use floating IP addresses with Node Manager
• Configure persistent storage for server recovery

Node Manager: Review
Node Manager:
• Is a process that accepts remote commands to start, stop,
or suspend servers on the same machine
• Monitors server availability and restarts failed ones
• Can be used to migrate servers on a failed machine to
another machine
• Can communicate by using SSL if desired
Machine
Server
Node Start/stop
Manager Monitor/
restart Server
Node Manager: Review

Server instances in a WebLogic Server production environment are often distributed across
multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server
utility with which you can start, shut down, and restart Administration Server and Managed
Server instances from a remote location. Although Node Manager is optional, it is
recommended if your WebLogic Server environment hosts applications with high-availability
requirements. A Node Manager process is not associated with a specific WebLogic domain
but with a machine. You can use the same Node Manager process to control server instances
in any WebLogic Server domain, as long as the server instances reside on the same machine
as the Node Manager process.
If a server instance that was started by using Node Manager fails, Node Manager
automatically restarts it. If Node Manager fails or is explicitly shut down, upon restart, it
determines the server instances that were under its control when it exited. Node Manager can
restart any failed server instances as needed.

Node Manager: Architecture
Admin Server, SSL

Node Manager
WLST, EM Credentials
Trusted NM
NM Properties
Domains List Credentials for
Domain XYZ
Node Manager: Architecture

The Administration Console can be used to issue commands to Node Managers that are
running on remote machines. WLST also serves as a Node Manager command-line interface
that can run in the absence of a running Administration Server. Other FMW tools such as
Enterprise Manager Grid Control support integration with Node Manager as well.
Each Node Manager is associated with a file named nodemanager.properties that configures
its listen address and port number, as well as other settings for SSL, logging, and so on.
The nodemanager.domains file specifies the domains that a Node Manager instance controls,
as well as their locations on the file system. This file provides additional security by restricting
Node Manager client access to the domains listed in this file.
The nm_password.properties file contains the Node Manager username and password.
These are used to authenticate the connection between a client (for example, the
Administration Server) and Node Manager. These credentials are independent from the WLS
administrative account used to boot servers. The Node Manager username and password do
not have to be identical for every domain running on the same machine. When you create a
new domain, it automatically generates a random username and password that the
Administration Server will use to access Node Managers. But these default credentials can be
modified using the WebLogic console.

Node Manager: Configuration Tasks
1. Copy template NM files from

<WL_HOME>/common/nodemanager and
<WL_HOME>/server/nodemanager.
2. Configure nodemanager.properties.
3. Update each machine definition and specify the location of
the node manager.
4. Run the nmEnroll WLST command to establish trust
between the admin server and each node manager.
Node Manager: Configuration Tasks

Though you can launch Node Manager from within the WebLogic product installation, this is
not a recommended practice. Product installations should be treated as read-only. Instead,
create a separate file system location from which to run each Node Manager.
A WebLogic Server machine resource associates a particular machine with the server
instances it hosts, and specifies the connection attributes for the Node Manager process on
that system. Configure a machine definition for each machine that runs a Node Manager
process and then edit the settings in the machine’s Node Manager tab in the Administration
Console. Enter the DNS name or IP address upon which Node Manager listens in the Listen
Address field.

Recommended NM Storage Configuration
exalogic
Separate project for

HR each business unit
nodemanager
node1 node2
Pool
Separate folder for
Project
each compute node
Share
Recommended NM Storage Configuration

Oracle recommends that, whenever possible, you use the shared storage appliance to store
products, configuration files, logs, and applications, and that you not use the compute node's
local flash storage device. A node's local device is intended primarily as a boot device. Oracle
also recommends that each independent business or IT unit be modeled as a separate
project on the storage appliance. Then, within each project, include a “nodemanager” or
similar shared file system for storing all Node Manager configuration files and logs.
Within an Exalogic environment, each compute node that hosts WebLogic Servers runs a
single Node Manager process. Therefore, create a separate folder on the shared file system
with the name of the compute node.

Basic Node Manager Properties
Property Description
NodeManagerHome Location of NM property and log files
ListenAddress, Address and port from which NM will accept
ListenPort commands
Authentication Requires a username/password to use this NM
Enabled
SecureListener Requires SSL
StartScriptEnabled Starts servers by using local scripts
StopScriptEnabled Stops servers by using local scripts
StartScriptName Name of the script used to start local servers
(<domain>/bin/startWebLogic.sh by default)
StopScriptName Name of the script used to stop local servers
(<domain>/bin/stopWebLogic.sh by default)
Basic Node Manager Properties

Node Manager properties define a variety of configuration settings for a Java based Node
Manager process. In many environments, the SSL-related properties in
nodemanager.properties may be the only Node Manager properties that you must
explicitly define. However, nodemanager.properties also contains non-SSL properties
that you might need to specify, depending on your environment and preferences. Any values
supplied on the command line override the values in nodemanager.properties.
By default, the StartScriptEnabled, StopScriptEnabled, and
CrashRecoveryEnabled properties are false. The default value for
AuthenticationEnabled is true.
You can also use nodemanager.properties to specify the default restart parameters for
servers launched from this Node Manager instance:
RestartInterval: The amount of time Node Manager will spend attempting to restart a
failed server. By default, Node Manager will attempt to restart a server indefinitely until the
FAILED_NOT_RESTARTABLE state is reached.
RestartMax: The number of times Node Manager will attempt to restart a failed server within
the interval defined by RestartInterval. RestartMax is recognized only if
RestartInterval is defined.

Enrolling a NM with a Domain
Use the WebLogic Scripting Tool (WLST) to establish trust

between an NM and a specific domain.
Establish trust between a Node Manager and a running admin server:
java weblogic.WLST
connect('mydomainuser','mypassword','myadminhost:7001')
nmEnroll('/u01/HR/domains/MyDomain1',
'/u01/HR/nodemanager/node5')
Trusted NM
The second arg is not required if Domains List Credentials for
running WLST from this location. Domain XYZ
Enrolling a NM with a Domain

Use the nmEnroll command to create and update the necessary Node Manager
configurations files after creating a domain. You should run nmEnroll on each machine that
is running a managed server. Additionally, you should run nmEnroll for each domain
directory on each machine. WLST must be connected to an Administration Server to run this
command. WLST does not need to be connected to Node Manager.
The following files are created or updated:
• nm_password.properties
• SerializedSystemIni.dat (used to encrypt/decrypt credentials)
• nodemanager.domains
Optionally, specify the path of the root domain directory to which you want to save
nm_password.properties and SerializedSystemIni.dat. This argument defaults to
the directory in which WLST was started.
Optionally, specify the path of your Node Manager home and the nodemanager.domains
file. This argument defaults to <WL_HOME>/common/nodemanager, where <WL_HOME>
refers to the top-level installation directory for WebLogic Server.

Starting Servers from the Console
You can start or stop:

• Individual servers
• All servers in a cluster
• All servers in a domain
2
1
4
Starting Servers from the Console

You may only start a server from the WebLogic console if the Node Manager is running on the
machine to which the server is targeted. A server may be shut down from the console,
however, regardless of the state of the Node Manager.
1. From the Domain Structure panel, click either the domain name, the Servers link, or the
Clusters link. If Clusters is clicked, then also click the name of a cluster to start.
2. Click the Control tab.
3. Select one or more check boxes in the list of available servers. Then click the Start
button. You will be prompted for confirmation to start the selected servers. If a selected
server is not in an appropriate state to be started (for example, it is already running), it
will be ignored.
After a start request has been successfully sent to the server’s Node Manager, the server’s
state will indicate STARTING. Assuming that no critical errors occur during startup, the
server’s state will then transition to RESUMING and finally RUNNING.
If the Administration Server cannot establish a communication channel with the server’s Node
Manager, the start request will be ignored and the server’s state will be set to
FAILED_NOT_RESTARTABLE.

Starting Servers from the Command Line
Connect to a node manager and start an admin server:
java weblogic.WLST
nmConnect('nmuser','nmpass','myhost','5556','MyDomain',
'/u01/HR/domains/MyDomain')
nmStart('AdminServer')
Connect to an admin server and use it to start other servers, via the NM:
java weblogic.WLST
connect('domainuser','domainpass','myhost:7001')
start('Server2')
start('MyCluster','Cluster')
Starting Servers from the Command Line

For the special case of remotely starting the Administration Server, you must use WLST to
connect to the Node Manager process directly with the nmConnect command. You must
manually specify the location of the domain along with Node Manager credentials for the
domain. Once connected to the Node Manager, you may then use the nmStart command.
Similar to managed servers, Node Manager supports monitoring, stopping, and restarting the
Administration Server.
For convenience, if the domain’s Administration Server is running, you may issue server start
commands to the Node Manager via the Administration Server. Connect to the Administration
Server with WLST and then use the start command. You can start a single server or all
servers in the cluster. Note that in this scenario, you no longer have to provide the location of
the domain or the Node Manager credentials, because the Administration Server is now
responsible for doing these tasks.

Starting NM on Machine Startup
• Develop a custom OS init script that:

– Responds to the start argument sent by the init process
– Executes the NM start script
• Link the init script to the appropriate run level or register it
as a service.
/etc/init.d/startNodeManager:
...
case "$1" in
start)
/u01/HR/nodemanager/exl1cn01/startNodeManager.sh
;;
...
esac
Starting NM on Machine Startup

There are several techniques on both Linux and Solaris for running custom scripts during
machine startup. One method is to create a symbolic link to the custom start script within a
run-time control folder, such as /etc/rc.d/rc5.d/. The symbolic link’s name should start
with “S” to indicate a startup script, as well as a number to indicate the order in which it should
be run. For example, S99startNM.
Another method involves registering the custom start script as an OS service. On Solaris, this
can be accomplished by using the svccfg and svcadm utilities.

WebLogic Logs
Process Log Files

Node Manager <NM_HOME>/nodemanager.log
Server <DOMAIN>/servers/<SERVER>/logs/<SERVER>.log
<DOMAIN>/servers/<SERVER>/logs/<SERVER>.out
WebLogic Logs
Node Manager creates a log file located in NodeManagerHome/nodemanager.log by
default. This log file stores data about all of the domains administered by Node Manager. Log
output is appended to the current nodemanager.log. Log rotation is disabled by default, but
can be enabled by setting LogCount and LogLimit in nodemanager.properties.
Node Manager creates the server output log for a server instance in the server instance’s logs
directory, with the name <server>.out. If the debug property is enabled as a remote start
property for the server instance, or if the Node Manager debug property is enabled, Node
Manager will include additional debug information in the server output log information. You
cannot limit the size of these log files.
Each WebLogic Server maintains its own log file. These log files also support rotation and
filtering, similar to the Node Manager.

Practice 10-1
Start Servers with the Node Manager
• Configuring Node Manager properties
• Enrolling a domain with the Node Manager
• Starting admin and managed servers remotely
• Automatically restarting a failed server

Whole-Server Migration
The WebLogic whole-server migration infrastructure:

• Allows a server on a failed machine to be restarted on
another machine
• Requires a running Node Manager on each machine
participating in migration
• Requires servers to be members of a cluster and to be
assigned unique virtual IP addresses
• Supports both manual and automatic failover
Machine A Machine B Machine C
Whole-Server Migration
When a migratable server becomes unavailable for any reason (for example, if it hangs or
loses network connectivity, or if its host machine fails), migration is automatic. Upon failure, a
migratable server is automatically restarted on the same machine if possible. If the migratable
server cannot be restarted on the machine where it failed, it is migrated to another machine.
In addition, an administrator can manually initiate migration of a server instance.
Node Manager is used by the Administration Server or a stand-alone Node Manager client to
start and stop migratable servers and is invoked by the cluster master to shut down and
restart migratable servers, as necessary.
Server migration has the following additional requirements:
• There is no built-in mechanism for transferring files that a server depends on between
machines. Using a disk that is accessible from all machines is the preferred way to
ensure file availability. If you cannot share disks between servers, you must ensure that
the contents of domain_dir/bin are copied to each machine.
• Although migration works when servers are not time synchronized, time-synchronized
servers are recommended in a clustered environment.

Automatic Server Migration: Example
Cluster
1
Node Check Machine Status Node
Manager 3 Manager
Server 1 Server 2
Check
Lease Status
Cluster
Master 2 Leasing
Service
4 Add IP 5 Renew
Lease
Server 3
Start Server
Node Node
Server 2
Manager Manager
Automatic Server Migration: Example

1. The machine that hosts Server 2 fails.
2. Upon its next periodic review of the lease table, the cluster master detects that Server
2’s lease has expired.
3. The cluster master tries to contact the Node Manager on the backup machine to restart
Server 2, but fails because the entire machine is unreachable. Alternatively, if Server 2’s
lease had expired because it was hung, but its machine was still reachable, the cluster
master would use the Node Manager to restart Server 2 on the same machine.
4. The cluster master contacts the Node Manager on the backup machine, which is
configured as an available host for migratable servers in the cluster. The Node Manager
adds the IP address for Server 2 to the operating system and then starts Server 2.
5. Server 2 starts and contacts the Administration Server to obtain its configuration and
finally obtains a migratable server lease.

Leasing Types
WebLogic Server supports two leasing implementation options.
Type Description
Consensus • Servers renew leases by contacting the master directly.
(In-Memory)
• Leasing data is maintained in memory on the master and cached
on other servers for failover.
• This offers better performance.
Database Servers record lease information to a database by using a JDBC

data source.
Leasing Types
Using the database leasing implementation, lease information is maintained within a table in a
high-availability database. A high-availability database is required to ensure that leasing
information is always available. Each member of the cluster must be able to connect to the
database to access leasing information. This method of leasing is useful for customers who
already have a high-availability database within their clustered environment. This method also
enables you to use leasing functionality for JMS and Java Transaction API (JTA) service
migration features without also being required to use Node Manager. Database connectivity is
provided through a JDBC data source. Note, however, that XA data sources are not
supported for server migration.
In the nondatabase version of Consensus leasing, WebLogic Server maintains leasing
information in-memory. This removes the requirement of having a high-availability database to
use features that require leasing. One member of a cluster is chosen as the cluster leader and
is responsible for maintaining the leasing information. The cluster leader is chosen based on
the length of time that has passed since startup. The Managed Server that has been running
the longest within a cluster is chosen as the cluster leader. Other cluster members
communicate with this server to determine leasing information. However, the leasing table is
replicated to other nodes of the cluster to provide failover.

Migration Configuration Tasks
1. Create machine definitions for your domain.

2. Configure Node Manager network migration settings.
3. Configure the cluster leasing service.
4. Enable automatic server migration on all or a subset of
clustered servers.
5. Configure a list of candidate machines for server migration.
Most migration
settings require
server restart.
Migration Configuration Tasks

Node Manager must be running and configured to allow server migration. The Java version of
Node Manager can be used for server migration on Windows or UNIX. The SSH version of
Node Manager can be used for server migration on UNIX only. Refer to the Node Manager
Administrator's Guide in the WLS documentation for the available configuration and security
options.

Node Manager Network Properties
• Edit the Node Manager configuration and set:

– The name of each OS network interface and its range of IP
addresses
– The subnet masks to use for these addresses
• Node Manager delegates the tasks of adding and
removing addresses to the wlsifconfig script.
nodemanager.properties
...
bond0=10.0.0.1-10.0.0.17,NetMask=255.255.255.224
bond1=10.1.0.1-10.1.0.17,NetMask=255.255.255.224
UseMACBroadcast=true
Node Manager Network Properties

You must edit your nodemanager.properties file and add your OS environment’s network
interfaces and subnet masks. Any unique IP address configured for a server's network
channels will be sent to the Node Manager during migration. They must all be valid IP
addresses. To specify multiple interface names for the different addresses, the following
syntax can be used:
InterfaceName=StartIP-EndIP,NetMask=Mask
The IP address range will be used to determine which interface the IP address will be bound
to. The line can also include an optional subnet mask. If not specified, the default subnet
mask of the OS will be used. You can also include additional properties named Interface and
NetMask. These values will be used if a supplied IP address does not match one of the
previous expressions.
Node Manager uses another script, WL_HOME/common/bin/wlsifconfig.sh/cmd, to add
the address to the host machine before starting the server. The server must be able to run the
ifconfig utility, which is available only to privileged users. Configure sudo to grant access
to the user that is associated with the Node Manager.

Configuring Cluster Leasing
Leasing Existing
type data source
3 Existing
lease table
Configuring Cluster Leasing

Before configuring leasing, ensure that you have created machines and have assigned
managed servers to them. Also ensure that you configured and started your node managers
on each machine.
Shut down all of your clustered servers before modifying the cluster’s migration settings. Then
perform the following:
1. In the left pane of the console, expand Environment and select Clusters. Then select an
existing cluster.
2. Select Configuration > Migration.
3. Select a Migration Basis for the cluster leasing service. For Database leasing, also
configure the Data Source For Automatic Migration and Auto Migration Table Name
fields . Alternatively, click New to create a new data source. When finished, click Save.

Database Leasing Schema
Create schema by using

supplied DDL files.
Database Leasing Schema

1. Create the leasing table in the database. This is used to store the machine-server
associations used to enable server migration. The schema for this table is located in
WL_HOME/server/db/<dbname>/leasing.ddl, where dbname is the name of the
database vendor. By default, the database table name is active, but this can be modified
if desired.
2. Set up and configure a data source. This data source should point to the database
configured in the previous step.

Candidate Machines
Administrators can restrict which machines servers are able to

migrate to:
• At the cluster level
• For individual servers (overrides cluster setting)
1
2
Candidate Machines
For each server that will support automatic migration to another machine, perform the
following:
1. Select a server in the console.
3. Select the Automatic Server Migration Enabled check box and click Save. You must
restart the server for this setting to take effect.
To configure the default list of candidate machines for server migration for a cluster:
1. Select a cluster in the console. Then select Configuration > Migration.
2. Use the Candidate Machines field to limit the available machines for migration. You can
also change the order of preference.
To configure a list of candidate machines to which a specific server in the cluster can be
migrated:
3. Use the Candidate Machines field to limit the available machines for migration. You can
also change the order of preference.

Machine Failback
• When a failed machine becomes available again,

WebLogic Server does not automatically restore migrated
servers back to it.
• Instead, perform the following:
1. Gracefully shut down the migrated server by using Node
Manager.
2. Restart the failed machine and Node Manager.
Machine Failback
When a machine that previously hosted a server instance that was migrated becomes
available again, the reversal of the migration process—migrating the server instance back to
its original host machine—is known as failback. WebLogic Server does not automate the
process of failback. An administrator can accomplish failback by manually restoring the server
instance to its original host.
The general procedures for restoring a server to its original host are as follows:
1. Gracefully shut down the new instance of the server.
2. After you have restarted the failed machine, restart Node Manager and the Managed
Server.

Manual Server Migration
• Alternatively, administrators may use the console to

manually perform server migration:
– Gracefully bring down a machine for hardware maintenance.
– Perform OS or WebLogic Server maintenance.
– Move a migrated server back to its original machine.
• To help prevent others from making changes during the
migration, administrators must first obtain the console lock.
Manual Server Migration

If automatic server migration is not enabled on any of the servers in a cluster, administrators
can still use the console to manually initiate the migration of a server to another candidate
machine. However, even if automatic migration is enabled, manual migration may still be
desirable to help handle situations in which a server must be brought offline for maintenance.
In this scenario, the following occurs:
1. The administration server contacts the Node Manager on the source machine.
2. The Node Manager on the source machine shuts down the specified server.
3. As part of shutting down, the server removes its lease from the leasing service.
4. The Administration Server contacts the Node Manager on the target machine.
5. The Node Manager on the target machine starts the specified server.

Migrating a Server
1
2
Migrating a Server
To manually migrate a server to another machine, perform the following steps:
2. Click the Control tab.
3. Click the Migration second-level tab.
4. In the Migrate to Machine field, select a machine from the list of available candidates.
Then click Save.

Pinned Server Resources
• Applications and session data can be replicated across

servers in a cluster for failover.
• Migration is especially important for other types of
resources that can run on only one server at a time:
– JMS persistent stores
– Transaction logs
– Custom services
• The admin server:
– Is not part of a cluster and cannot participate in automatic
migration
– Can be started on another machine manually if its current
host fails
Pinned Server Resources

In a WebLogic Server cluster, most subsystem services are hosted homogeneously on all
server instances in the cluster, enabling transparent failover from one server to another. In
contrast, pinned services, such as JMS related services and the WebLogic transaction
recovery service, are hosted on individual server instances within a cluster. For these
services, the WebLogic Server migration framework supports failure recovery with service
migration, as opposed to failover. WebLogic Server also supports whole server-level
migration, where a migratable server instance and all of its services are migrated to a different
physical machine upon failure.
WebLogic Server also provides a framework for migrating custom singleton services as well.
You must create a Java class that implements a specific interface and register it at the
domain or application level. When a custom migratable service fails or becomes unavailable
for any reason (for example, because of a bug in the service code, server failure, or network
failure), it is deactivated at its current location and activated on a new server. The process of
migrating these services to another server is handled by using the singleton master. The
singleton master is a lightweight singleton service that monitors other services that can be
migrated automatically. The server that currently hosts the singleton master is responsible for
starting and stopping the migration tasks associated with each migratable service.

Recommended Storage Configuration for JMS
and Transaction Recovery
exalogic
Separate project for

HR
each business unit
Separate folder for each recovery

domain and cluster
MyDomain
jms
Pool
Cluster1
Project tlogs
Share

Configuring a Server's Transaction Store
By default, transaction logs are placed at

<domain>/servers/<server>/data/store/default.
Configuring a Server’s Transaction Store

Each server has a transaction log, which stores information about committed transactions
coordinated by the server that may not have been completed. WebLogic Server uses the
transaction log when recovering from system crashes or network failures. You cannot directly
view the transaction log. The records are in a binary format and are stored in the default
persistent store for the server.
To take advantage of the migration capability of the Transaction Recovery Service for servers
in a cluster, you must store the transaction log in a location that is available to a server and its
backup servers:
1. Edit a server.
2. Click the Configuration > Services tab.
3. Within the Default Store section, edit the Directory field. Provide the path to which this
server should store transaction logs and should later try to recover from in the event of a
failure.

Creating a JMS Persistent Store
Creating a File Store

When configuring a file store directory, the directory must be accessible to the server instance
on which the file store is located. Perform the following steps:
1. In the left pane of the console, expand Services and select Persistent Stores.
2. On the Summary of Persistent Stores page, select New > Create FileStore.
3. Update the following:
- Name: Name of the store
- Target: Server instance on which to deploy the store
- Directory: Path name to the directory on the file system where the file store is
placed. This directory must exist on your system, so be sure to create it before
completing this tab. For highest availability, use either a Storage Area Network
(SAN) or a dual-ported SCSI disk.
If you edit an existing file store, you can also modify its Synchronous Write Policy field, which
defines to what extent a file store will try to flush records to the disk. The available values are
Direct-Write (default), Cache-Flush, and Disabled.

Quiz
Which of the following is not a valid Node Manager property?

a. ListenAddress
b. StartScriptName
c. StartScriptEnabled
d. StoreDirectory
e. NodeManagerHome
Answer: d

Quiz
Who in a cluster is responsible for initiating an automatic whole

server migration?
a. Node Manager
b. Cluster master
c. Recovery log
d. Persistent store
Answer: b

Quiz
Name two types of migration leasing supported by WLS.

a. Transaction
b. Consensus
c. Database
d. Enrollment
Answer: b, c

Summary

• List the capabilities of Node Manager
• Configure a node manager for a compute node
• Start and kill servers by using Node Manager
• Explain the process of server migration
• Use floating IP addresses with Node Manager
• Configure persistent storage for server recovery

Practice 10-2
Migrate Failed Servers
• Configuring a node manager to migrate IP addresses
• Configuring candidate machines for servers in a cluster
• Confirming server migration upon machine failure

Exalogic Database Connectivity

Objectives

• Explain basic Oracle RAC concepts
• Compare Exalogic connectivity with Exadata and other
hardware
• Describe the capabilities of a GridLink data source
• Connect WLS to a DB cluster by using GridLink

Ethernet Database Connectivity
• Generally speaking, Exalogic is designed to handle high

processing requirements and not the massive storage
needs of an enterprise database.
• Applications running on Exalogic can access external
database systems by using the InfiniBand gateway
switches (bond1).
Exalogic
Compute Node
Gateway
InfiniBand 10 Gb Ethernet
Application DB
Application
Ethernet Database Connectivity

You can connect a machine running an Oracle Real Application Clusters (RAC) database to
Exalogic Machine by using a 10 Gb Ethernet link. That is, the database is accessed by
software on Exalogic just as if it accessed any other external machine on your intranet. If the
Ethernet network hosting your database is configured as a virtual LAN (VLAN), you must also
configure the Exalogic gateway switch port with the same VLAN ID to permit traffic between
the two networks.

Oracle Real Application Clusters (RAC): Overview
• Oracle RAC:
– Supports multiple Oracle database servers for greater
scalability
– Relies on database servers having access to a shared and
highly available storage device
• The Oracle driver provides basic load balancing and
failover across RAC instances.
RAC Node 1
Application Driver Shared
RAC Node 2 Storage
Oracle Real Application Clusters (RAC): Overview

Oracle Real Application Clusters (RAC) is a software component that you can add to a high-
availability solution that enables users on multiple machines to access a single database with
increased performance. RAC comprises two or more Oracle Database instances running on
two or more clustered machines and accessing a shared storage device via cluster
technology. To support this architecture, the machines that host the database instances are
linked by a high-speed interconnect to form the cluster. The interconnect is a physical network
used as a means of communication between the nodes of the cluster. Cluster functionality is
provided by the operating system or by a compatible third-party clustering software.

Oracle Exadata: Overview
The Oracle Exadata solution:

• Combines Sun hardware and optimized Oracle software,
similar to Exalogic
• Is engineered to run very large RAC deployments
• Utilizes an InfiniBand fabric to connect nodes or “cells”
• Dedicates separate hardware to computational-intensive
processing and data-intensive processing
• Helps standardize and consolidate your data warehouse
and transaction processing
Oracle Exadata: Overview

The Oracle Exadata Database Machine is an easy-to-deploy solution for hosting Oracle
Database that delivers the highest levels of database performance. The Exadata Database
Machine is composed of database servers, Oracle Exadata Storage Servers, and an
InfiniBand fabric for networking all the components. It delivers outstanding I/O and SQL
processing performance for online transaction processing (OLTP) and data warehousing.
There are two versions of the Exadata Database Machine. The Exadata Database Machine
X2-2 expands from two 12-core database servers with 192 GB of memory and three Exadata
Storage Servers to eight 12-core database servers with 768 GB of memory and 14 Exadata
Storage Servers, all in a single rack. The Exadata Database Machine X2-8 comprises two 64-
core database servers with 2 TB of memory and 14 Exadata Storage Servers in a single rack.
Exadata Smart Flash Cache dramatically accelerates Oracle Database processing by
speeding I/O operations. The Flash provides intelligent caching of database objects to avoid
physical I/O operations. The Oracle database on the Database Machine is the first Flash-
enabled database. Exadata storage utilizes an advanced compression technology, Exadata
Hybrid Columnar Compression, that typically provides 10 times and higher levels of data
compression than a traditional database server.

Direct InfiniBand Database Connectivity
• Exalogic and Exadata racks are designed to be easily

interconnected by using InfiniBand cables.
• You can connect up to eight full Exalogic and/or Exadata
racks without additional InfiniBand switch hardware.
Exalogic Exadata
Compute Node
InfiniBand
Application DB
Application
Direct InfiniBand Database Connectivity

It is possible to connect as many as eight full racks of Exalogic hardware (or any combination
of Exalogic and Exadata configurations) without the need for any external switches. In cases
where more than eight racks of Exalogic or Exadata hardware are required, Oracle offers a
choice of several high-capacity data center switches that enable the creation of Exalogic
clouds comprising hundreds of racks and tens of thousands of processors.

Exalogic Quarter Rack and Exadata Quarter Rack
Exadata DB Servers
2 1-2 2
Cells 1-3
3 3
7
NM2-36 #1 NM2-36 #2
2 2
2 2
NM2-GW #1 NM2-GW #2
6
2 Storage 2
Heads 1-2
8 Compute 8
Exalogic Nodes 1-8
Exalogic Quarter Rack and Exadata Quarter Rack

You require only eight cables to connect an Exalogic machine quarter rack and an Oracle
Exadata Database Machine together. If these systems are side by side, you can use the 5-
meter InfiniBand cables included in the spares kit to complete these connections.
In this multi-rack configuration scenario, all connections within both racks will remain as they
are when delivered to the customer site. No connections will be moved or removed to make
the cross connections. In addition, a dedicated spine switch in either rack is not required. All
cross connections between the racks are made using four ports on each of the existing
switches.
Refer to the Exalogic Machine Multirack Cabling Guide for detailed instructions.

Exalogic Full Rack and Exadata Full Rack
12 unused
IB ports
Exadata
NM2-36 2
4
NM2-GW NM2-GW NM2-36 NM2-36

7
6 6
NM2-GW NM2-GW 2 2
12 unused NM2-36
Exalogic IB ports
Exalogic Full Rack and Exadata Full Rack

Each Sun Network QDR InfiniBand Gateway Switch (NM2-GW) in the Exalogic machine full
rack is connected to the local spine switch (NM2-36P). Subsequently, each leaf switch in the
Exalogic machine full rack is connected to the spine switch of the Oracle Exadata Database
Machine.
3-meter cables should be used to connect leaf switches in the Oracle Exadata Database
Machine to the spine switch (NM2-36P) in the Oracle Exadata Database Machine full rack.
The 3-meter cables are bundled with the Oracle Exadata Database Machine.
5-meter cables should be used to connect leaf switches of Oracle Exadata Database Machine
to the spine switch (NM2-36P) of the Exalogic machine full rack. The 5-meter cables are
bundled with the Oracle Exadata Database Machine.

JDBC: Overview
• The Java Database Connectivity (JDBC) specification:

– Is a platform-independent and vendor-independent
mechanism for accessing and updating a database
– Provides transparency from proprietary vendor issues
– Requires the use of a driver
• JDBC drivers are supplied by WebLogic Server or by your
database vendor.
Get connection JDBC

Application DB
Perform SQL driver
JDBC: Overview
The JDBC API is a natural Java interface for working with SQL. It builds on Open Database
Connectivity (ODBC) rather than starting from the beginning, so programmers familiar with
ODBC find it very easy to learn.
The value of JDBC lies in the fact that an application can access virtually any data source and
run on any platform with a Java Virtual Machine (JVM). That is, with JDBC, you do not have to
write one program to access a Sybase database, another to access an Oracle database,
another to access an IBM DB2 database, and so on. You can write a single program by using
the JDBC API. Because the application is written in Java, you do not need to write different
applications to run on different platforms, such as Windows and Linux.
JDBC accomplishes database connections by using a driver mechanism that translates the
JDBC calls to native database calls. Although most available drivers are fully written in Java
(Type 4) and are thus platform independent, some drivers (Type 2) use native libraries and
are targeted to specific platforms. Oracle WebLogic Server includes several Type 4 JDBC
drivers, which are compliant with the JDBC 3.0 specification.

WebLogic Data Source: Overview
Data sources:
• Enable database connectivity to be managed by the
application server
• Use a dynamic pool of reusable database connections
Server Data Source
App Pool
Lookup DS Connection
Get connection Connection
App DB
Perform SQL Connection
Return conn. Connection
App
JDBC Driver
WebLogic Data Source: Overview

Oracle WebLogic Server can manage your database connectivity through JDBC data sources
and multidata sources. Each data source that you configure contains a pool of database
connections that are created when the data source instance is created—when it is deployed
or targeted, or at server startup. The connection pool can grow or shrink dynamically to
accommodate the demand, as indicated by the dotted connection at the top of the pool.
Applications look up a data source on the Java Naming and Directory Interface (JNDI) tree or
in the local application context (java:comp/env), depending on how you configure and deploy
the object, and then request a database connection. When finished with the connection, the
application uses the close operation on the connection, which simply returns the connection
to the connection pool in the data source.
Oracle WebLogic Server data sources allow connection information such as the JDBC driver,
the database location (URL), and the username and password to be managed and
maintained in a single location, without requiring the application to tend to these details. In
addition, limiting the number of connections is important if you have a licensing limitation on
your database or it can support only a specific capacity.

Active GridLink for RAC
WebLogic provides a GridLink JDBC data source that:

• Performs intelligent load balancing based on the current
RAC workload
• Implements RAC’s Fast Connection Failover (FCF) pattern
• Supports SDP when RAC is hosted on Exadata
• Ensures that all parts of a transaction are routed to the
same node (“affinity”)
WebLogic Server
RAC Node
Application
TCP or
GridLink Data Source RAC Node
SDP
Application
Driver Connections
RAC Node
Active GridLink for RAC

A single GridLink data source provides connectivity between WebLogic Server and an Oracle
database service that has been targeted to an Oracle RAC cluster. This type of data source
automatically adjusts the distribution of work based on the current performance metrics
reported by each RAC node, such as CPU usage, availability, and response time. If this
capability is disabled, GridLink data sources instead use a round-robin, load-balancing
algorithm to allocate connections to RAC nodes.
A GridLink data source implements Oracle's Fast Connection Failover (FCF) pattern, which:
• Provides rapid failure detection
• Aborts and removes invalid connections from the connection pool
• Performs graceful shutdown for planned and unplanned Oracle RAC node outages
• Adapts to changes in topology, such as adding or removing a node
• Distributes run-time work requests to all active Oracle RAC instances, including those
rejoining a cluster
XA affinity ensures all the database operations performed on an RAC cluster within a global
transaction are directed to the same RAC instance. This increases performance and also
helps ensure data integrity after a failure.

GridLink and FCF
• Typically when a system goes down, applications must

wait for the network to time out (minutes).
• The Oracle Notification Service (ONS) delivers events
about RAC availability/workload to registered subscribers.
• The GridLink data source can subscribe to ONS and
immediately respond to nodes that are added or removed.
RAC Node
WebLogic Server
ONS
GridLink Data Source
FAN events
ONS Client RAC Node
ONS
GridLink and FCF

When a database outage occurs, two problems confront applications: errors and hangs.
Applications encounter errors because the work they were doing (queries, transactions) is
interrupted. Even worse, those errors may take some time to arrive. When the database
outage occurs, the client application may have to wait for network timeouts (which may be
minutes) before being informed of the outage. This can cause the application to hang for
some time, leading to user frustration. Oracle Database provides several features that Java
applications such as WebLogic Server can use to increase failure responsiveness and to help
mask errors from the end user.
The Oracle Notification Service (ONS) uses a simple publish/subscribe method to produce
and deliver event messages for both local and remote consumption. ONS daemon processes
are automatically created during the RAC installation process and are configured to run on
each node.
A GridLink data source uses ONS to adaptively respond to state changes in an Oracle RAC
instance. This ensures that the connection pool in the data source contains valid connections
(including reserved connections) without the need to poll and test connections. The data
source also distributes connections to Oracle RAC instances based on these Fast Application
Notification (FAN) events.

GridLink and Services
• Oracle Database supports services that:

– Act as gateways to a subset of RAC nodes
– Automatically start on another node if the current one fails
– Are accessed by clients using the service name
– Enable you to control and prioritize the available capacity on
your grid for different clients
• Create a separate GridLink data source for each service.
A RAC Node
Application GridLink DS
Service A A
RAC Node
B
Application GridLink DS
Service B B RAC Node
GridLink and Services

Services are an entity defined for a RAC database that enables its workload to be managed.
Services also decouple any hardwired mapping between a connection request and a RAC
instance. Each service represents a workload with common attributes, thresholds, and
priorities. For example, online users can be a service whereas batch processing can be
another and reporting can be yet another service type. A service can span one or more
database instances and a single instance can support multiple services. The use of FAN
events and run-time connection load balancing requires the configuration of services.
Services hide the complexity of a cluster from the database client by providing a single logical
entity for managing work. Applications or middleware such as WebLogic Server specify a
service by supplying its name when making the initial connection. On WebLogic Server in
particular, you scale the number of GridLink data sources as the number of services
increases in the database, independent of the number of nodes in the cluster.

GridLink and Single Client Access Name (SCAN)
• Oracle Database 11gR2 provides a SCAN service that:

– Accepts a database cluster alias
– Returns the locations of cluster members
– Can run independently or integrate with your corporate DNS
• GridLink data sources can either use a list of database
node locations or a single SCAN address.
RAC Node GridLink DS
GridLink DS RAC Node OR RAC Node
SCAN RAC Node

RAC Node
RAC Node
GridLink and Single Client Access Name (SCAN)

Single Client Access Name (SCAN) is a new RAC 11g Release 2 feature that provides a
single name for clients to access databases running in a cluster. The benefit is that the client’s
connection information does not have to change if you add or remove nodes in the cluster.
Having a single name to access the cluster allows clients to access any database running in
the cluster, independently of which server or servers in the cluster the database are active.
SCAN provides load balancing and failover for client connections to the database. The SCAN
works as a cluster alias for databases in the cluster.
During the interview phase of the Oracle Grid Infrastructure installation, you are prompted to
provide a SCAN name. There are two options for defining the SCAN:
• Use an existing Domain Name Service (DNS) implementation
• Use the Oracle Grid Naming Service (GNS)
If you choose the DNS option, you must ask your network administrator to create a single
name that resolves to three IP addresses by using a round-robin algorithm. Three IP
addresses are recommended for the SCAN service for high-availability reasons, regardless of
the number of servers in the cluster. The IP addresses must be on the same subnet as your
public network in the cluster.

Creating a GridLink Data Source
Supports a list or a
Initial list of nodes to single SCAN address
access service from
Initial list of ONS

processes to subscribe to
Creating a GridLink Data Source

1. From the WLS console, select Services > JDBC > Data Sources. Then click New >
GridLink Data Source.
2. Supply a Name and JNDI Name for the data source, and click Next. Supply the Service
Name, User Name, and Password. Use the Add button to specify either a list of initial
database listeners to connect to or a Single Client Access Name (SCAN) address. A
GridLink data source that contains SCAN addresses does not need to change if you
add or remove Oracle RAC nodes. Click Next.
3. (Optional) Test the supplied addresses and click Next. Confirm that FAN Enabled is
selected if you want WebLogic to subscribe to events. Use the Add button to specify
either a list of initial ONS daemon processes to connect to or a SCAN address. You can
also optionally configure WebLogic to receive events over SSL. Complete the remaining
steps.

Enabling SDP for Exadata Connectivity
Edit the default data source configuration and change all

occurrences of “TCP” to “SDP” in the URL.
Enabling SDP for Exadata Connectivity

Oracle Net Services provides support for the Sockets Direct Protocol (SDP) for InfiniBand
high-speed networks. For example, InfiniBand can be used to connect an Exalogic machine
to an Exadata machine. SDP is characterized by short-distance, high-performance
communications between multiple server systems.
Simply connecting the machines with InfiniBand cables is not sufficient—WebLogic and RAC
will still communicate by using TCP (IPoIB). First, configure an SDP address in the
listener.ora file on the database server. Then edit your GridLink data source by using
the WLS console. On the Configuration > Connection Pool tab, locate the URL field. Replace
all instances of the text PROTOCOL=TCP with PROTOCOL=SDP. Finally, use the Control tab to
restart the data source.
Note that in order to use SDP, you will also need to add a command-line argument when
starting your WebLogic Servers: -Djava.net.preferIPv4Stack=true. For example, edit
startWebLogic.sh.

Multi Data Source: Overview
• WebLogic also provides a high-availability solution for non-

Oracle database clusters.
• A multi data source:
– Is a pool of data sources
– Is used by applications exactly like a standard data source
– Transparently provides load balancing or failover across the
member data sources
Multi Data Source: Overview

A multidata source is an abstraction of a group of data sources that provides load balancing
or failover processing between the data sources associated with the multidata source.
Multidata sources are bound to the JNDI tree or local application context just like data sources
are bound to the JNDI tree. Applications look up a multidata source on the JNDI tree just like
they do for data sources, and then request a database connection. The multidata source
determines which data source to use to satisfy the request depending on the algorithm
selected in the multidata source configuration: load balancing or failover.
All data sources used by a multidata source to satisfy connection requests must be deployed
on the same servers and clusters as the multidata source. A multidata source always uses a
data source deployed on the same server to satisfy connection requests. Multidata sources
do not route connection requests to other servers in a cluster or in a domain. To deploy a
multidata source to a cluster or server, you select the server or cluster as a deployment target.
When a multidata source is deployed on a server, WebLogic Server creates an instance of
the multidata source on the server. When you deploy a multidata source to a cluster,
WebLogic Server creates an instance of the multidata source on each server in the cluster.

Multi Data Source: Architecture
Multi Data Source
Data Source A
1. DS lookup. Connection
2. Get connection. Connection
App
3. Perform SQL. Connection
4. Return connection.
App JDBC Driver
Synchronize
App Data Source B
Data Source C
Multi Data Source: Architecture

A multidata source can be thought of as a pool of data sources. Multidata sources are best
used for failover or load balancing between nodes of a highly available database system,
such as redundant databases or Oracle Real Application Clusters (RAC). Multidata sources
do not provide any synchronization between databases. It is assumed that database
synchronization is handled properly outside of WebLogic Server so that data integrity is
maintained.
You create a multidata source by first creating data sources, then creating the multidata
source by using the Administration Console or the WebLogic Scripting Tool (WLST). Finally,
you assign the data sources to the multidata source.
The data source member list for a multidata source supports dynamic updates. You can
remove a database node and corresponding data sources without redeployment. This
capability provides you the ability to shut down a node for maintenance or shrink a cluster.

Licensing Review
• Standard WebLogic licenses permit the use of the Multi

Data Source solution (no ONS or FCF).
• A license for Exalogic Elastic Cloud Software also permits
the use of:
– The Active GridLink for RAC feature (GridLink data source)
– The SDP protocol for data source connectivity
Licensing Review
Active GridLink for RAC is not a binary distribution like WebLogic Server. It is simply the name
given to the use of WebLogic Server's GridLink Data Source feature that includes Fast
Connection Failover and FAN events.

Quiz
GridLink data sources support SDP when Exalogic is

connected directly to ____.
a. ONS
b. Exadata
c. Cyberdata
d. WebLogic
Answer: b

Quiz
Which Oracle RAC feature do GridLink data sources support?

a. Fast Connection Failover
b. Data Harvester
c. Role Mapper
d. Enterprise Message Queue
Answer: a

Summary

• Explain basic Oracle RAC concepts
• Compare Exalogic connectivity with Exadata and other
hardware
• Describe the capabilities of a GridLink data source
• Connect WLS to a DB cluster by using GridLink

Recorded Demonstration:
Active GridLink
This demonstration covers the following tasks:
• Creating a RAC service
• Creating a WLS GridLink data source
• Testing the data source and confirming load distribution
• Stopping a database node and confirming data source
failover

Additional FMW Infrastructure

Objectives

• Describe the features of Coherence
• Configure a Coherence server and add it to a domain
• Describe the features of Web Tier
• Configure Web Tier to proxy to a cluster
• Start Coherence and Web Tier processes

Coherence: Overview
Oracle Coherence:
• Is a JVM process
• Provides a distributed, in-memory data caching solution
• Offers very high performance and scalability
• Is based on a cluster of cache servers
• Automatically distributes or partitions cached data across
the cluster
• Can be installed and managed independently or as part of
a WebLogic domain
Coherence Cluster
Applications
Coherence: Overview
One of the primary uses of Oracle Coherence is to cluster an application’s objects and data.
In the simplest sense, this means that all the objects and data that an application delegates to
Coherence clusters are automatically available to and accessible by all servers in the
application cluster. None of the objects or data will be lost in the event of server failure. By
clustering the application’s objects and data, Coherence solves many of the difficult problems
related to achieving availability, reliability, scalability, performance, serviceability, and
manageability of clustered applications.
The partitioning feature dynamically load-balances data evenly across the entire server
cluster, whereas replication ensures that a desired set of data is always available and up-to-
date at all times in the cluster. Replication enables operations that are running on any server
to obtain the data that they need locally, at basically no cost, because that data has already
been replicated to that server. The only downside of partitioning is that it introduces latency
for data access, and in most applications, the data access rate far outweighs the data
modification rate. To eliminate the latency associated with partitioned data access,
Coherence can use local or “near caching" as well. Frequently and recently used data from
the partitioned cache is maintained on the specific servers that are accessing that data, and
this near data is kept up-to-date by using event-based invalidation.

Coherence: Architecture
Domain (optional) Admin Server

(optional)
Automatic load
balancing and failover
Machine Machine
Update data Coherence Coherence

Application Server Server
Coherence Coherence
Application Server Server
Get data
Coherence Cluster
Cached
data
Node Manager Node Manager

(optional) (optional)
Coherence: Architecture
With Coherence, there is neither master nor slave, but rather all members work together as a
team to manage the cluster as a whole, allowing cluster members to join and leave without
impacting the remainder of the cluster or client applications.
Due to Coherence cluster heartbeat communication, all members are known and it is possible
to provide redundancy within the cluster, such that the death of any one JVM or server
machine does not cause any data to be lost. Also, it is possible to load balance
responsibilities across the cluster. Load balancing automatically occurs to respond to new
members joining the cluster, or existing members leaving the cluster. The death or departure
of a cluster member is automatically and quickly detected, so failover occurs very rapidly, and
more importantly, it occurs transparently, which means that the application does not have to
do any extra work to handle failover.
Coherence servers and clusters can be configured, administered, and started manually
through XML configuration files and custom scripts. Alternatively, Coherence servers and
clusters can be configured as members of a WebLogic domain. The latter approach gives you
the option of modifying basic Coherence settings from the WebLogic console or WLST. You
can also take advantage of the WebLogic Node Manager to remotely start and stop
Coherence servers and to automatically monitor their health.

Role of Coherence Applications
• Applications themselves:
– Are members of the Coherence cluster and automatically
discover other members
– Can participate in data partitioning (“storage-enabled”)
Local storage
disabled
Coherence Coherence
Server Server
Application
Application
Coherence Coherence
Server Server
Coherence Cluster
Role of Coherence Applications

Coherence is a represented at the highest level by a cluster. A cluster is a collection of
Coherence elements known as members. Within a cluster, all members share common
information such as cluster name. Each member may contain a set of caches. Members
typically replicate, partition, or otherwise contain data and backup data, all seamlessly shared
across the cluster.
There is no configuration required to add members to a Coherence cluster. Subject to
configurable security restrictions, any JVM running Coherence will automatically join the
cluster and be able to access the caches and other services provided by the cluster. This
includes Java EE application servers, Coherence cache servers, dedicated cache loader
processes, or any other JVM that is running with the Coherence software. When a JVM joins
the cluster, it is called a cluster node, or alternatively, a cluster member.
Coherence can be used by applications according to a simple client/server model. Such
clients are typically storage- disabled. They do not participate in data management, nor affect
partitioning and rebalancing. Storage-enabled clients, on the other hand, are full members of
a cluster, and locally store and manage cached data just like Coherence servers.

Basic Coherence Configuration Tasks
1. Use the WLS console or WLST to define Coherence

servers and clusters in a domain:
a) Assign bond0 (private) addresses if all applications are in
the same data center.
b) Assign ports to each server and to an entire cluster.
c) If you are using Node Manager to start Coherence,
configure server startup arguments if necessary.
2. Configure additional settings for a Coherence cluster by
editing the generated XML file.
3. Create a cache configuration XML file if the default is not
sufficient.
4. Start Coherence servers by using scripts or Node
Manager.
Basic Coherence Configuration Tasks

Oracle Coherence is installed by default as part of the WebLogic Server installation.
Coherence servers running on Exalogic typically bind to the private InfiniBand network,
particularly in cases in which all client applications (such as WebLogic Server applications)
are running within the same data center.
To help minimize packet loss, the operating system socket buffers need to be large enough to
handle the incoming network traffic while the Java application is paused during garbage
collection. By default, Coherence will attempt to allocate a socket buffer of 2 MB. If your
operating system is not configured to allow for large buffers, Coherence will use smaller
buffers. For example, on Oracle Linux run:
sysctl -w net.core.rmem_max=4192608
sysctl -w net.core.wmem_max=4192608

Creating a Coherence Cluster
1
4
2
Creating a Coherence Cluster

1. From the Domain Structure panel, click Coherence Clusters.
2. Click the New button.
3. Enter a cluster name and click Next. By default, WebLogic will generate a new
configuration XML file for this cluster. If you have created a custom template
configuration file that should be used instead, use the Use a Custom Cluster
Configuration File and File Path fields.
4. Enter the addresses and ports that cluster members should use by default for unicast
and multicast communication. Unlike WebLogic clusters, Coherence clusters use both
multicast and unicast messaging by default. The Unicast Port Auto Adjust field specifies
whether the unicast port should be automatically incremented if Coherence detects that
the port is already in use by other cluster members or other non-Coherence processes.
5. Optionally, target the Coherence cluster to WebLogic servers or clusters. This action will
make this Coherence cluster accessible to applications that are deployed to these
targets. Click Finish.

Creating a Coherence Server
Overrides cluster
defaults

1. From the Domain Structure panel, click Coherence Servers.
2. Click the New button.
3. Enter a name for the Coherence server and optionally assign it to a cluster. In order to
use Node Manager to start this Coherence server, you must assign the server to a
machine and configure the machine for the Node Manager. The remaining Unicast fields
can be used to override the same settings configured in the Coherence cluster. Click
Finish.

2 3
Override the default start

path or Java classpath
used by Node Manager.
Specify additional
JVM arguments.
Creating a Coherence Server (continued)

When you use Node Manager to start WebLogic Servers remotely, you can configure Node
Manager to utilize a start script (recommended) or you can provide the specific JVM
arguments that Node Manager should execute. For Coherence servers, Node Manager does
not have the ability to run a start script. Instead, it launches a JVM process to start Coherence
and includes some default arguments. However, you can override the startup settings that
Node Manager will use to start a Coherence server on a remote machine.
1. Edit an existing Coherence server.
2. Click the Configuration > Server Start tab.
3. Edit these fields and click Save:
- Root Directory: The directory from which to start this Coherence process. The
domain directory is used by default.
- Class Path: By default, the JVM class path is set to coherence.jar. If you
override the class path, be sure to still include the default entries.
- Arguments: Other JVM arguments to include when starting this Coherence
process. Refer to the documentation for a complete list of Coherence command-
line arguments.

Starting Coherence Servers
3
1
Start by using
2 Node Manager.
Start from the command line:
cd <COHERENCE_HOME>
./bin/cache-server.sh
Starting Coherence Servers

Start and shut down a Coherence server by using the same techniques that you use to start
and shut down WebLogic Servers. Both the Administration Console and WLST are supported.
Also, similar to WebLogic Servers, Coherence servers can be started from the command line
by using scripts. A sample Coherence start script is included with the product installation.
Note that the JAVA_HOME environment variable should be set prior to running the Coherence
start script. Also note that on Linux distributions the script should be run from the root of the
install rather than from the /bin directory.

Coherence*Web: Overview
• WLS can offload its application session (in-memory) data

to a Coherence cluster to reduce its footprint.
• A central cache infrastructure can be more flexible,
scalable, and tunable.
Coherence Cluster
WebLogic Cluster
Cache Node Cached HTTP
sessions
Cache Node
WebLogic Cluster
Cache Node
Servers with the

Coherence plug-in
Coherence*Web: Overview
Coherence*Web is an application server plug-in dedicated to managing session state in
clustered environments. It brings the scalability, availability, reliability, and performance
characteristics of a Coherence data grid to in-memory session management and storage.
Also, because it is not constrained by the deployment topologies of the application server, it
enables session sharing and management across different web applications, domains, and
even different application server products. Sometimes you may want to explicitly prevent
session data from being shared by different Java EE applications that participate in the same
Coherence cluster, so Coherence*Web supports this approach as well.
The Coherence*Web SPI for WebLogic Server is configured with local-storage disabled. This
means a Coherence cache server must be running in its own JVM, separate from the JVM
running WebLogic Server.
With Coherence*Web, session data is stored outside of the application server, thereby freeing
server heap space. This architecture also enables you to individually tune and scale the sizes
of your application server clusters and session data grids.

WLS Cluster: Review
A WLS cluster supports additional features:

• To provide high availability for applications and services
• To perform load balancing and failover
• That are transparent to both applications and clients
Domain
Hardware or
software Cluster
Server 1
HTTP
Web client Proxy
Server 2
EJB client
JMS client Server n
WLS Cluster: Review

one instance, whether the client is a web client or a Java application. By replicating the
WebLogic Server provides clustering support for Web applications by replicating the HTTP
session state of clients. You can balance the web application load across a cluster by using a
WebLogic Server proxy plug-in or the external load-balancing hardware.
Failover for Enterprise JavaBeans (EJB) objects is accomplished by using the object’s
replica-aware stub. When a client makes a call through a replica-aware stub to a service that
fails, the stub detects the failure and retries the call on another replica.
WLS also supports clustering for Java Message Service (JMS) applications. Applications that
use a distributed destination are more highly available than applications that use stand-alone
destinations because WebLogic JMS provides load balancing and failover for the members of
a distributed destination in a cluster.

Proxy Plug-Ins
WebLogic plug-ins are available to provide intelligent load

distribution and failover for a cluster.
Hardware Load
Balancer
Firewall
Web Servers and Web Servers and
WLS Plug-in WLS Plug-in
Firewall
Web applications Web applications

in data center A in data center B
Proxy Plug-Ins
An organization's DMZ typically includes an external load balancer or router to handle
external requests from the Internet or your Intranet. The load balancer then forwards the
requests to a web server such as Oracle WebTier or Apache. WebLogic is responsible for
handling the application requests that require dynamic Java EE functionality. The web server
then hosts static content and also, in the case of a cluster, performs load balancing and
failover.
The plug-in does a simple round-robin between all available servers in a cluster. The server
list specified in this property is a starting point for the dynamic server list that the server and
plug-in maintain. WebLogic Server and the plug-in work together to update the server list
automatically with new, failed, and recovered cluster members.
The plug-in also supports “sticky” session behavior, similar to hardware load balancers. This
means that the plug-in directs HTTP requests that contain a session cookie (or URL-encoded
session ID or a session ID stored in the POST data) to the same server in the cluster that
originally created the cookie. That is, the client is pinned to a specific server in the cluster.
This behavior ensures that users do not lose their session data between requests.

Recommended Proxy Topology
• Oracle recommends that proxy software run on its own

dedicated hardware and not Exalogic.
• Most types of proxy software are not optimized for
InfiniBand and SDP.
Exalogic
Server Compute Node

Servers
Proxy
Compute Node
Servers

Obtaining and Using Plug-Ins
• Plug-ins are:
– Bundled with WLS and Oracle HTTP Server (OHS)
– Also available online to download separately
– Released on a separate cycle than that for the server itself,
so always confirm that a newer version is not available
• Although all plug-ins share the same major capabilities, the
installation and configuration steps vary slightly by vendor.
Obtaining and Using Plug-Ins

The WebLogic Sun Java System Web Server plug-in module is distributed as a shared object
(.so) on UNIX platforms and as a dynamic-link library (.dll) on Windows. These files are
located in the WL_HOME/server/plugin/OperatingSystem/Architecture directory
of your WebLogic Server distribution. WL_HOME represents the top-level installation directory
for your WebLogic platform. The server directory contains installation files for WebLogic
Server. OperatingSystem refers to the operating system, such as UNIX or Windows. Add
the following lines to the beginning of the magnus.conf file. These lines instruct Sun Java
System Web Server to load the native library (the .so or .dll file) as a module:
Init fn="load-modules" funcs="wl_proxy,wl_init"\
shlib=/usr/local/netscape/plugins/(.so or .dll file) Init
fn="wl_init"
To install the Apache HTTP Server Plug-In as a dynamic shared object, locate the shared
object directory for your specific platform, such as linux/i686, solaris/sparc, or
win/32. Then copy the mod_wl_20.so file to the APACHE_HOME/modules directory and
add the following line to your APACHE_HOME/conf/httpd.conf file:
LoadModule weblogic_module modules/mod_wl_20.so

Oracle Web Tier: Overview
Product Description
Oracle iPlanet Web Server Supports the WLS plug-in but must be
installed manually
Oracle HTTP Server • Based on Apache (httpd.conf)
• WLS plug-in is preinstalled (mod_wl_ohs).
Oracle Web Cache An additional proxy that caches full or partial
web pages
Web Tier: Overview

Oracle HTTP Server is based on the Apache infrastructure, and includes modules developed
specifically by Oracle. The features of single sign-on, clustered deployment, and high
availability enhance the operation of the Oracle HTTP Server. Oracle HTTP Server can also
be a proxy server, both forward and reverse. A reverse proxy enables content served by
different servers to appear as if coming from one server. Configuration for Oracle HTTP
Server are specified through directives in configuration files in the exact same manner as
Apache HTTP Server configuration files.
Oracle HTTP Server directories are divided between the Oracle home and the Oracle
instance. The Oracle home directories are read-only, and contain the Oracle Fusion
Middleware binaries. The Oracle instance directories contain the modules, applications, and
logs for Oracle HTTP Server. Each OHS component has a root configuration directory found
at <instance>/config/OHS/<component>, which includes the WLS plug-in configuration
file, mod_wl_ohs.conf. Similarly, each component’s log files are found at
<instance>/diagnostics/logs/OHS/<component>.

OHS Architecture
Similar to WLS domains:

• A single OHS product installation can support multiple
processes or components.
• Processes that are managed together are called an
instance.
• Instances have a root directory under which process
configuration and log files are stored.
Component1 Component2
Client Client
Config, Logs Config, Logs
Instance
OHS Installation
OHS Architecture
Oracle HTTP Server directories are divided between the Oracle home and the Oracle
instance. The Oracle home directories are read-only and contain the Oracle Fusion
Middleware binaries. The Oracle instance directories contain the modules, applications, and
logs for Oracle HTTP Server.
Each OHS component has a root configuration directory found at
<instance>/config/OHS/<component>, which includes the WLS plug-in configuration
file, mod_wl_ohs.conf. Similarly, each component’s log files are found at
<instance>/diagnostics/logs/OHS/<component>.
When Oracle HTTP Server starts, it writes the process ID (PID) of the parent httpd process to
the httpd.pid file located, by default, in the following directory:
<instance>/diagnostics/logs/OHS/<component>
The process ID can be used by the administrator when restarting and terminating the
daemon. If a process stops abnormally, it is necessary to stop the httpd child processes by
using the kill command. The PidFile directive in httpd.conf specifies the location of the
PID file.

Oracle Process Manager and Notification Server
(OPMN)
OPMN is:
• A process that is used to start, manage, and monitor other
Oracle Web Tier and Identity Management processes
• Similar conceptually to the WLS node manager
• Accessed by using the opmnctl command-line tool or
Oracle web-based tools
Start Virtual
Directory
Stop
OHS 1
OPMNCTL OPMN
Status
OHS 2
Restart
Web Cache
Oracle Process Manager and Notification Server (OPMN)

Oracle Fusion Middleware provides a high-availability infrastructure integration with Oracle
Process Manager and Notification Server (OPMN) for process management, failure detection,
and failover for Oracle HTTP Server processes.
Oracle Fusion Middleware components that are managed by OPMN should never be started
or stopped manually. Do not use command-line scripts or utilities from previous versions of
Oracle Fusion Middleware for starting and stopping system components. OPMN must be the
last service turned off whenever you restart or turn off your computer.
Oracle Process Manager (PM) is the centralized process management mechanism used to
manage system processes. The PM is responsible for starting, restarting, stopping, and
monitoring every process it manages. The PM handles all requests sent to OPMN associated
with controlling a process or obtaining status about a process. The PM is also responsible for
performing death-detection and automatic restart of the processes it manages. The system
processes that PM is configured to manage are specified in the opmn.xml file.
Oracle Notification Server (ONS) is the transport mechanism for failure, recovery, startup, and
other related notifications between components in Oracle Fusion Middleware. It operates
according to a publish-subscribe model: A system component receives a notification of a
certain type for each subscription to ONS. When such a notification is published, ONS sends
it to the appropriate subscribers.

OPMNCTL: Examples
Start OPMN and all managed processes, if not already started:
opmnctl startall
Get the name, status, memory usage, and port number of processes:
opmnctl status -l
Restart a specific OHS process:
opmnctl restartproc ias-component=PayrollWeb1
Stop all OHS processes:
opmnctl stopproc process-type=OHS
OPMNCTL: Examples
The opmnctl program provides a centralized way to control and monitor system components
from the command line. The available commands include:
• start: Start the OPMN server for a local Oracle instance without starting system
processes.
• startall: Start OPMN as well as the system processes for a local Oracle instance.
startall is equivalent to start followed by startproc without arguments.
• stopall: Shut down the OPMN server as well as the system processes for the local
Oracle instance. This request operates synchronously. It waits for the operation to
complete before returning.
• startproc, restartproc, stopproc: Use these commands to start, restart, or stop system
processes. The OPMN server must be up and running.
The following attributes are supported. Any of these attributes may be omitted, and treated as
a wildcard:
• ias-component: The name of a specific managed process, as defined in opmn,xml
• process-type: The type of managed process to command, as defined in opmn.xml
• process-set: The name of a custom process group, as defined in opmn.xml

Basic Plug-In Parameters
Parameter Description
WebLogicHost, Proxy to a single server
WebLogicPort
WebLogicCluster • Proxy to this initial list of clustered servers
• Latest server list is refreshed with each response
ConnectTimeoutSecs Maximum total wait time to establish a connection
WLSocketTimeout Wait time for a connection attempt
Secs
MaxSkipTime Wait time before retrying a server marked as failed
ErrorPage URL to which to direct users if all servers are unavailable
Debug ON: Only error and informational messages
ERR: Only error messages
ALL: All messages and HTTP headers
Basic Plug-In Parameters

• WebLogicCluster: List of WebLogic Servers that can be used for load balancing. The
server or cluster list is a list of host:port entries. The plug-in does a simple round-robin
between all available servers. The server list specified in this property is a starting point
for the dynamic server list that the server and plug-in maintain. WebLogic Server and
the plug-in work together to update the server list automatically with new, failed, and
recovered cluster members.
• ErrorPage: Create your own local error page that is displayed when your web server is
unable to forward requests to WebLogic Server.
• WLSocketTimeoutSecs: Set the timeout for the socket while connecting, in seconds. The value
must be greater than 0 and the default is 2 seconds.
• MaxSkipTime: The amount of time after which the plug-in will retry the server marked as “bad.”
The plug-in attempts to connect to a new server in the list each time a unique request is received
(that is, a request without a session cookie). The default is 10 seconds.

OHS Plug-In: Example
Plug-ins route to servers or clusters based on the incoming

URL path, requested file type, or virtual host definition.
mod_wl_ohs.conf: Default parameters

<IfModule weblogic_module> for all locations
Debug ON
</IfModule> Proxy based
on URL path
<Location /hrWeb>
SetHandler weblogic-handler
WebLogicCluster host1:7001,host2:7001,host3:7001
</Location>
<Location /console> Parameters for this

SetHandler weblogic-handler specific location
WebLogicHost host4
WebLogicPort 7001
</Location>
OHS Plug-In: Example

For Apache HTTP Server, edit the httpd.conf file, or alternatively create a separate
configuration file and use the Include directive in httpd.conf. This technique is already
done for you if you are using Oracle HTTP Server. The separate configuration file is named
mod_wls_ohs.conf.
Oracle recommends that you use an Apache IfModule block to define parameters only when
the plug-in module is found. If you choose not to use the IfModule, you can instead directly
place the WebLogic properties inside Location or VirtualHost blocks. If you use an Apache
HTTP Server VirtualHost block, you must include all configuration parameters
(MatchExpression, for example) for the virtual host within the VirtualHost block.
If you are proxying requests to a cluster of WebLogic Servers, use the WebLogicCluster
parameter instead of the WebLogicHost and WebLogicPort parameters.
To proxy requests by path, use the Location block and the SetHandler statement. SetHandler
specifies the handler for the Apache HTTP Server Plug-In module. To proxy requests by
MIME type, add a MatchExpression line to the IfModule block. Note that if both MIME type
and proxying by path are enabled, proxying by path takes precedence over proxying by MIME
type. You can also use multiple MatchExpressions.

Quiz
A ____ application participates in the partitioning of cache data

in a Coherence cluster.
a. Proxy-enabled
b. Proxy-disabled
c. Storage-enabled
d. Storage-disabled
Answer: c

Quiz
Name three concepts associated with Oracle HTTP Server.

a. Cache
b. Instance
c. Component
d. Node Manager
e. Process Manager
Answer: b, c, e

Summary

• Describe the features of Coherence
• Configure a Coherence server and add it to a domain
• Describe the features of Web Tier
• Configure Web Tier to proxy to a cluster
• Start Coherence and Web Tier processes

Practice 12-1:
Configure a Cluster Proxy
• Adding an instance to a Web Tier installation
• Creating and start an OHS component
• Configuring OHS to proxy to a WebLogic cluster
• Deploying an application to a cluster
• Monitoring failover within a cluster

Managing Hardware with ILOM

Objectives

• Describe the features of ILOM
• Update device firmware
• View device status and health
• Configure conditions that trigger alerts
• Access ILOM from the command line

Integrated Lights Out Manager (ILOM)
ILOM:
• Is made up of specialized hardware and software for
managing and monitoring Sun devices
• Is preinstalled on Exalogic compute nodes, storage nodes,
and InfiniBand switches
• Runs independently of any operating systems
• Is accessible from dedicated management ports
• Can still be accessed in standby power mode
• Provides browser-based and command-line interfaces
• Supports SNMP and IPMI clients
Integrated Lights Out Manager (ILOM)

The Oracle ILOM provides advanced service processor hardware and software that you can
use to manage and monitor your Oracle Sun servers. ILOM’s dedicated hardware and
software are preinstalled on a variety of Oracle Sun server platforms, including x86-based
Sun Fire servers, Sun Blade modular chassis systems, and Sun Blade server modules, as
well as on SPARC based servers. ILOM is a vital management tool in the data center and can
be used to integrate with other data center management tools already installed on your
systems. ILOM gives operators a consistent management interface across multiple product
lines.
The ILOM service processor (SP) runs its own embedded operating system and has a
dedicated Ethernet port, which together provide out-of-band management capability. ILOM
enables you to actively manage and monitor the server independently of the host operating
system state, such as Linux or Solaris.
ILOM automatically initializes as soon as power is applied to your server. It provides a full-
featured, browser-based web interface and has an equivalent command-line interface (CLI).
There is also an industry-standard Simple Network Management Protocol (SNMP) interface
and an Intelligent Platform Management Interface (IPMI) implementation.

ILOM Tasks
With ILOM, you can remotely:

• Power the system on or off
• Determine the current hardware, firmware, and network
configurations
• Upgrade device firmware
• Obtain information about hardware errors and faults
• View the current status of sensors and LED indicators
• Configure generated alerts for system events by using
SNMP, IPMI, or email
• Define administrative users and access rights
• Browse device logs
ILOM Tasks
The system monitoring features in ILOM enable you to easily determine the health of the
system and to detect errors, at a glance, when they occur. All Oracle Sun server platforms are
equipped with a number of sensors that measure voltages, temperatures, fan speeds, and
other attributes about the system. Each sensor in ILOM contains nine properties that describe
various settings related to a sensor such as sensor type, sensor class, and sensor value, as
well as the sensor values for upper and lower thresholds. Additionally, if an alert rule was
enabled in the system that matched the crossing threshold level, ILOM would automatically
generate an alert message to the alert destination that you have defined.
ILOM receives error telemetry about error events that occur within the major system
components on the host (CPU, memory, and I/O hub) and the environmental subsystem
within the chassis (such as fans, power supplies, and temperature). The components and
conditions are then diagnosed as fault events and captured in the ILOM event log.
The Component Management features in ILOM enable you to monitor the state of various
components that are installed on the server or managed by the Chassis Monitoring Module
(CMM). You can identify the component name and type, enable or disable it, or check the
current fault status.

Sideband Management
• Out-of-band management: The ILOM processor's and host

processor's network interfaces are independent.
• Sideband management: The ILOM processor uses the
same network interface as the host OS.
• Compute and storage nodes:
– Have separate management and host Ethernet ports
– Are configured with sideband by default, to avoid additional
switch and cabling requirements
Management
Host CPUs
Switch
Service
Processor
Sideband Management
You can connect to the server’s service processor (SP) by using the out-of-band network
management port (NET MGT). Alternatively, with the ILOM sideband management feature,
you can select one of the server’s Gigabit Ethernet ports (NET 0, 1, 2, 3), which are in-band
ports, to send and receive ILOM commands to and from the server SP. In-band ports are also
called sideband ports. Exalogic compute nodes are configured at the time of manufacturing to
use sideband management via the NET0 port.
The advantage of using a sideband management port to manage the server’s SP is that one
less cable connection and one less network switch port is needed. In configurations where a
great number of servers is being managed, such as data centers, sideband management can
represent significant savings in hardware and network utilization. One potential disadvantage
is that while the server is being powered on, you may experience a brief interruption in ILOM
network connectivity.
You require at least one static IP address for Service Processor ILOM access. If you
reconfigure these IP addresses during the initial configuration of the Exalogic machine that is
using Oracle OneCommand and its associated tools, you must use those IP addresses to
access ILOM.

Accessing the Browser Interface
http://<mgmt_address>
Accessing the Browser Interface

You can establish communication with ILOM through a console connection to the serial
management port on the server or through an Ethernet connection to the network port on the
server that ILOM is configured to use.
The ILOM web interface uses a model of hierarchical tabbed pages that you select by clicking
the tab name. When a page is displayed, you can provide information, set parameters, or
access other subtabs. For some pages, initiating a task might spawn an additional window,
which accepts further parameters.
Open a web browser and connect to the ILOM web interface by specifying the management
controller’s network address in the URL. The default ILOM host names in Exalogic are of the
form <host>-c. For example, exl1cn01-ilom or exlsn01-ilom. Refer to the Exalogic
documentation for the default ILOM IP addresses that are configured at the factory. The
default credentials are root/welcome1.
ILOM supports a maximum of 10 active user sessions, including serial, Secure Shell (SSH),
and web interface sessions per service processor (SP).

Management Settings
1
2
3
Change ports for this

interface (default is 443)
Change the
physical port
1
3
2
Management Settings
By using the Configuration tab you can customize ILOM communication on this device. From
the Web Server subtab, enable/disable the ILOM web interface, enable/disable SSL, or
change the default web server ports (80/443). From the Configuration > Network tab, change
the network settings for ILOM, including the IP address, subnet mask, and gateway address.
You can also change the physical port that ILOM binds to. Keep in mind, however, that Oracle
only supports sideband management for Exalogic (using NET0 by default).
Additional settings are available from the Configuration tab to:
• Enable/disable the ILOM command-line interface (CLI) and to configure inactivity
timeout
• Configure SSL certificates for HTTPS and SSH access to ILOM
• Enable and configure SNMP and/or IPMI client access

Adding Administrative Users
Accounts can be stored locally or in an external LDAP.
Adding Administrative Users

ILOM supports local authentication or remote, centralized authentication via Lightweight
Directory Access Protocol (LDAP), Active Directory, or Remote Authentication Dial-In User
Service (RADIUS).
Single Sign-On (SSO) is a convenient authentication service with which you can log in to
ILOM once to establish your credentials, thus reducing the number of times you need to enter
your password to gain access to ILOM. SSO is enabled by default.
To create a locally defined user account:
1. Click the User Management tab. Confirm that the User Accounts subtab is selected.
2. Within the Users section, click Add.
3. The username of an account must be at least four characters and no more than 16
characters. Usernames are case-sensitive and must start with an alphabetical character.
You can use alphabetical characters, numerals, hyphens, and underscores. Do not
include spaces in usernames.

ILOM Roles
Role Privileges
Read Only View but not modify all configuration settings

Admin Access all features except those covered below
User Management • Create and update users
• Configure external LDAP
Console Access the host OS by using the KVM remote
console
Reset and Host • Power on/off the device
Control • Enable/disable hardware components
• Manage hardware faults
Service • Can assist Oracle support personnel during an on-
site service engagement
ILOM Roles
For backward compatibility, ILOM 3.0 supports ILOM 2.x user accounts such that users with
ILOM 2.x Administrator or Operator privileges are granted ILOM 3.0 roles that match those
privileges. The 2.x Administrator role maps to Admin (a), User Management (u), Console (c),
Reset and Host Control (r), and Read Only (o). The 2.x Operator role maps to Console (c),
Reset and Host Control (r), and Read Only (o).

Powering the Device On or Off
Powering the Device On or Off

1. Click the Remote Control tab.
2. Click the Remote Power Control subtab.
3. Select an action and click Save:
- Reset: This option immediately reboots the remote host server.
- Immediate Power Off: This option immediately turns off the power on the remote
host server.
- Graceful Shutdown and Power Off: This option shuts down the OS gracefully
prior to powering off the remote host server.
- Power On: This option turns on full power to a remote host server running on
standby power.
- Power Cycle: This option immediately turns off the power on the remote host
server, and then applies full power to the remote host server.

Updating Device Firmware
1
2
Provide a URL or
upload from your
client machine.
Updating Device Firmware

The process for updating the firmware version installed on your Sun server or CMM involves
the following steps:
1. Downloading the firmware image for your server or CMM from the Sun platform’s
product website and placing the image on your TFTP, FTP, or HTTP server
2. If required by your platform, shutting down the host operating system before changing
the firmware on your server SP
3. Logging in to ILOM using an Admin role account
4. Loading the firmware image on the server service processor (SP) by using the ILOM CLI
or the web interface
5. (Optional) Preserving the current configuration in ILOM. By default, all prior configuration
changes are lost.
6. Verifying that the appropriate firmware version was installed after the system reboots

ILOM Glossary
Term Definition
MB Motherboard
P Processor
D Memory module
FB Fan board
FM Fan module
F Fan
DBP Disk array (backplane)
HDD Disk (hard or solid state)
PS Power supply
SP Service processor
ILOM Glossary
Below are some examples of ILOM sensors and indicators:
/SYS/MB/P1/SERVICE – Host CPU service LED. If the CPU fails, this LED illuminates
amber
/SYS/MB/P1/D1/PRSNT – DIMM 1 on host CPU 1 is present.
/SYS/FB/FM1/OK – This is a fan OK LED. It illuminates green to indicate that the fan module
is operating properly.
/SYS/DBP/PRSNT – Disk backplane is present sensor.
/SYS/DBP/HDD1/SERVICE – Hard disk 1 service LED. Illuminates amber when the hard
disk needs servicing.
/SYS/PS1/VINOK – The voltage of power supply 1 is OK.

Using Indicator LEDs
2
View current
LED state.
3
Change
LED state.
Using Indicator LEDs

System indicator LEDs are generally illuminated on the system by ILOM based on the server
platform policy. Typically, the system indicator LEDs are illuminated by ILOM when any of the
following conditions occur:
• A fault or error is detected on a component.
• A field-replacement unit (FRU) requires service.
• A hot-plug module is ready for removal.
• Activity is occurring on FRU or system.
You can view the states of system indicators from the ILOM web interface or the CLI.
Additionally, in some instances, you might be able to modify the state of a system indicator.

Viewing Sensor Readings
Viewing Sensor Readings

All Oracle Sun server platforms are equipped with a number of sensors that measure
voltages, temperatures, fan speeds, and other attributes about the system. Each sensor in
ILOM contains nine properties that describe various settings related to a sensor such as
sensor type, sensor class, and sensor value, as well as the sensor values for upper and lower
thresholds.
ILOM regularly polls the sensors in the system and reports any events it encounters about
sensor state changes or sensor threshold crossings to the ILOM event log. Additionally, if an
alert rule was enabled in the system that matched the crossing threshold level, ILOM
automatically generates an alert message to the alert destination that you have defined.

Managing Power Consumption
1
2
Managing Power Consumption

Actual Power represents the consumed power for the server.
The Peak Permitted power represents the maximum input power that the server guarantees it
will consume at any instant. Hardware maximum power identifies the maximum input power
that a system is capable of consuming at any instant, given the hardware configuration of the
system. Therefore, the hardware configuration maximum power is the sum of the maximum
power that each processor, I/O module, memory module, fan, and so forth is capable of
consuming.
The Target Limit property represents the power consumption limit value that was configured
for the server. Configure limits using the Limit subtab. Advanced options include power
capping policies and violation actions. For example, after power consumption has reached the
specified limit for a specified grace period, you can force the server to cycle its power.
By using the Notification Threshold settings, you can generate two power consumption
notifications when the specified power consumption value exceeds the threshold. An ILOM
event is then generated, which can in turn trigger alerts.

Analyzing Power Consumption
Analyzing Power Consumption

By using the Power Management > Allocation feature in ILOM, you can monitor, in real time,
the amount of power that is allocated to server components. ILOM provides an allocated
server power value in wattage that represents the sum of the maximum power consumed by
either a single server component (such as a memory module), a category of server
components (all memory modules), or all server components. It also identifies whether or not
each allocated power value can be capped, if you have configured a power limit policy.
The Power Management > History feature provides a rolling average of power consumption
by the server in 15-, 30-, and 60-second intervals. You can also obtain and analyze a sample
data set of the power consumed by the system for a specific duration: 1 hour or 14 weeks.

Alert Concepts
An ILOM alert rule is composed of the following:

• A level, which specifies the types of events that trigger this
alert: Minor, Major, Critical, or Down
• A destination: email, SNMP, or IPMI
• Optional advanced filtering criteria for email alerts
Alert Concepts
Each server platform is equipped with a number of sensors that measure voltages,
temperatures, and other service-related attributes about the system. ILOM automatically polls
these sensors and posts any events crossing a threshold to an ILOM event log. It also
generates alert messages to one or more customer-specified alert destinations.
ILOM supports alerts in the form of Intelligent Platform Management Interface (IPMI) PET
alerts, Simple Network Management Protocol (SNMP) traps, and email notifications. Alerts
provide advance warning of possible system failures.
In ILOM, you can configure up to 15 alert rules by using the ILOM web interface or CLI. For
each alert rule you configure in ILOM, you must define three or more properties about the
alert, depending on which of the three alert types you choose.
Alert levels act as a filter mechanism to ensure alert recipients only receive the alert
messages that they are most interested in receiving. Each time you define an alert rule in
ILOM, you must specify an alert level. The lowest level alert generates alerts for that level and
for all alert levels above it. ILOM does not support alert-level filtering for SNMP traps.
The Down alert level generates alerts for only upper non-recoverable and lower non-
recoverable events. The Disabled level disables the alert altogether.

Configuring Alerts
1
2
3
4
Configuring Alerts
1. Click the Configuration tab.
2. Click the Alert Management subtab.
3. Under Alerts, select one of the empty table rows and click Edit.
4. Provide an alert Level and Type. For each IPMI PET alert you configure in ILOM, you
must specify an IP address. For each SNMP trap alert, you must specify an IP address,
a port number, the version of SNMP to use, and an SNMP user/community name. For
each email alert, you specify an email address, the name of the sender, and, optionally,
any advanced filtering criteria.
5. When you finish, you can test the alert by using the Test Rule button.

Command-Line Interface (CLI) Example
$ ssh root@exl3cn01-ilom
...
-> show /SYS/T_AMB
Properties:
type = Temperature
ipmi_name = T_AMB
class = Threshold Sensor
value = 23.000 degree C
upper_nonrecov_threshold = 50.000 degree C
upper_critical_threshold = 45.000 degree C
Command-Line Interface (CLI) Example

Use the help command to display help information about commands and targets. Available
commands include cd, create, delete, exit, help, load, reset, set,
show, start, stop, and version.
The targets and properties below /SP are used for configuring the ILOM service processor
(SP) and for viewing logs and consoles. For example, to change the IP address of the ILOM
SP:
set /SP/network pendingipdiscovery=static pendingipaddress=
198.162.101.17 commitpending=true
The targets and properties below /SYS provide hardware inventory, monitoring, and
management. The targets directly correspond to the nomenclature for all hardware
components, some of which are printed onto the physical hardware.
The targets and properties below /HOST are used for monitoring and managing the host
operating system.

Connecting to the Host OS
ILOM includes a Java based remote console to redirect the

OS keyboard, video, and mouse (KVM) inputs to the
management network.
3
1 exl3cn01-ilom
Connecting to the Host OS

The Oracle ILOM Remote Console is a Java application that you can launch from the ILOM
web interface. It enables the devices on your local client to behave as if they were directly
attached to the remote host server. When you use the Oracle ILOM Remote Console, you can
remotely redirect and control the following devices on a remote host server:
• Keyboard
• Mouse
• Video console display
• Storage devices or images (CD/DVD, floppy device, ISO image)
To run the Oracle ILOM Remote Console, you must have the JRE 1.5 or higher (Java 5.0 or
higher) software installed on your local client. When you launch it from the ILOM web
interface, you must sign in using an Admin (a) or Console (c) role account.

Quiz
Name three tasks that you can perform with ILOM.

a. Analyze power consumption
b. Monitor hardware sensors
c. Create OS users and roles
d. Create alert destinations
e. Repair failed hardware
Answer: a, b, d

Quiz
Which type of ILOM network configuration does Exalogic use

by default?
a. In band
b. Out of band
c. Net band
d. Side band
Answer: d

Summary

• Describe the features of ILOM
• Update device firmware
• View device status and health
• Configure conditions that trigger alerts
• Access ILOM from the command line

Recorded Demonstration:
Compute Node ILOM
This demonstration covers the following tasks:
• Inspecting hardware faults
• Locating hardware component specifications
• Monitoring hardware sensors and LED indicators
• Monitoring power consumption and allocation
• Locating ILOM access and user configurations
• Launching the KVM remote console

Oracle Exalogic Elastic Cloud Administration: D71299GC20 Edition 2.0 May 2011 D73135

Uploaded by

Copyright:

Available Formats

Oracle Exalogic Elastic Cloud Administration: D71299GC20 Edition 2.0 May 2011 D73135

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oracle Exalogic Elastic Cloud Administration: D71299GC20 Edition 2.0 May 2011 D73135

Uploaded by

Copyright:

Available Formats

Oracle Exalogic Elastic Cloud

This document contains proprietary information and is protected by copyright and

2 The Exalogic Solution

4 Initial Machine Configuration

8 Creating FMW Domains

9 Optimizing FMW Domains on Exalogic

10 Starting and Migrating Servers

12 Additional FMW Infrastructure

13 Managing Hardware with ILOM

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

After completing this course, you should be able to:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 2

• This course is intended for data center administrators.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 3

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 4

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 5

3 AM 10: Starting and Migrating Servers

PM 11: Exalogic Database Connectivity

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Course Schedule (continued)

Oracle Exalogic Elastic Cloud Administration 1 - 6

• Many topics are reinforced with a hands-on practice.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 7

• The instructor starts each session at the scheduled time.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 8

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

For More Information

Oracle Exalogic Elastic Cloud Administration 1 - 9

OBEs are free online step-by-step tutorials that cover specific

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle by Example (OBE)

Oracle Exalogic Elastic Cloud Administration 1 - 10

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 1 - 11

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

After completing this lesson, you should be able to:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 2 - 2

• Traditional distributed applications suffer from:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Why Data Centers?

Oracle Exalogic Elastic Cloud Administration 2 - 3

Common concerns include:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Data Center Challenges

Oracle Exalogic Elastic Cloud Administration 2 - 4

CPUs, storage, Optimized Oracle

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Exalogic Elastic Cloud Administration 2 - 5

Oracle Exalogic hardware:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

What Is Exalogic? (continued)

Oracle Exalogic Elastic Cloud Administration 2 - 6