Exadata Database Machine

Administration Workshop
Student Guide Volume I

D73668GC21
Edition 2.1
February 2014
D85490

Author
Peter Fusek
Technical Contributors
& Reviewers
Akshay Shah
Alex Tsukerman
Amit Ganesh
Andrew Babb
Aslam Edah-Tally
Barb Lundhild
Bharat Baddepudi
Bill Hodak
Boris Erlikhman
Branislav Valny
Bruce Kyro
Caroline Johnston
Christian Craft
Dan Norris
Darryl Presley
Dave Winter
David Hernandez Mendoza
David Hitchcock
Deba Chatterjee
Douglas Utzig
Ed Gilowski
Eric Siglin
Georg Schmidt
Harald van Breederode
Ira Singer
James He
James Womack
Jean-Francois Verrier
Jia Shi
Jignesh Patel
Jim Hall
Jim Spiller
Jim Viscusi
Joel Goodman
Juan Loaiza

Kam Shergill

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Kevin Jernigan

Disclaimer

Kodi Umamageswaran

This document contains proprietary information and is protected by

copyright and other intellectual property laws. You may copy and print
this document solely for your own use in an Oracle training course.
The document may not be modified or altered in any way. Except
where your use constitutes "fair use" under copyright law, you may
not use, share, download, upload, copy, print, display, perform,
reproduce, publish, license, post, transmit, or distribute this document
in whole or in part without the express authorization of Oracle.

Krishnanjani Chitta
Lachlan Williams
Larry Justice
Lawrence To

Maria Billings

The information contained in this document is subject to change

without notice. If you find any problems in the document, please
report them in writing to: Oracle University, 500 Oracle Parkway,
Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.

Maria Colgan

Restricted Rights Notice

Mark Fuller

If this documentation is delivered to the United States Government or

anyone using the documentation on behalf of the United States
Government, the following notice is applicable:

Louis Nagode
Mahesh Subramaniam

Mark Scardina
Mark Van de Wiel
Marshall Presser
Martin Jensen
Michael Cebulla
Michael Nowak
Naoki Kato

U.S. GOVERNMENT RIGHTS

The U.S. Governments rights to use, modify, reproduce, release,
perform, display, or disclose these training materials are restricted by
the terms of the applicable Oracle license agreement and/or the
applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its
affiliates. Other names may be trademarks of their respective owners.

Nilesh Choudhury
Ravindra Dani
Raymond Dutcher
Richard Exley
Robert Carlin
Robert Pastijn
Roger Hansen
Sabyasachi Banerjee
Sean Kim
Selcuk Aya
Scott Gossett
Sriram Palapudi
Steven Lemme
Sue Lee
Sugam Pandey
Sumeet Lahorani

Varun Malhotra
Vern Wagman
Vijay Sridharan
Vikram Kapoor
Vimala Jacob
Editors
Smita Kommini
Aju Kumar
Anwesha Ray
Graphic Designer
Seema Bopaiah

Sundararaman Sridharan
Tim Shelter
Umesh Panchaksharaiah
Uwe Hesse

Publishers
Pavithran Adka
Veena Narasimhan
Jobi Varghese
Sumesh Koshy

Contents

Introduction
Course Objectives 1-2
Audience and Prerequisites 1-3
Course Contents 1-4
Terminology 1-5
Additional Resources 1-6
Practice 1 Overview: Introducing the Laboratory Environment 1-7

Exadata Database Machine: Overview

Objectives 2-2
Introducing Database Machine 2-3
Why Database Machine? 2-4
Introducing Exadata Storage Server 2-6
Exadata Storage Server Architecture: Overview 2-7
Exadata Storage Server Features: Overview 2-8
Exadata Storage Server X3-2 Hardware Overview 2-10
Exadata Storage Server X3-2 Configuration Options 2-11
Database Machine X3-2 Full Rack 2-12
X3-2 Database Server Hardware: Overview 2-13
Start Small and Grow 2-14
Database Machine X3-8 Full Rack 2-16
X3-8 Database Server Hardware: Overview 2-17
Exadata Storage Expansion Racks 2-18
InfiniBand Network: Overview 2-19
Database Machine Support: Overview 2-20
Oracle Platinum Services: Enhanced Support at No Additional Cost 2-21
Database Machine Benefits for Data Warehousing 2-22
Database Machine Benefits for OLTP 2-24
Quiz 2-25
Summary 2-27
Additional Resources 2-28

iii

Exadata Database Machine Architecture

Objectives 3-2
Database Machine Architecture: Overview 3-3
Database Machine Network Architecture 3-5
InfiniBand Network Architecture 3-7
Leaf Switch Topology 3-8
Spine and Leaf Topology 3-9
Scale Performance and Capacity 3-10
Typical Scaling Scenarios 3-11
Scaling Out to Eight Racks 3-13
Scaling Out Between 9 and 18 Racks 3-14
Scaling Out Between 19 and 36 Racks 3-15
Interconnecting Quarter Racks and Eighth Racks 3-16
InfiniBand Network External Connectivity 3-18
Database Machine Software Architecture: Overview 3-19
Database Machine Software Architecture Details 3-22
Disk Storage Entities and Relationships 3-23
Flash Storage Entities and Relationships 3-25
Disk Group Configuration 3-26
Quiz 3-27
Summary 3-31
Practice 3 Overview: Introducing Exadata Cell Architecture 3-32

Key Capabilities of Exadata Database Machine

Objectives 4-2
Classic Database I/O and SQL Processing Model 4-3
Exadata Smart Scan Model 4-4
Exadata Smart Storage Capabilities 4-5
Exadata Smart Scan Scale-Out: Example 4-8
Exadata Hybrid Columnar Compression: Overview 4-11
Exadata Hybrid Columnar Compression Data Organization 4-12
Exadata Smart Flash Cache: Overview 4-13
Exadata Smart Flash Cache Intelligent Caching: Overview 4-14
Exadata Smart Flash Cache Intelligent Caching Details 4-15
Using Exadata Smart Flash Cache: Write-Through Cache 4-17
Using Exadata Smart Flash Cache: Write-Back Cache 4-18
Exadata Smart Flash Log: Overview 4-20
Exadata Storage Index: Overview 4-21
Storage Index with Partitions: Example 4-23
Database File System 4-24
I/O Resource Management: Overview 4-25

iv

Multiplied Benefits 4-26

Quiz 4-27
Summary 4-28
Additional Resources 4-29
Practice 4 Overview: Introducing Exadata Features 4-31
5

Exadata Database Machine Initial Configuration

Objectives 5-2
Database Machine Implementation: Overview 5-3
Key Documentation 5-5
Database Machine Site Preparation 5-6
Oracle Exadata Deployment Assistant: Overview 5-7
Oracle Exadata Deployment Assistant: Customer Details 5-8
Oracle Exadata Deployment Assistant: Hardware Selection 5-9
Oracle Exadata Deployment Assistant: Networking 5-10
Oracle Exadata Deployment Assistant: Administration Network 5-11
Administration Network IP Address Allocation: Example 5-12
Oracle Exadata Deployment Assistant: Client Ethernet Network 5-13
Client Ethernet Network IP Address Allocation: Example 5-14
Oracle Exadata Deployment Assistant: InfiniBand Network 5-15
InfiniBand Network IP Address Allocation: Example 5-16
Oracle Exadata Deployment Assistant: Backup / Data Guard Network 5-17
Oracle Exadata Deployment Assistant: OS Configuration 5-18
Oracle Exadata Deployment Assistant: Home and Database 5-19
Choosing the Right Disk Group Redundancy Setting 5-21
Oracle Exadata Deployment Assistant: Cell Alerting 5-23
Oracle Exadata Deployment Assistant: Oracle Configuration Manager 5-24
Oracle Exadata Deployment Assistant: Auto Service Request 5-25
Oracle Exadata Deployment Assistant: Grid Control Agent 5-26
Oracle Exadata Deployment Assistant: Review and Edit Details 5-27
Oracle Exadata Deployment Assistant: Create Files 5-28
Database Machine Hardware Installation: Overview 5-29
Configuring Oracle Exadata Database Machine: Overview 5-30
Selecting the Database Server Operating System 5-31
Deploying Solaris on the Database Servers 5-32
Reclaiming Unused Operating System Disk Space Using Linux 5-33
Reclaiming Unused Operating System Disk Space Using Solaris 5-35
Performing Initial Network Configuration 5-36
Loading the Configuration Information and Installing the Software 5-38
Running OneCommand on Database Machine 5-39
Exadata Storage Configuration 5-40

Result After Installation and Configuration 5-42

Supported Additional Configuration Activities 5-43
Hardware Re-Racking 5-44
Unsupported Configuration Activities 5-45
Quiz 5-46
Summary 5-48
Practice 5 Overview: Using the Oracle Exadata Deployment Assistant 5-49
6

Exadata Storage Server Configuration

Objectives 6-2
Exadata Storage Server Administration: Overview 6-3
Exadata Storage Server Administrative User Accounts 6-4
dcli: Overview 6-5
dcli: Examples 6-6
Testing Storage Server Performance by Using CALIBRATE 6-7
CALIBRATE: Example 6-8
Configuring the Exadata Cell Server Software 6-9
Starting and Stopping Exadata Cell Server Software 6-10
Configuring Cell Disks 6-11
Configuring Grid Disks 6-12
Interleaved Grid Disks 6-13
Interleaved Grid Disks and ASM Intelligent Data Placement 6-14
Creating Smart Flash Log 6-15
Configuring Hosts to Access Exadata Cells 6-16
Configuring ASM and Database Instances to Access Exadata Cells 6-17
Configuring ASM Disk Groups by Using Exadata Storage 6-18
Specifying Content Type for a Disk Group 6-19
Reconfiguring Exadata Storage 6-21
Optional Configuration Tasks 6-23
Exadata Storage Security: Overview 6-24
Exadata Storage Security Implementation 6-25
Quiz 6-27
Summary 6-30
Additional Resources 6-31
Practice 6 Overview: Configuring Exadata 6-32

I/O Resource Management

Objectives 7-2
I/O Resource Management: Overview 7-3
I/O Resource Management Concepts 7-5
I/O Resource Management Plans 7-6

vi

I/O Resource Management Plans: Example 7-7

IORM Architecture 7-10
Getting Started with IORM 7-11
Setting the IORM Objective 7-12
Enabling Intradatabase Resource Management 7-13
Intradatabase Plan: Example 7-14
Enabling IORM for Multiple Databases 7-15
Interdatabase Plan: Example 7-16
Using Share-Based Allocation in the Interdatabase Plan 7-17
Setting Database I/O Utilization Limits 7-18
Interdatabase Plans and Database Roles 7-19
Category Plan: Example 7-20
Using Database I/O Metrics 7-21
IORM and Exadata Storage Server Flash Memory 7-22
Complete Example 7-23
Quiz 7-26
Summary 7-30
Additional Resources 7-31
8

Recommendations for Optimizing Database Performance

Objectives 8-2
Optimizing Performance 8-3
Flash Memory Usage 8-4
Influencing Caching Priorities 8-5
Choosing the Flash Cache Mode 8-6
Setting the Flash Cache Mode 8-7
Compression Usage 8-8
Index Usage 8-10
ASM Allocation Unit Size 8-11
Minimum Extent Size 8-12
Exadata Specific System Statistics 8-13
Quiz 8-14
Summary 8-16
Additional Resources 8-17
Practice 8 Overview: Optimizing Database Performance with Exadata 8-18

Using Smart Scan

Objectives 9-2
Exadata Smart Scan: Overview 9-3
Smart Scan Requirements 9-4
Situations Preventing Smart Scan 9-6

vii

Monitoring Smart Scan in SQL Execution Plans 9-7

Smart Scan Execution Plan: Example 9-8
Example of a Situation Preventing Smart Scan 9-10
Smart Scan Join Processing with Bloom Filters 9-11
Smart Scan Join Filtering: Example 9-12
Other Situations Affecting Smart Scan 9-13
Exadata Storage Server Statistics: Overview 9-14
Exadata Storage Server Wait Events: Overview 9-15
Smart Scan Statistics: Example 9-16
Smart Scan Wait Events: Example 9-17
Concurrent Transaction: Example 9-18
Extreme Concurrent Transaction: Example 9-19
Migrated Rows: Example 9-20
I/O Sent Directly to Database Server to Balance CPU Usage: Example 9-21
Column Filtering: Example 9-22
Summary 9-23
Quiz 9-24
Practice 9 Overview: Using Smart Scan 9-26
10 Consolidation Options and Recommendations
Objectives 10-2
Consolidation: Overview 10-3
Different Consolidation Types 10-4
Core Principles for Database Consolidation 10-5
Recommended Consolidation Approach 10-6
Recommended Storage Configuration for Consolidation 10-7
Alternative Storage Configurations 10-8
Benefits and Limitations of Partitioned Storage Configurations 10-9
Cluster Configuration Options 10-10
Operating System Parameter Recommendations 10-11
Database Memory Recommendations 10-13
CPU Management Recommendations 10-14
Process Management Recommendations 10-16
Other Recommendations 10-18
Isolating Management Roles 10-19
Schema Consolidation Recommendations 10-21
Maintenance Considerations 10-22
Quiz 10-23
Summary 10-25
Additional Resources 10-26

viii

11 Migrating Databases to Exadata Database Machine

Objectives 11-2
Migration Best Practices: Overview 11-3
Performing Capacity Planning 11-4
Database Machine Migration Considerations 11-5
Choosing the Right Migration Path 11-6
Logical Migration Approaches 11-7
Physical Migration Approaches 11-9
Reducing Down Time for Migration by Using Transportable Tablespaces 11-11
Other Approaches 11-12
Post-Migration: Best Practices 11-13
Quiz 11-14
Summary 11-16
Additional Resources 11-17
Practice 11 Overview: Migrating to Databases Machine by Using Transportable
Tablespaces 11-19
12 Bulk Data Loading by Using Oracle DBFS
Objectives 12-2
Bulk Data Loading Using Oracle DBFS: Overview 12-3
Preparing the Data Files 12-4
Staging the Data Files 12-5
Configuring the Staging Area 12-6
Configuring the Target Database 12-10
Loading the Target Database 12-11
Quiz 12-13
Summary 12-15
Additional Resources 12-16
Practice 12 Overview: Bulk Data Loading Using Oracle DBFS 12-17
13 Exadata Database Machine Platform Monitoring: Introduction
Objectives 13-2
Monitoring Technologies and Standards 13-3
Simple Network Management Protocol (SNMP) 13-4
Intelligent Platform Management Interface (IPMI) 13-5
Integrated Lights Out Manager (ILOM) 13-6
Exadata Storage Server Metrics, Thresholds, and Alerts 13-7
Automatic Diagnostic Repository (ADR) 13-8
Enterprise Manager Cloud Control 12c 13-9
Enterprise Manager Database Control 13-10
Quiz 13-11

ix

Summary 13-12
Practice 13 Overview: Environment Reconfiguration 13-13
14 Configuring Enterprise Manager Cloud Control 12c to Monitor Exadata
Database Machine
Objectives 14-2
Enterprise Manager Cloud Control 12c: Architecture Overview 14-3
Enterprise Manager Cloud Control 12c: Supported Exadata Configurations 14-4
Cloud Control Monitoring Architecture for Exadata Database Machine 14-5
Configuring Cloud Control to Monitor Exadata Database Machine 14-6
Pre-Discovery Configuration and Verification 14-7
Deploying the Oracle Management Agent 14-9
Discovering Exadata Database Machine 14-10
Discovering Additional Targets 14-11
Configuring an Exadata Database Machine Dashboard 14-12
Post-Discovery Configuration and Verification 14-13
Quiz 14-14
Summary 14-17
Additional Resources 14-18
Practice 14 Overview: Exadata Monitoring Configuration 14-19
15 Monitoring Exadata Storage Servers
Objectives 15-2
Lesson Overview 15-3
Exadata Metrics and Alerts Architecture 15-4
Monitoring Exadata Storage Server with Metrics 15-6
Monitoring Exadata Cell Metrics: Examples 15-8
Monitoring Exadata Storage Server with Alerts 15-9
Monitoring Cell Alerts and Creating Thresholds: Examples 15-11
Isolating Faults with Exadata Storage Server Quarantine 15-13
Monitoring Exadata Storage Server with Active Requests 15-15
Monitoring Exadata Storage Server with Enterprise Manager: Overview 15-16
Monitoring Hardware Failure and Sensor State 15-18
Monitoring Exadata Storage Server Availability 15-19
Checking for Undelivered Alerts 15-20
Checking for Disk I/O Errors 15-21
Checking for Network Errors 15-22
Monitoring File System Free Space 15-23
Comparing Metrics Across Multiple Storage Servers 15-24
Monitoring Metrics in a Storage Server 15-25
Third-Party Monitoring Tools 15-26

Quiz 15-27
Summary 15-29
Practice 15 Overview: Monitoring Exadata Storage Server 15-30
16 Monitoring Exadata Database Machine Database Servers
Objectives 16-2
Monitoring Database Servers: Overview 16-3
Monitoring Hardware 16-4
Monitoring the Operating System 16-5
Monitoring Oracle Grid Infrastructure 16-6
Monitoring Oracle Database 16-7
Monitoring Oracle Management Agent 16-8
Database Monitoring with Enterprise Manager Cloud Control 12c 16-9
Quiz 16-10
Summary 16-11
Practice 16 Overview: Oracle Database Monitoring 16-12
17 Monitoring the InfiniBand Network
Objectives 17-2
InfiniBand Network Monitoring: Overview 17-3
InfiniBand Network Monitoring with Enterprise Manager Cloud Control 12c 17-4
Monitoring the InfiniBand Switches 17-5
Monitoring the InfiniBand Switch Ports 17-6
Monitoring the InfiniBand Ports on Database Machine Servers 17-7
Monitoring the InfiniBand Fabric: Subnet Manager Master Location 17-8
Monitoring the InfiniBand Fabric: Network Topology and Link Status 17-9
Quiz 17-10
Summary 17-11
Practice 17 Overview: InfiniBand Monitoring 17-12
18 Monitoring Other Exadata Database Machine Components
Objectives 18-2
Monitoring the Cisco Ethernet Switch 18-3
Monitoring the Sun Power Distribution Units 18-4
Monitoring the KVM Switch 18-5
Quiz 18-6
Summary 18-7
19 Other Useful Monitoring Tools
Objectives 19-2
Exachk: Overview 19-3

xi

Running Exachk 19-4

Exachk Output 19-5
Exachk Daemon 19-6
DiagTools: Overview 19-7
Using ADRCI on Exadata Storage Servers 19-8
imageinfo: Overview 19-9
imagehistory: Overview 19-10
OSWatcher: Overview 19-11
Quiz 19-12
Summary 19-14
Additional Resources 19-15
20 Backup and Recovery
Objectives 20-2
Backup and Recovery: Overview 20-3
Using RMAN with Database Machine 20-4
General Recommendations for RMAN 20-5
Disk-Based Backup Strategy 20-7
Disk-Based Backup Recommendations 20-8
Disk-Based Backup on Non-Exadata Storage 20-10
Tape-Based Backup Strategy 20-11
Tape-Based Backup Architecture 20-12
Tape-Based Backup Recommendations 20-13
Connecting the Media Server by Using Ethernet 20-15
Tape-Based Backup Recommendations 20-16
Hybrid Backup Strategy 20-17
Restore and Recovery Recommendations 20-18
Backup and Recovery of Database Machine Software 20-19
Quiz 20-20
Summary 20-21
Additional Resources 20-22
Practice 20 Overview: Using RMAN Optimizations for Database Machine 20-23
21 Exadata Database Machine Maintenance Tasks
Objectives 21-2
Database Machine Maintenance: Overview 21-3
Powering Database Machine Off and On 21-4
Safely Shutting Down a Single Exadata Storage Server
Replacing a Damaged Physical Disk 21-6
Replacing a Damaged Flash Card 21-8
Moving All Disks from One Cell to Another 21-9

xii

21-5

Using the Exadata Cell Software Rescue Procedure 21-10

Quiz 21-12
Summary 21-15
22 Patching Exadata Database Machine
Objectives 22-2
Patching and Updating: Overview 22-3
Maintaining Exadata Storage Server Software 22-4
Maintaining Database Server Software 22-6
Assisted Patching Using OPlan 22-7
Assisted Patching Using the DB Node Update Utility 22-8
Maintaining Other Software 22-9
Recommended Patching Process 22-10
Test System Recommendations 22-12
Quiz 22-13
Summary 22-14
Additional Resources 22-15
23 Exadata Database Machine Automated Support Ecosystem
Objectives 23-2
Auto Service Request: Overview 23-3
ASR Process 23-4
ASR Requirements 23-5
Configuring the ASR Manager 23-6
Configuring Exadata Database Machine for ASR 23-7
Activating ASR Assets 23-8
Verifying the ASR Configuration 23-9
Oracle Configuration Manager: Overview 23-10
Configuring Oracle Configuration Manager 23-11
Quiz 23-12
Summary 23-15
Additional Resources 23-16
A Exadata Database Machine and Oracle Database 12c
Objectives A-2
Introducing Oracle Database 12c: The Database for Cloud Computing A-3
Oracle Database 12c on Exadata Database Machine A-4
Summary A-5
B Exadata Release 11.2.3.3.0 New Features
Objectives B-2

xiii

Flash Cache Compression B-3

How Flash Cache Compression Works B-4
Enabling Flash Cache Compression B-5
Monitoring Flash Cache Compression B-6
Automatic Flash Caching for Table Scan Workloads B-8
Exadata Network Resource Management B-9
Automatic Hard Disk Scrubbing and Repair B-10
Cell Alert Summary B-11
Active-Active InfiniBand Connectivity B-12
Simplified Disk Controller BBU Replacement Procedure B-13
Safer Disk Removal B-14
Fast File Initialization B-15
Fast File Initialization Requirements B-16
Faster Disk Rebalance Operations B-17
Summary B-18
C Exadata Release 12.1.1.1.0 New Features
Objectives C-2
Complete Exadata Feature Support for Oracle Database 12c C-3
Using Exadata I/O Resource Management with Oracle Database 12c C-4
Support for Mixed Database Versions C-5
Cell to Cell Data Transfer C-6
Summary C-7
D Exadata X4 Hardware Update
Objectives D-2
Exadata Storage Server X4-2 Hardware Overview D-3
Exadata Storage Server X4-2 Configuration Options D-4
Exadata Database Machine X4-2 Database Server Hardware Overview D-5
Exadata Database Machine X3-8 Update Overview D-6
Exadata Rack-Level Changes D-7
Exadata Database Machine X4-2 and X3-8 Infrastructure Requirements D-8
Summary D-9
Additional Resources D-10

xiv