Case of the Unexplained Support Issue – Troubleshooting steps for diagnosing HFM, Workspace and FDM

#Kscope
Case of the Unexplained Support Issue
Charles Beyer
Troubleshooting steps for diagnosing HFM, Workspace, and FDM

#Kscope
About the Speaker
 15+ years professional development
experience in various languages/technologies
(6 years w/ Hyperion products)
 Significant experience reverse engineering
software
 Loves to help others solve problems
 Posts under beyerch2 on OTN Discussion
Boards

#Kscope
Before we Begin…..
 To expedite session, we will use both screen
caps and live demos
 ‘All-in-one’ Virtual Machine created for session,
available upon request*
 Feel free to ask questions

#Kscope
Session Goals
 Discover some new tools to be used for
troubleshooting (Oracle / Third Party)
 Acquire some new troubleshooting ideas

#Kscope
Challenges
 Responsible for a large amount of products!
● Foundation Services (OHS, Weblogic, Shared Services,
Workspace, EPMA, Calc Manager, LCM)
● HFM (HFM Web, Client, Smartview), FDM
● Reporting and Analysis (FR, IR, SQLPR, WA)
● Planning, Performance Scorecard, Strategic
Finance, Probability Cost, Disclosure Management,
Financial Close Management, Data Relationship
Management, Essbase, Crystal Ball, ……..
Hyperion Kitchen Sink.

#Kscope
Challenges (continued)
 Need to understand underlying technologies!
● OS Servers (Windows/Unix)
● Application Servers (Weblogic, Tomcat), Web Servers
(IIS, Apache, OHS)
● Authentication Options (AD, Native, Custom)
● Languages (Java, .NET, ASP Classic, JSP, VBScript)
● Communication Frameworks (DCOM, Web Services)
● Database (SQL Server, Oracle, Berkeley DB)
● Network Principles (Routing, Firewall, VPN)

#Kscope
 End Users!
● Quantity – Intermittent issues due to number of users
stressing system
● Internal vs. External
● Less control over environment
● More points of failure
● Remote troubleshooting more difficult
● User Experience
● New Users - More hands on, but easier problems.
● Power Users – Likes to experiment

#Kscope
 Internal Issues a.k.a. Politics!
● Separation of Duties
● Complicates troubleshooting efforts due to lack of access
● Delays resolution due to co-ordination efforts
● Restrictive Policies
● Could prevent/limit certain types of troubleshooting
● Network/HTTP sniffing
● No monitor programs on production

#Kscope
 Cryptic Error Messages!
But I didn’t click on a folder!

#Kscope
 Cryptic Error Messages!

#Kscope
 Limited OOTB (Out Of The Box) Tools!
● Mostly consists of Log Files
● System 11 has expanded diagnostic capability
● Support.Oracle.Com

#Kscope
Troubleshooting Strategy

#Kscope
Troubleshooting Strategy
Final Solution
Follow
Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem

#Kscope
Troubleshooting Strategy (continued)
 Clearly Define the Problem
● Ask the right questions
● Verify Information Where Possible
 Research, Prioritize, and Evaluate potential
solutions
● Use Initial Information and Resources to propose
possible causes/solutions
● Prioritize in order of likelihood
● Evaluate Potential Solutions
● Determine Final Solution or Escalate

#Kscope
 Close Out The Problem
● Document / Get Sign-Off
● Add Information To Internal Knowledgebase

#Kscope
 Notes
● Process Is Easier As You Learn More About The
System
● Development / Test Environment Makes This Easier
● Don’t Fear Leveraging Other Resources
● Practice, Practice, Practice

#Kscope
 Clearly Define The Problem – Answer the Who /
What / When / Where / How
● Who? (is affected)
● All Users or Select Group?
● If subset, is there commonality? (Geographic,
Hardware/Software, Data Provider)
● What ? (is happening / not happening)
● What was the attempted procedure?
● What has changed?

#Kscope
● When? (does the issue occur)
● Consistently? Intermittently? Pattern?
● Where ? (are failures occurring)
● Client Machine(s)?
● Database Server(s)?
● Web Server(s)?
● App Server(s)?

#Kscope
● How? (should the process work)
● Do you know how it should work?
● Important to understand expected operation to help
determine point of failure, probable cause, and solution.
● Research this if you are not sure!

#Kscope
solutions
● Iterative Process
● Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools =
Potential Solutions
● Prioritize Potential Solutions
● KISS / Occam’s Razor – Start with most likely / obvious
first!

#Kscope
solutions
● Evaluate Potential Solutions
● In Test/Dev if possible
● Promote to production after verifying your fix doesn’t cause
additional problems
● If you exhaust all possible causes, escalate. Don’t spin
wheels.
● Do not discount workarounds!

#Kscope
 Close Out The Problem
● Get Positive Confirmation That Issue Is Resolved
● Add Issue to Internal Knowledgebase For Future
Review
● SharePoint
● Track-It
● Shared Outlook Folder
● Share Information When Necessary

#Kscope
Troubleshooting Tools - Internal
 Knowledge Tools
● http://support.oracle.com – Knowledgebase, Patches &
Update, and “Communities”

#Kscope
Troubleshooting Tools – Internal (cont)
 Knowledge Tools
● OTN Discussion Forum (http://forums.oracle.com)

#Kscope
Troubleshooting Tools – Internal (cont)
 Knowledge Tools
● Product Documentation
● System 11.1.2.1 - http://docs.oracle.com/cd/E24674_01/index.htm
● System 9.3.x - http://docs.oracle.com/cd/E10530_01/welcome.html
 NOTE : These move periodically! Google
‘Hyperion documentation library’ if you cannot
find what you are looking for

#Kscope
Troubleshooting Tools - Internal

#Kscope
Troubleshooting Tools – Internal (cont.)
 Knowledge Tools
● Tips and Tricks from EPM System Infrastructure
Development
● Issues 1 to 72 -
http://www.oracle.com/technetwork/middleware/bi-
foundation/epm-tips-issues-1-72-427329.pdf
● Issues 73 and up -
http://www.oracle.com/technetwork/middleware/bi-
foundation/epm-tips-issues-73-up-399995.pdf
● NOTE: Constantly updating, so there will be more, etc.

#Kscope

#Kscope
 Data Gathering
● Log Files (see Table)
● HFMErrorLookup.exe – Resolves Error Codes to Text
Description
● System 11- <EPM ORACLE
HOME>EPMSystem11R1productsFinancialManagementConsultant
Utilities
● System 9 - <HYPERION HOME>FinancialManagementConsultant
Utilities

#Kscope
 Data Gathering
● HFMErrorLogViewer.exe – Utility to view HFM error logs

#Kscope
 Data Gathering
● FRConfig.cmd – FR utility that shows app Mem / CPU
utilization
● System 11 - <EPM_ORACLE_HOME>productsfinancialreportingbin

#Kscope
 Data Gathering
● ManageUserPOV.cmd – View/Edit POV for users

#Kscope
 Data Gathering
● Remote Diagnostic Agent (RDA) – Gathers system
information about environment. [See KB ID 1304885.1]
● oPatch [System 11 Only] – Install and list installed patches
● Location - <EPM ORACLE HOME>Opatch
● List Installed products = Opatch lsinventory
● NOTE: If you get an error running about OUI Home,
make sure environment variable pointing to EPM
ORACLE HOME which is most like
OracleMiddlewareEPMSystem11R1

#Kscope
 Data Gathering
● Comps.xml [System 11 Only] – List of all installed components
on machine
● <EPM ORACLE HOME>inventoryContentsXML

#Kscope
 Data Gathering
● Epmsys_registry.bat [System 11 Only] – Generates a detailed
report of everything registered in Shared Services Registry.
● Program - <MIDDLEWARE HOME>user_projectsepmsystem1bin
● Reports - <MIDDLEWARE HOME>user_projectsdiagnosticsreportsregistry.html

#Kscope
 Data Gathering
● ‘About Us’ pages – Shows currently installed programs and
specific version info. (Applies to Workspace, FDM, Shared
Service web pages, etc.)
● Click on the ‘Show Details’ button to see version info per product
● NOTE : This version info *MAY NOT* change even if you have
installed patches!

#Kscope
 Data Gathering
● Informational Web Pages
● http://hostname:port/workspace/browse/configInfo - Returns
detailed information about products installed and links to
other configuration files

#Kscope
 Data Gathering
● http://<shared_services_server>:28080/interop/frameowrk/g
etCSSConfigFile – Returns configuration information for
Shared Services
● System 9 – Port Number is 58080
● System 11 – You may need to administratively enable this
page for it to be visible!

#Kscope
 Data Gathering
● http://<hfm_web_server>:80/hfm – Returns ‘hfm’ if HFM is
running properly and integrated with workspace. (else
opens HFM Web)

#Kscope
 Utility Programs
● Validate.bat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation.
Generates detailed report.
● Can be run on product by product basis or all at once
● Program - <MIDDLEWARE HOME>user_projectsepmsystem1bin
● Report - <MIDDLEWARE
HOME>user_projectsepmsystem1diagnosticsreports
● NOTE : If you are building an environment, you should be
running this after your complete install/config!

#Kscope
 Utility Programs
● DB_Verify / DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues.

#Kscope
Troubleshooting Tools – 3rd Party
 Data Gathering
● WebEx / GoTo Meeting / Log Me In / VNC / Lync –
Remote Access Software. A must for remote
support.
● Windows Event Log

#Kscope
 Basic Connectivity Tools
● Ping / Tracert – Simple connectivity test tools

#Kscope
 Basic Connectivity Tools
● Telnet – Connectivity tool for any port

#Kscope
Troubleshooting Tools – 3rd Party (cont.)
 Advanced Connectivity Tools
● Fiddler – Web proxy that allows you to monitor (and
simulate) HTTP/HTTPS traffic.
http://fiddler2.com/fiddler2/
● Great Tool for HTTP/HTTPS traffic
● Warning : You can sniff / decode HTTPS traffic
● Warning : Some programs may send logins /
passwords in plain text.

#Kscope

#Kscope
 Advanced Connectivity Tools
● Wireshark – Network traffic analyzer –
http://www.wireshark.org
● Network packet analyzer
● Monitors ALL network traffic, not just HTTP/HTTPS
● Warning : Can intercept all types of traffic such as:
● Streaming Audio / Video
● VoiP
● Types of Encrypted Data
● Plain Text User Name / Password

#Kscope
 Server Tools
● Service Manager (services.msc) – View services to see
which are running
● Process Monitor – Allows you to monitor processes’ file
use, registry access, memory utilization, etc.
http://technet.microsoft.com/en-us/sysinternals/bb896645

#Kscope
 Server Tools
● SQL Profiler – Allows you to monitor database activity in
SQL Server – http://msdn.microsoft.com/en-
us/library/ms181091.aspx
● [Oracle Enterprise Manager has similar functionality for Oracle
databases]

#Kscope
Troubleshooting Problems

#Kscope
Migrate VM Server Error

#Kscope
Migrate VM Server Error
 To aid with session, built VM/Virtual PC System
11 “All In One” Environment

#Kscope
Migrate VM Server Error (cont)
 After copying VM to another machine…

#Kscope
 But not everything was dead… VM to another
machine…

#Kscope
 IDEAS??
●What changed?

#Kscope
 Lets get some machine info

#Kscope
 Lets get some config (epmsys_registry.bat)

#Kscope
 IDEAS??
●Anyone see anything suspect so far?
●I did, and have a suspicion, but need
more information to confirm.

#Kscope
 Lets do some sniffing…(Wireshark)

#Kscope
 IDEAS??
●Why would there be calls to an IP
address that does not belong to the
machine?

#Kscope
 Lets do some sniffing…(WinGrep)

#Kscope
 Lets do some more sniffing…(SQLLocator)

#Kscope
 IDEAS??
●Looks pretty reasonable that hardcode
IP addresses are breaking things since
the machine no longer has the IP
address buried in config / database!

#Kscope
 How To Fix
●Reinstall?
●Reconfigure?
●Manually Search and Replace?

#Kscope
Manual Repair!
(Because we can)

#Kscope
 Lets fix the SQL Damage First

#Kscope
 Lets fix config file damage
● Content.XML
● Config.XML
● GlobalContext.XML
● httpd.conf
● mod_wl_ohs.conf
● Server.xml
 Warning: In some places you DO NOT want to
replace the hardcoded IP address as it is part
of a Unique Identifier. That could be bad.

#Kscope

#Kscope
 Reboot

#Kscope
 Mission Accomplished!

#Kscope
Questions?
 If you have any questions with the sample code
or presentation, feel free to ask/contact me
● charles@charlescbeyer.com

Case of the Unexplained Support Issue – Troubleshooting steps for diagnosing HFM, Workspace and FDM

More Related Content

Case of the Unexplained Support Issue – Troubleshooting steps for diagnosing HFM, Workspace and FDM

Editor's Notes