Platform Support Plan

System Name: Database SQL Support Plan

Document ID:

Document Revision: 1.5

Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Approvals
TECHNICAL UNIT APPROVAL:
By affixing their signature, the individual below acknowledges that they have reviewed and approve the technical
contents of the Infrastructure Support Plan and attest that they are accurate.
Merck Wave Lead Name
{See Appended Electronic Signature Page}
Fletcher James, SQL Server, Database Services
BUSINESS UNIT APPROVAL:

By affixing their signature, the individual below acknowledges that they have reviewed and approve the technical
contents of the Infrastructure Support Plan and attest that they are accurate.
Merck Tower Lead
{See Appended Electronic Signature Page}
Chuck Denecke, Database Services

Revision History
ISSUE # DATE PERSON CHANGE
1.0 2-Dec-2013 Parameswaran, Sangeeth First issuance of document.
Kumar
1.1 5-Dec-2013 Parameswaran, Sangeeth Formatting changes done
Kumar
1.2 29-Apr-2014 Jayapalreddy Vajrala Added Pre-Auth, SQLSafe and SQLDM WI’s
1.3 30-Sep-2014 Jayapalreddy Vajrala Added WI’s for AWS RDS SQL Instances
1.3 30-Sep-2014 Jayapalreddy Vajrala Added WI’s for On-Boarding SQL Server
1.4 16-Oct-2014 Jayapalreddy Vajrala IQOQ/QG Instructions
1.5 10-Aug-2015 Linda Han Updated in areas in New Builds, IQOQ QG processes;
Idera SQL DM monitoring configuration; CV Backup
configuration, migration; On-boarding an existing DB,
etc
1.6 21-Aug-2015 Linda Han Added SQL Server Class A DR Plan document link

Page 2 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

TABLE OF CONTENTS

TABLE OF CONTENTS............................................................................................................................................................3
1 INTRODUCTION.............................................................................................................................................................5
1.1 PURPOSE..........................................................................................................................................................................5
1.2 SCOPE..............................................................................................................................................................................5
1.3 AMENDMENTS TO THE SUPPORT PLAN............................................................................................................................5
2 SUPPORT STRATEGY AND ENVIRONMENT..........................................................................................................5
2.1 SUPPORT OBJECTIVE.......................................................................................................................................................5
2.2 SUPPORT STRATEGY........................................................................................................................................................5
2.3 SUPPORT ENVIRONMENT.................................................................................................................................................6
3 ABBREVIATIONS AND ACRONYMS..........................................................................................................................7
4 ROLES AND RESPONSIBILITIES................................................................................................................................7
5 TOOLS, TECHNIQUES AND METHODS....................................................................................................................7
5.1 SUPPORT TOOLS..............................................................................................................................................................7
5.2 VENDOR SUPPORT DETAIL..............................................................................................................................................8
5.3 COMMUNICATION METHOD.............................................................................................................................................9
6 IT SERVICE MANAGEMENT PROCEDURES...........................................................................................................9
6.1 SERVICE REQUEST MANAGEMENT..................................................................................................................................9
6.2 INCIDENT MANAGEMENT................................................................................................................................................9
6.3 PROBLEM MANAGEMENT................................................................................................................................................9
6.4 CONFIGURATION MANAGEMENT...................................................................................................................................10
6.5 CHANGE MANAGEMENT................................................................................................................................................10
6.6 SERVICE KNOWLEDGE MANAGEMENT..........................................................................................................................10
6.7 IT CHANGE MANAGEMENT PROCESS – PRE-AUTHORIZED CHANGES..........................................................................10
7 SECURITY AND ACCOUNT MANAGEMENT.........................................................................................................12
8 PLATFORM SUPPORT PROCEDURES....................................................................................................................13
8.1 SQL DATABASE SERVER STANDARDS..........................................................................................................................13
8.2 SQL SERVER OPERATIONAL ACTIVITIES......................................................................................................................14
8.3 STARTING, STOPPING, AND RESTARTING SQL SERVER SERVICES...............................................................................15
8.4 SQL SERVER SERVICE STARTUP OPTIONS....................................................................................................................16
8.5 CHANGING SQL SERVER CONFIGURATION SETTINGS..................................................................................................16
8.6 APPLYING SQL SERVER SERVICE PACKS.....................................................................................................................16
8.7 CHANGING SQL SERVER PORT NUMBER......................................................................................................................17
8.8 SQL SERVER AGENT JOB CREATION............................................................................................................................17
8.9 SQL SERVER LINKED SERVER CREATION....................................................................................................................17
8.10 ENABLING TRACE FILES...........................................................................................................................................17
8.11 CREATE DATABASE TRIGGER...................................................................................................................................18
8.12 IDERA........................................................................................................................................................................18
8.13 ON-BOARDING AN EXISTING SQL SERVER INSTANCE.............................................................................................19
8.14 SQL SERVER DATABASE MANAGEMENT.................................................................................................................19
8.15 SQL SERVER INSTANCE DECOMMISSION.................................................................................................................20
8.16 SQL SERVER STORAGE MANAGEMENT...................................................................................................................20

Page 3 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

8.17 IQOQ/QUALITY GATE REPORTS..............................................................................................................................20

9 PLATFORM SUPPORT PROCEDURES – AWS RDS SQL INSTANCE...............................................................20
9.1 AWS RDS SQL INSTANCES IN MERCK ENVIRONMENT...............................................................................................20
9.2 AWS RDS SQL INSTANCE PROVISIONING...................................................................................................................21
9.3 ACCESSING RDS SQL INSTANCE..................................................................................................................................21
9.4 AWS RDS SQL INSTANCE RETIREMENT.....................................................................................................................21
9.5 IMPORT/EXPORT DATABASE DATA...............................................................................................................................21
9.6 RDS DATABASE RECOVERY.........................................................................................................................................21
9.7 RDS MANUAL SNAPSHOT BACKUP..............................................................................................................................21
9.8 MODIFY DB PARAMETER GROUP.................................................................................................................................22
9.9 CHANGE RDS INSTANCE TYPE AND UPDATE TAGS.......................................................................................................22
9.10 RDS MONITORING....................................................................................................................................................22
9.11 RDS INSTANCE IQOQ..............................................................................................................................................22
9.12 OTHER AMAZON RDS DATABASE ACTIVITY INFORMATION...................................................................................22
10 BACKUP AND RECOVERY.........................................................................................................................................23
10.1 TSM TDPSQL BACKUP................................................................................................................................................23
10.2 IDERA SQL SAFE BACKUP.............................................................................................................................................24
10.3 SIMPANA COMMVAULT BACKUP....................................................................................................................................24
10.4 CHECKING AND MONITORING DAILY SQL SERVER DATABASE BACKUPS.......................................................................24
11 SQL SERVER MONITORING AND ALERTING......................................................................................................25
11.1 ALARMPOINT............................................................................................................................................................25
11.2 ALERT AND EMAIL NOTIFICATION CREATION.........................................................................................................25
11.3 ALERT NOTIFICATION RESPONSE.............................................................................................................................25
11.4 GENERAL DATABASE MAIL......................................................................................................................................25
11.5 LOGIN FAILURE REPORT...........................................................................................................................................26
11.6 IDERA SQL DIAGNOSTIC MANAGER........................................................................................................................26
12 DATABASE SOX AUDITING & REPORTING.........................................................................................................26
13 SQL SERVER BUILD....................................................................................................................................................27
13.1 DATABASE ACQUISITION PROCESS...........................................................................................................................27
13.2 DATABASE PROFILE FORM.......................................................................................................................................27
13.3 DEFAULT DATABASE PARAMETERS.........................................................................................................................27
13.4 APEX DATABASE REQUEST PROCESS.....................................................................................................................27
13.5 CREATING SERVICE ACCOUNT (NON-PERSON ACCOUNT).......................................................................................28
14 DISASTER RECOVERY/BCP......................................................................................................................................29
15 SLA MANAGEMENT....................................................................................................................................................31
15.1 SLA..........................................................................................................................................................................31
15.2 SHIFT TIMINGS..........................................................................................................................................................31
16 SCHEDULED ACTIVITIES..........................................................................................................................................32
16.1 DAILY ACTIVITIES....................................................................................................................................................32
16.2 WEEKLY ACTIVITIES................................................................................................................................................33
16.3 YEARLY ACTIVITIES.................................................................................................................................................34
16.4 AUDIT REPORT.........................................................................................................................................................34
16.5 AD-HOC ACTIVITIES.................................................................................................................................................34
17 TRAINING.......................................................................................................................................................................34

Page 4 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

18 REFERENCES................................................................................................................................................................35
19 RESOURCES...................................................................................................................................................................35
20 APPENDIXES..................................................................................................................................................................36
APPENDIX – A: AMAZON RDS – INFORMATIONAL GUIDELINES FOR USE AT MERCK...........................................................36

Page 5 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

1 Introduction

1.1 Purpose
The purpose of this document is to define the roles, responsibilities and procedures involved in managing the SQL
Server Database Platform.

1.2 Scope
The scope of this document includes the necessary information to manage the SQL Server Database Services
Platform. The content of this document will provide engineering and operational guidelines and the basic
principles that govern the platform.

1.3 Amendments to the Support Plan

The content material contained within this document shall be in effective until the document is retired. Minimally,
the content of the document will be examined by platform subject matter experts and management on an annually
basis to ensure the content is reflective of current support processes and strategies.
Amendments to this support plan will be accomplished through the revision of this document with signature
approval by all original signatories, their delegates, or functional equivalents. A history of approved issues of this
document will be reflected in the Document Issue History section.

2 Support Strategy and Environment

2.1 Support Objective

The Platform Service provides server environments to host Merck applications. Specific support objectives are
identified following Operational Level Agreements.
 GISLM-OLA-GEN-001413 – Database Services Operational Level Agreement

2.2 Support Strategy

DATABASE SERVICES Platform support is divided into three levels. Each level is supported by the HCL Managed
Service.
The different levels of support allow the SQL Server Database Platform to be supported 24x7.
Issues to the platform are reported as follow:
 Data Center Technicians raise Remedy incidents cases about issues related to the
DATABASE_SERVICES_RESOURCES or CFDBA Resources reported by various sources not
limited to OVO events, application groups, etc.
 Application groups submit helpdesk cases about issues they are experiencing with the platform.
 Monitoring tools generate alerts based on some pre-defined threshold. These events are managed by
DATABASE_SERVICES_RESOURCES or CFDBA Resources using the incident management
process.
All requests for SQL Server Database Support must first be initiated by a Remedy Work Order, Service Request,
or Incident. The Requestor should include a detailed description of the request or problem, the database name if
applicable, and the server’s name. The Remedy ticket should initially be submitted to the
“DATABASE_SERVICES_RESOURCES” Remedy Group for initial service management.
HCL actively monitors these queues on a 24x7 basis to respond, manage, and complete the request according to
the SLA for the requests’ priority.

Page 6 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

2.2.1 Queues Names *

There are three Remedy Provider Groups associated with the HCL team, one for Shared Services, CFDBA Group,
and SQL Server Group:
 CFDBA Queue Name: “Database DBA Resources”
 Shared Services Queue Name: “Database Services Resources”
 SQL Server Queue Name: “Database SQL Server Resources”

There are 2 Merck Groups and 3 Queues from where most of the cases will come:
 Merck CFDBA Queues:
o Database HH
o Database Corp/MRL/MMD

Note: E-mails/IMs/Phone Calls will be responded to for Emergency Requests, but a Remedy ticket must be
submitted shortly thereafter.

2.3 Support Environment

A majority of the supported SQL Database Platform servers are located in the following global and regional
datacenters: Charlotte, Whitehouse Station, Brussels and Singapore. Asset information is tracked in Remedy
(CMDB). Physical and virtual servers are located in Merck locations globally.
The SQL Database Platform is supported by the following products:
 SQL Server configuration Enterprise Manager is used to start and stop the SQL server services.
 TSM Backup is a tool used for backing up the operating system, database and application data.
 Remedy is used for change, asset, and knowledge, problem and incident management.

2.3.1 Contact Information

All support requests must have a support case. A Consumer can call the Help Desk or the Charlotte Technology
Center (CTC) Command Center and a case will be created. If they cannot provide immediate support, they will
contact the Service Provider for assistance. Requests that are not initiated by the CTC Command Center will be
responded to by the Service Provider according to the support model detailed below.
If issues arise that cannot be resolved between the Client and the Support Lead, the following escalation procedure
should be followed.
 Contact the Help Desk and ask for your call to be routed to the Command Center.
 Ask the Command Center for details on escalation chains, which can be found in My Support Center
and AlarmPoint.
Automated alerts are generated by the different monitoring tools and reported through Alarmpoint. Alarmpoint is
also used by application groups and data center technicians to page a DBA about a problem or incident.

Page 7 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

3 Abbreviations and Acronyms

Acronym - Term Definition
Active Roles Tool used to manage Active Directory.
Asset Manager Person(s) responsible for the assignment of node names.
CI Stewards Those responsible for Remedy Configuration Item CMDB Updates.
EVOC Merck Project Vocabulary Database
Hardware The mechanical and electronic physical components that make up a computer system.
IQOQ Installation Qualification / Operational Qualification
KM Remedy Knowledge Management
Department responsible for the proactive real-time monitoring of server health, utilization,
Monitoring Services
availability of business applications and IT Infrastructure.
Monitoring Services distributed client-server software solution designed to help System
HP OpenView Operations
Administrators detect, resolve and prevent problems occurring in networks, platform systems and
for Windows (OVOW)
applications.
BMC Software used for asset and change management, from procurement through retirement,
Remedy
including change control, version management and configuration status accounting.
TSM Tivoli Storage Manager

4 Roles and Responsibilities

Role Role Responsibility
Database Administrator Build and Manage the SQL Database Platform
Shared Services (SS) DBA Shared Services Team performs the infrastructure activities for the databases.
Mostly, they get requests from the Client Facing DBAs (CFBDA) for the activities
to be performed. Shared Service Team is made up of Engineering DBAs
Client Facing DBA (CFBDA) Client facing DBAs coordinate with application team and business users to
understand the application requirement and their needs. CFDBA collects all the
information and provide the support.
Managed Services Operations DBAs that perform routine tasks and operational activities.

5 Tools, Techniques and Methods

5.1 Support Tools

5.1.1 Alarmpoint for escalation and notification

AlarmPoint is an alerting tool that interacts with monitoring software to provide the following functions to Merck:
 Alerting relevant personnel to critical IT events, and
 Escalating and acknowledging the events as required throughout the business enterprise --
application management, network / system monitoring, business continuity and information security.
Most production database servers are being monitored using the HP Openview tool, SPI. The EMS team is
responsible to configure the tool on the servers. Whenever an incident happens, it sends an alert to corresponding
Page 8 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

team (in case of database servers, goes to “Database Operations Resources”

(1datoperr@northamerica.msx.merck.com ) or cfdba resources (1CFDBAR@NorthAmerica.msx.merck.com). The
team is responsible to respond to the alert within 15 minutes failing which it will get escalated to higher level.

5.1.2 My Support Center (Remedy)

BMC Software’s Remedy application is an enterprise wide, web based ticketing system for Business Service
Management. The following Remedy modules support Merck’s ITIL-based Service Management Processes:
 Change Management
 Incident Management
 Problem Management
 Configuration Management
 Service Level Management
 Knowledge Management
 Service Request Management

5.1.3 Idera Admin Tool

This tool is being used to monitor SQL Server database backups. Complete detail is given in Daily Activity.

5.1.4 TSM Backup Agent

This tool is in use by Backup team to take backup of SQL Server databases.

5.1.5 Backup Dashboard

This is in-house developed application. The database of this application is in Oracle and backup status data is
uploaded to this application from where report is generated. This has been discussed in detail in backup section of
daily activity.

5.1.6 SPI
This is HP OpenView tool for monitoring SQL Server parameters. It has been discussed in details in monitoring
section.

5.2 Vendor Support Detail

5.2.1 SQL Support

Merck has premium support from Microsoft. Account information is available with Onsite DBA / Merck DBA. If
required to open a case with Microsoft, HCL will consult with Merck DBA over the issue/requirement before
opening a case with Microsoft. Please use the below WI link for How to contact Microsoft Technical Support.

http://webtop.merck.com/webtop/drl/objectId/090030f980f1518c

Page 9 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

5.2.2 Idera Admin Tool

Merck has 10 user licenses of Idera Admin tool to be used by DBAs (HCL and Merck). If required, support is
available from Idera. Support team can be contacted with reference of License detail in use. Here are some
licenses allocated to HCL.

STD-70NQU-N12FA-NGD9-783H
STD-PJ94K-X1242-47MN-74Q8
STD-SA612-WJ1K2-P7T6-75CE
STD-H3T0N-124PN-A34K-75S7

5.3 Communication Method

Scheduled and unscheduled service disruptions will be communicated to clients by email notifications from My
Support Center. Scheduled downtime will be communicated at least one week in advance. Unscheduled downtime
will be communicated though the Command Center as soon as is practicable.

6 IT Service Management Procedures

6.1 Service Request Management

Datacenter Technician submits a service request/ Case to “Database Services Resources " inbox. Tier2 SA
monitors the support Group “Database Services Resources “inbox 24x7. Tier 2 SA investigates the case and if
within scope resolve incident using KB articles. If not then escalate incident to Tier3 On-Call DBA.
The procedure to support Service Request Management is as follows
 SDLC-SOP-05-08: Global Services (GS) Service Request Fulfillment

6.2 Incident Management

Incident Management is the process responsible for managing the lifecycle of all incidents. Status of all incidents can be
obtained by calling the Help Desk or the Command Center.
Whenever an incident occurs, an alert is generated through the monitoring tools. Once alert is received, it is registered in
remedy tool by user or support team.
Once the incident is registered, a ticket is created and assigned to the concerned support team (here Database Team). The
person who is assigned the ticket to work upon is called assignee. Assignee will be owner of the ticket until it is resolved or
it is transferred to other person.
The remedy group name for Database Support team is “Database Services Resources”. The ticket is further assigned to
specific Database Team member who works on the incident ticket. The time between ticket assigned to group and the
assignation to assignee is measured as response time. Database Team must be careful about the ticket response time to meet
the agreed SLA.
The procedure to support Incident Management is as follows
 SDLC-SOP-05-01: IT Incident Management

6.3 Problem Management

Database Administrators are trained on how to report and resolve problems. Problems are tracked using My
Support Center.
There are two scenarios where we need to review the incident in more depth. 1) When a major outage occurred
and 2) when there is repetitive incident. In both these scenarios, incident should be investigated in detail and a
problem ticket is created.

Page 10 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

To resolve problem ticket, root cause of the incident needs to be identified. Generally 3 times “Why” is asked for
the reason of incident and the root cause is identified. However, in some case, you may need to dig it further.

Once the root cause is identified, it mostly comes with some recommendations / change in environment which is
then taken care of by Change Management system separately. The Problem ticket can be closed once the root
cause is identified
The procedure to support Problem Management is as follows:
 SDLC-SOP-05-02: IT Problem Management

6.4 Configuration Management

Configuration Management is the process that maintains information about Configuration Items required to deliver
an IT Service, including their Relationships. Its primary objective is to provide accurate data to all IT Service
Management Processes when and where it is needed
The procedure to support Configuration Management is as follows:
 ITSM-SOP-GEN-000068 - GTS Configuration Management Process

6.5 Change Management

Change Management is the process responsible for controlling the lifecycle of all changes to the platform. It
manages the change processes involving hardware, communications equipment and software, system software and
all processes, procedures and documentation associated with support and maintenance of live systems, processes
and services. All major changes to the system will be coordinated through the Change Management Process.
The procedures to support Change Management are as follows:
 ITSOP-0015 – IT Change Management Process – Pre-Authorized Changes
 ITSOP-0020 - IT Change Management Process – Planned and Unplanned Changes

6.6 Service Knowledge Management

Knowledge Management is the process that governs the gathering, analysing, storing and sharing of knowledge
information. Its primary objective is to improve efficiency by reducing the need to rediscover knowledge. All
operational tasks should be entered into a KM article. This information in KM articles is used to manage the
platform by each tier group; therefore, it is important that such information is up to date and accurate.
The procedure to support Knowledge Management is as follows:
 SDLC-SOP-05-07: Administration and Control of Service Knowledge Management Articles for Global
Services (GS)

6.7 IT Change Management Process – Pre-Authorized Changes

The change management process ensures that standardized, consistent methods are used for efficient and prompt handling of
all changes to the Merck IT environment, in order to minimize the impact of change-related incidents upon service quality,
and consequently improve the day-to-day operations of the organization. IT Change Management processes must not impact
the regulatory business processes and products. This procedure defines the three classes of Pre-Authorized Change
sanctioned by the ETAS Change Management Authority and are contained in a Common Catalog.

 Standard Change - A Pre-Authorized Change that is low risk, relatively common and may follow a Procedure
or Work Instruction. Pre-Approval and Post Approval are not required

Page 11 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

 Pre-Approved Change - A Pre-Authorized planned change that does not require Pre-Approval but requires
Post Approval
 Administrative Change - A Pre-Authorized Change that can be implemented without the creation of an RFC
and without any signature events for more details please refer “IT Change Management Process - Pre-
Authorized Changes”.

For more details please refer “IT Change Management Process - Pre-Authorized Changes”.

6.7.1 Pre-Authorized CRQ Implementation Instructions

6.7.1.1 Remove Old Disk Backup files
The document “Remove old disk backup files” at below location discusses about how to
remove old disk backup files in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef45bb

6.7.1.2 Enable database mail/mail profile

The document “Enable database mail/mail profile” at below location discusses about how to
enable database mail/mail profile in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef45b3

6.7.1.3 Shrink Data file

The document “Shrinking Database Log Files” at below location discusses about how to shrink
Database Log Files in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef45bc

6.7.1.4 Recover Suspect Database

The document “Recovering Database from Suspect Mode” at below location discusses about
how to recover Databases from Suspect Mode in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef45ba

6.7.1.5 Update SQL Server Services Accounts with new password credentials
The document “Update SQL Server Services Accounts with new password credentials” at below
location discusses about how to Update SQL Server Services Accounts with new password credentials
in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef45bd

6.7.2 General Pre-Authorized CRQ Implementation Instructions  

The document “WI - General Pre-Authorized RFC's for SQL Server” at below location provides
detailed WI’s for General Pre-Authorized CRQ Implementation for SQL Server
http://webtop.merck.com/webtop/drl/objectId/090030f980ef8e48

Page 12 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

7 Security and Account Management

Security and Account Management is the process responsible for controlling access to applications and systems
within the Merck IT environment; ensuring only appropriate access and privileges are provided and maintained.
Corporate policy 26 and its supporting Functional Policies outline the mandatory security requirements and elements
for enforcing Account Access and Security in the Merck IT environment.

The core procedure to support Security and Account Management is as follows

 ITSOP-0014: Account and Access Management for IRM&C Supported Applications and Systems
To be able to access data from a database, a user must pass through two stages of authentication: one at the SQL Server level
and the other at the database level. These two stages are implemented using logins names and user accounts, respectively. A
valid login is required to connect to SQL Server and a valid user account is required to access a database.

Merck Manager Approval is required for Account Administration in all Environments.

7.1.1 SQL Server Login

Windows authentication is the recommended security mode, as it is more secure and you don’t have to send login
names and passwords over the network. You should avoid using SQL Server authentication unless required by the
application.

7.1.2 Each DBA Should Have their Own ISID Account Added to the Windows Groups
NORTHAMERICA\USHHSQLSRV.ClientFacing.DBA
NORTHAMERICA\USMMDMCCCORP.SQLSRV.CFDBA
NORTHAMERICA\USSQLSRV.DBPlatform.DBA

This group should be granted Local Admin of the Windows servers and also added to the SQL Server Instance as SysAdmin.

7.1.3 SQL Server Service Account

Each SQL Server Instance should run under a service account requested during Installation. Here, we use dedicated service
account for every instance, however, it is not mandatory.
We use standard naming convention (Srvs<xxxx> where <xxxx> is the server number for which the SQL Server is
running on) for using service account for SQL Servers. As detailed information about choosing / getting created
and password reset has been discussed in the SQL installation document.

7.1.4 Temporary Grant of SYSADMIN Privilege

The document provides compete guideline to grant temporary SYSADMIN privilege on SQL Server in Merck
Environment based on below scenarios.
 To support the execution of a vendor installation/upgrade procedure.
Please refer the work instruction to grant temporary SYSADMIN privilege at link below:
http://webtop.merck.com/webtop/drl.html?objectId=090030f980f0e17a

Page 13 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

7.1.5 Backup Group

TSM tool is used for backup of SQL Servers. To backup databases from SQL Server, TSM Tool should be able to
connect to the SQL instances and should have db_backup_operator role at database level. However, this would be
overhead to manage this permission on thousands of database server, SysAdmin permission has been granted to
NORTHAMERICA\ESR_BACKUPADMINS account which is used to connect to the Servers by TSM tool for
backup / restore purpose.

7.1.6 Request TSM Backup Configuration

For new SQL instance, it is clearly mentioned in the WI- document to raise a request with Backup team to
configure backup on the SQL instance.
(Please refer to the document Installation checks and guide.doc in Documentum folder.)

Retention period:

All Production Database Backups are retained for 30 days. Rest environments backups are kept for 7 days.

7.1.7 Local Backup Job in SQL Server Agent

A Local Backup may be needed for an exception cases when TSM is not an option.

However, this needs to be discussed and approved by Merck DBA before we proceed with local backup. It also needs a
planning how the backup would be retained for the defined retention period and what would be recovery procedure in case
server crashes or not available.
Please refer the work instruction to setup local backup at link below:
Setup Native Backup Job in SQL Server Agent

7.1.8 Backup Exemption

Application Teams are responsible to complete the Backup Exemption Form to exclude the Database Operations
Team from performing database backup monitoring and database backup and recovery operations. The detail of
those databases with backup exemption can be found in “Backup Exemption Form”.

Completed Backup Exemptions are stored in the following location:

http://teamspace.merck.com/sites/InfSolEng/ISEDBEMS/ISEDB/Shared%20Documents/Forms/AllItems.aspx?
RootFolder=%2Fsites%2FInfSolEng%2FISEDBEMS%2FISEDB%2FShared%20Documents%2FDatabase
%20Operations%2FBackup
%20Exemptions&FolderCTID=0x012000B38AE7D17346FC44BBCAC2B0A618BBBF&View={A7414DE2-C79B-4E2A-
BCED-42447188F2E1}

8 Platform Support Procedures

8.1 SQL Database Server Standards

SQL Server 2008 R2 and SQL Server 2012 are current standard release.

Page 14 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

8.1.1 Naming Standard

Below is the naming convention being followed at Merck by Windows Team.
For example: let’s consider below name of a server.
US –CT- AP-P- XXXXXX
There are 5 parts in the server names as shown below:
1. First 2 Character (US/BR) - Country name (France – FR, Brussels – BR).
2. Next 2 Characters (CT/WS) - Site Name/ Location Name (for this example CTC is the place. If the place is
WHS then these two letters would be WS).
3. Next 2 Characters (AP) - This denotes the purpose of the server (AP- Application server TS – Terminal server)
SQL server will fall in to AP.
4. Next 1 Character (P/T/D) – This refers to the Role of the server (P- Production T- Test, D- Development).
5. Next 6 Characters (XXXXXX) – Sequence of 6 numbers (It should not conflict with any currently available
numbers).
We have two KM articles for 2k, 2k3 and 2k8 windows server naming convention.
Please check the below KM articles.
1. KBA00002665: What is the Windows 2008 Server naming standard for physical and virtual servers?
KBA00002663: Windows Server - Windows 2000 & 2003 Server Naming Standards for physical and virtual
servers.

If we create a virtual resource group name on cluster server, we add suffix S1/S2/S3 for each resource group name
(on cluster server) while the instance name follow the naming convention, like environment. Where environment =
Prod / Test / Dev. etc. and XX is serial number from 01 to 99.

8.1.2 Microsoft SQL Server Standards

The document “Microsoft SQL Server Engineering Standard” provides guideline to implement industry best
practice with Merck default installation /deployment.

8.1.3 SQL Server Roadmap

The document “DB Platform Standards 2013” discusses the roadmap of technology being used. Currently QL
Server 2008 and SQL Server 2012 are the technology being used.
Note: Currently, work in progress to modernize the older versions SQL Server. The modernization task being
handled by Modernization Team and is out of scope for this document.

8.1.4 SQL Server Security Standards

The document “SQL Server 2005 2008 Technical Security Standards v1” at below location discusses about standard security
practice in Merck Environment.
documentum\Docbases\gdmsprd\IS\Global Infrastructure\GDS\Team Folders\Database Services\Standards

Page 15 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

8.2 SQL Server Operational Activities

8.2.1 SQL Server 2005/2008/2012 Configuration Manager

SQL Server Configuration Manager combines the functionality of the following SQL Server tools: Server
Network Utility, Client Network Utility, and Service Manager. SQL Server Configuration Manager allows you to
stop, start, or pause various SQL Server services.

8.2.2 Accessing SQL Server Configuration Manager

On the Start menu, point to All Programs, point to Microsoft SQL Server 2008 R2, point to Configuration Tools,
and then click SQL Server Configuration Manager.

8.2.3 Starting and Restarting Services

You can start or restart an instance of SQL Server automatically or manually. Both the automatic and manual
methods start an instance of SQL Server as a Windows 2000 or Windows Server 2003 service. You can start SQL
Server with a variety of options such as single-user mode, minimal configuration mode, or with trace flags. You
can start SQL Server by starting the SQL Server service from the command prompt. Automatic is the standard for
all non-clustered services.

8.3 Starting, Stopping, and Restarting SQL Server Services

8.3.1 Starting the SQL Server Service Automatically

Configure the service to start automatically at start up with SQL Server Configuration Manager. However, for
cluster servers, SQL Services, SQL Agent Services are in manual mode. It will be running only on active node
according to the resource being online on there.

8.3.2 Starting SQL Server Service Manually

You can manually start an instance of Microsoft SQL Server or SQL Server Agent using the following methods:

Method Description
SQL Server Configuration Manager Start, pause, resume and stop an instance of a local SQL Server or SQL
Server Agent service.
Command prompt Start an instance of SQL Server or SQL Server Agent service from a
command prompt by the net start command or by running sqlservr.exe.

Use sqlservr.exe to start SQL Server from a command prompt only to troubleshoot SQL Server. Before you start an
instance of SQL Server using sqlservr.exe from a command prompt (independent of SQL Server Configuration Manager),
consider the following:

 SQL Server runs in the security context of the user, not the security context of the account assigned to run SQL
Server during setup.
 All system messages appear in the window used to start an instance of SQL Server.

Page 16 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

 You cannot pause, stop or resume an instance of SQL Server as a Microsoft Windows service using SQL Server
Management Studio, SQL Server Configuration Manager, the Services application in Control Panel, or any net
commands (for example, net start, net pause, net continue, and net stop).
 You must shut down the instance of SQL Server before logging off Windows.
 Any manually entered command prompt options take precedence over the default command prompt options written
to the Windows registry by SQL Server Setup.
 Minimizing the Command Prompt window will cause Windows to remove nearly all resources from SQL Server.
This is normal, because SQL Server is not designed to operate from a minimized Command Prompt window.
 SQL Server Configuration Manager and SQL Server Management Studio show the service as stopped.

Page 17 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

8.3.3 Starting SQL Server in Single-User Mode

This option has to be chosen only by the L3 user for recovery of master database or troubleshooting purpose.
Under normal condition this is not required.

8.3.4 Starting SQL Server with Minimal Configuration

This option has to be used only by L3 users for troubleshooting issues with SQL instance.

8.4 SQL Server Service Startup Options

This option has to be left default as per installation guideline. However, we may be required to make some
changes like, adding trace parameter or capturing deadlock for other purpose. We add trace here only if we want to
make sure after restart of service, the required trace flags are on.

8.5 Changing SQL Server Configuration Settings

All standard configuration settings have been already included in the post installation configuration checklist.
However, configuration setting may be required to be changed under exception cases like max degree of
parallelism, cost of parallelism threshold, CPU affinity, IO affinity, etc. These all will come as a root cause
analysis of problem investigation against performance tuning.

Merck SOPs and SDLC needs to be followed to make changes to configuration settings. In case required SOP does
not exist, one has to be created by HCL and the same has to be approved by Merck DBAs.

8.6 Applying SQL Server Service Packs

Merck has policy to apply only Service Packs after successfully testing it in the lower environment. Hotfix can be
applied as exceptional case to fix issues. Patches are not applied without approval from Merck engineering.

Which Service Pack to apply and when? As on 01 Jan of every year, we check for the latest available Service
Pack. Then we start roll out from the second quarter (01st April) through the environments and complete the
patching before end of year. From next year, again we repeat the process.

The process is well defined in the patching process document. For the entire environment, pre-approval from
business owner is needed as it involves down time. Here one point is to be noted: Database Team DOES NOT
reboot the servers. If required (which is obvious in service pack patching), Database Team will contact Windows
Team to reboot the servicer as per scheduled outage.

All related document “WI for SQL Server Patching.doc” at Documentum folder. In addition, we create the server list for
tracking of patching progress. Sample tracker can be found as “DBMPCalendar_SQL_MMMDDYYYY.xlsx” at below
link.
http://teamspace.merck.com/sites/Modernization/MidrangeDB/Shared%20Documents/Forms/AllItems.aspx?
RootFolder=%2Fsites%2FModernization%2FMidrangeDB%2FShared%20Documents%2FDatabase%20Work
%20Area%2FProject%20Documents%2FTarget%20Tracker%20and
%20Planning&FolderCTID=0x0120000067BA75AC991D4E82793A5CDCB1CBC3&View={19838A01-19DA-
4520-AD59-E49918A9677B}

Page 18 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

It is important to coordinate with business owner in 2 weeks advance. There is a notification template at above
link:
2. Inform the business owner about the activity
6. Reminder of information
Post activity confirmation notification

8.7 Changing SQL Server Port Number

Port number of SQL Servers in standalone environment is not changed and it works on default port unless specially
demanded by the application team. However, on clustered servers, the port number is generally fixed at the time of
installation. The document about the changing port number is attached in teamsite. See section 17.

8.8 SQL Server Agent Job Creation

For Cases regarding job creation, first determine which group will support the project. This will help you setting job failure
alert to the correct team.
Creating the job which is not within the purview of maintenance plan, should be approved by the respective
Application owner. However, such jobs that are not related to Applications but core DBA activity should be
approved by Merck DBA.

8.9 SQL Server Linked Server Creation

Creation of Linked Server will require the Change Management process. It is assumed that it is tested properly in
lower environment for any impact before it is moved to production.
Maintain the inventory of new linked server so that we have the information in case we need to re-create the same due to
some reason (corruption, migration of source/destination server etc.) Remote user/password, context etc. information is
important to save. Currently no such sheet is maintained.

8.10 Enabling Trace Files

Approval is needed for enabling trace files in SQL Server. Make sure support DBA monitors the server for its performance
and disk space while trace is running.

 Choose SQL profiler from the performance tools under the Microsoft SQL Server 2005/2008/2012.
 Choose New Trace and Connect the Desired Database Instance
 Choose required template and got to events selection to filter the login names, DB names, text, and etc. Precisely
select your columns so that you get what you need.
 Click on the Run to start the trace and stop it after desired time completes (if stop time has not been defined).
 Choose the folder D:\DBA hold the trace log. Keep the file size small (1 GB) after that it will roll over to new file.
 If you know the period of time the trace has to be run, define the time so that trace is stopped automatically after the
time is over.
 Alternate way of running profiler is through script and you can define everything in advance and schedule it to run
at specific time when needed.

Page 19 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

8.11 Create Database Trigger

A Create Database trigger should be deployed to each SharePoint SQL Server Instance in order to notify the DBA
Operations Team when a new database is created so the DBA can take immediate action (take backup before the
next scheduled backup). The transaction log / differential (if any) will continue to fail if we do not take a full back
up after the new database was created. An instruction to create the trigger for database creation can be found in
“NewDB_trigger_NonProd.xls”. The same document is referred for production servers too.

8.12 Idera
Merck SQL Server environment has been modernized with Idera Management tools primarily Idera SQL Safe for
database backups and Idera Diagnostic Manager for monitoring. Below are the instructions on how to use these
tools.

8.12.1 Configure Idera SQL Safe backup agent

The document “WI-Deploy_Idera_SQLSafe_Backup_Agent” at below location discusses about
how to configure Idera SQL Safe backup agent in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980eea6ca

8.12.2 Backup Databases using SQLSafe

The document “WI-Backp Databases Using SQLSafe” at below location discusses about how to
backup SQL Databases using Idera SQL Safe in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980eea6ea

8.12.3 Restore Databases to a different SQL Instance using SQLSafe

The document “WI-Restore_MSSQL_User_Database_to_a_different_instance” at below
location discusses about how to restore databases to a different instance using SQLSafe in Merck
Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980eea6cb

8.12.4 Configure Idera Diagnostic Manager Monitoring

The document “WI-Configure_Idera_DM_Monitoring” at below location discusses about how to
Configure Idera DM Monitoring in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef4240

8.12.5 Monthly Reboot of Prod/Dev SQL DM and SQL Safe Servers

After monthly Microsoft Windows Servers Global Patching, SQL Database Team would manually handle
production and non- production SQL DM and SQL Safe Management servers rebooting.

Please find the WI-Restart_Management_Services at the below location.

http://webtop.merck.com/webtop/drl/objectId/090030f980ef4a08

Windows Patching Schedule can be found from below link:

http://one.merck.com/sites/sa/en-us/sd/Lists/Windows%20FAQs/Categories.aspx

Page 20 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

8.12.6 Quarterly password change procedure for account northamerica\srvsiderap

SQL Team needs to change the password of northamerica\srvsiderap account quarterly as it is not
set to never expire. Once you are done with the password change, please update the
northamerica\srvsiderap service account password for the below services.
Prod SQLDM Services:
SQLdmCollectionService
SQLdmManagementService
SQLdmPredictiveAnalyticsService

Prod SQLSafe Services:

SQLSafe Backup Service
SQLSafe Management Service
It is recommended to change the password during monthly reboot of the Prod SQLSafe and SQLDM
Servers so that we can avoid additional downtime.

Please update the password XL file with new password for the account northamerica\srvsiderap

8.13 On-Boarding an Existing SQL Server Instance

The document WI_On-Boarding SQL Server provides compete guideline to onboard an existing SQL Server
instance in Merck Environment based on below scenarios.
 Reinstall/Rebuild or Migrate an instance on an existing old server
 Discovered orphan instance or installed/introduced instances from client/application/vendor side, which
were not under database operations standard support
More detailed procedures please follow the second half (CHECKLIST (SQL SERVER)) of “WI - Database
Onboarding Checklist” in Documentum or follow below URL
http://webtop.merck.com/webtop/drl/objectId/090030f980f1de4b

8.14 SQL Server Database Management

8.14.1 Database Creation

HCL DBA team will receive request to host database host / create a database for specific application. Destination
server, logins, users would be there with request and it will be pre-approved before the request comes to DBA
team. Remedy Ticket / Request will not be sufficient for creating / restoring a database on Production / UAT
environment.
The document Database Creation WI provides compete guideline for the database creation.

8.14.2 Database Decommission and Deletion

For Cases regarding database deletion, first determine which group will support the project.

Please ensure that approvals from business owner are in place and Merck DBAs are in loop and they are aware of the plan
before proceeding to decommission of a Database. An RFC is required for this task.

Connect to the Instance Using Microsoft SQL Management Studio

Backup the Database and right click on it and select delete and click on close existing connections and click on Ok.
Page 21 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

After the task has been completed intimated the user and close the case.

Also, make sure if the database backup has to be retained for more than retention period. In that case, application owner may
need to get approval to retain the backup for the desired period and backup team will have to be assigned a work order
through remedy to carry out the task.

We do not have WI for database decommission at the moment. Has to be created and approved by Merck DBAs.

8.14.3 Database Move from One Instance/Server to another Instance/Server

A database can be moved from one SQL Server Instance to another via Backup and Restore or Database Detach
and Attach. The process will require down time and hence needs business approval. Merck DBAs would be
involved while planning this activity.

8.15 SQL Server Instance Decommission

8.15.1 Checklist for Retiring a SQL Server Instance

There is complete detail in this checklist to be followed while decommissioning. Please refer the document “SQL
Server Decommissioning” for more detail.

8.16 SQL Server Storage Management

Manage Storage Sometimes; we get alert of low disk space on SQL server due to various reasons. To address such
issues, there are few points suggested. Please refer the document “Manage SQL Server Storage” for detailed
information.

8.16.1 Add Disk Space

In case we checked all options as per “Manage SQL Server Storage” (discussed in in above Para), we need to add
disk space to address the issue of low free disk space on SQL Servers. The details of how to add disk space can be
found in “WI - Adding Disk Space to SQL Server”.

8.17 IQOQ/Quality Gate Reports Based on SDLC and Activities

To decide what IQOQ or Quality Gate reports to generate for the activities such as New Builds, Upgrades,
Migrations in different environments/SDLC please follow the below chart.
Instance Activities Production & Formal Test Other Environments
New Build IQOQ QG
Major Upgrade / Modernization IQOQ QG
Minor Upgrade /Patch Apply QG QG
IQOQ if no previous IQOQ in place, QG otherwise.
Migration (no upgrade involved) QG
Always run a QG before migration for comparison

9 Platform Support Procedures – AWS RDS SQL Instance

9.1 AWS RDS SQL Instances in Merck Environment

To get a basic understanding of AWS RDS SQL Instances in Merck Environment please go through the Appendix
– A: Amazon RDS – Informational Guidelines for use at Merck

Page 22 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

9.2 AWS RDS SQL Instance Provisioning

The document “WI-RDS SQL Server Instance Provisioning” at below location discusses about how to create RDS
SQL Instances in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef743d

9.3 Accessing RDS SQL Instance

To access the RDS SQL Instance through the AWS Console please go through the below Work Instruction.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef7439

To access the RDS SQL Instance through SSMS (SQL Server Management Studio), you need to use AWS-RDS
connection string: <instance_name>.c1jjnkpskclo.us-east-1.rds.amazonaws.com,25881 and you need to provide SQL
Authentication login details. You can get the login details from password xl sheet.

9.4 AWS RDS SQL Instance Retirement

The document “WI-Retire_MSSQL_Instance_Checklist” at below location discusses about how to
retire/Decommission RDS SQL Instances in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef743e

9.5 Import/Export Database Data

AWS RDS SQL Instances does not support regular backup/restore of data from normal SQL instance to RDS SQL
Instance. To import/export the data we need you import/export data wizard which is provided by Microsoft.
For Cases regarding Import and export, first determine which group will support the project.
 Request user to provide approval from application owner
 Check the size of the table that needs to be exported in terms to number of rows.
 Proceed with the export if that is for few thousand rows from prod to Dev and during business hours
 If you see more than few thousand rows, schedule the export for off peak hours.
 If the request pertains to import into a production server, request user to create RFC and get all approvals as
needed
 Follow the steps included in the below path to Export/Extract database data
http://webtop.merck.com/webtop/drl/objectId/090030f980ef7a25

9.6 RDS Database Recovery

The document “WI-Perform RDS Database Recovery” at below location discusses about how to perform RDS SQL
Instances database recovery in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980efabab

9.7 RDS Manual Snapshot Backup

The document “WI-RDS Manual Snapshot Backup” at below location discusses about how to perform Manual
Snapshot Backup of RDS SQL Instance in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef7447

Page 23 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

9.8 Modify DB Parameter Group

The document “WI- How to Change DB Parameter Group” at below location discusses about how to Change DB
Parameter Group of RDS SQL Instance in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980ef743a

9.9 Change RDS Instance type and update tags

The document “Checklist-Modify RDS Instance Type” at below location discusses about how to Change RDS
Instance type and update tags of a RDS SQL Instance in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980efee7a

9.10 RDS Monitoring

The document “AWS-RDS Monitoring Plan” at below location discusses about Monitoring Plan for RDS SQL
Instance in Merck Environment.
http://webtop.merck.com/webtop/drl/objectId/090030f980efdfd7

9.11 RDS Instance IQOQ

***********************WORK IN PROGRESS**************************

9.12 Other Amazon RDS Database Activity Information

To perform the database activities on Amazon RDS SQL Instances like
 Changing Database Recovery Model
 Collations and Character Sets for SQL Server
 Transitioning a Database from OFFLINE to ONLINE
 Running a client-side trace on a SQL Server DB Instance
 Running a server-side trace on a SQL Server DB Instance
 Running Tuning Advisor with a Trace
 Using SQL Server Agent
 Viewing the SQL Server Agent Log
Please go through the Amazon RDS Cheat Sheet for SQL Server document available the below location.

http://webtop.merck.com/webtop/drl/objectId/090030f980ef2a89

Page 24 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

10 Backup and Recovery

All the databases in Merck environment has to be backed up unless it is declared in backup exception list by the
application team. Primary responsibility for the backup of databases goes to Backup team who are responsible to
back up all the databases. However, Database Team is also responsible to make sure all databases are being
backed up and for this purpose they (Database Team) generate a Backup Status Report.

10.1 TSM TDPSQL Backup

The document “Work Instructions for TSM Backup and Restore” is referred for how to take backup and restore
when needed.
Restore responsibility remains with Database Team and above document is used for help. In case there is any issue
restoring database, Backup Team would be escalation point for Database Team.

Currently majority SQL instances are migrated to SQL Safe backup since late 2013 to 2014. But SharePoint
production environment and a couple of SharePoint non-production instances are still under TSM tdpsql backups
because the application needs to have object level backups that SQL safe doesn’t support. There are also some
remote and old version of SQL instances are also under TSM backups.

10.2 Idera SQL Safe Backup

Starting 2014 majority SQL Server instances are implemented SQL Safe backups.

To Configure and Deploy SQL Safe backup please follow “WI-Deploy_Idera_SQLSafe_Backup_Agent” and
“WI-Add SQL Instance to Idera SQL Safe Backup Policies” in Documentum or follow the below URLs:
http://webtop.merck.com/webtop/drl/objectId/090030f980eea6ca and
http://webtop.merck.com/webtop/drl/objectId/090030f980f06f74

To Backup and Restore SQL databases using SQL Safe please follow “WI-Backp Databases Using SQLSafe” and
“WI-Restore_MSSQL_User_Database_to_a_different_instance” in Documentum or follow the below URLs:
http://webtop.merck.com/webtop/drl/objectId/090030f980eea6ea and
http://webtop.merck.com/webtop/drl/objectId/090030f980eea6cb

10.3 Simpana CommVault Backup

Merck has decided adopting Simpana Commvault as a backup tool for SQL Server farm starting 2015. Currently
(up to August, 2015) all AWS EC2 instances and some remote instances have been migrated to Commvault
backup. All instances that currently under SQL Safe backups will be migrated to Commvault backup in 2015 to
2016 time frame. However, The Commvault tool is not supporting SQL Server 2000 instance which still quite
much amount in Merck SQL Server farm. The vendor and Merck teams would work out a solution for it.

To migrate SQL Server backup from other tools to Commvault please follow the “WI-Migrate SQL Instance from
SQLSafe to Commvault Backup” in Documentum or follow the below URL:
http://webtop.merck.com/webtop/drl/objectId/090030f980f12b5e

To Backup SQL Server instances using Commvault please follow the “WI-Backp SQL Server Databases Using
CommVault” in Documentum or follow the below URL:
http://webtop.merck.com/webtop/drl/objectId/090030f980f0313d

Page 25 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

To Restore a SQl Server database using Commvault please follow the “WI-Restore SQL Server Database Using
CommVault” in Documentum or follow the below URL:
http://webtop.merck.com/webtop/drl/objectId/090030f980f0313e

10.4 Checking and Monitoring Daily SQL Server Database Backups

To ensure Merck SQL Server farm has successful backups is one of Database Operation Team’s top
responsibilities. The DB Operations Team should perform below checking up in daily basis. If identify any failed,
errored, skipped backups Database Team should collaborate with Backup Team to fix the problems as soon as
possible. We have below tools/reports to monitor and check SQL database backups:
1. Idera Daily Backup Reports for production and non-production instances (generate by SQL DBA Team)
2. Idera SQL Admin Toolset (ad-hoc checking)
3. Idera SQL Safe Consoles
4. Backup Job Summary Report - Daily_SQL_Report from Simpana_usctapp41325 (generated by Backup Team)
5. Apex Dashboard
You can also feel free to use any query scripts.

11 SQL Server Monitoring and Alerting

11.1 AlarmPoint
Most of production database servers are being monitored using HP Openview tool, SPI. EMS team is responsible
to configure the tool on the servers. Whenever an incident happens, it sends an alert to corresponding team (in case
of database servers, goes to “Database Operations Resources” (1datoperr@northamerica.msx.merck.com). The
team is responsible to respond to the alert within 15 minutes failing which it will get escalated to higher level. For
escalation matrix, please see the attached document “AlarmPoint Escalation.Doc” in section 17.

Based on your access, you can get information about AlarmPoint and its configuration at below link.
http://alerting.merck.com/alarmpoint/signOn.do
Once Database Team has resolved the issue, Database Team sends an email to “Database Operations Resources”
(1datoperr@northamerica.msx.merck.com).
It is desired to create an incident ticket (currently it not being created) so any investigation, root cause and
resolution provided can be recorded.
For more information about AlarmPoint please refer
http://webtop.merck.com/webtop/drl/objectId/090030f980edaa14
Note: Here we use different login and password (not the isid). Check with team for further information.

11.2 Alert and Email Notification Creation

A set of Standard Alerts is created on all Production SQL Server Instances.

Page 26 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

For Standard Alert notifications creation, see SQL Server Installation guide – Post Installation Steps.
Checklist SQL Server Installation.doc (http://webtop.merck.com/webtop/drl/objectId/090030f980f15c31).

11.3 Alert Notification Response

SPI tool has been configured on most of the production instances and it has been made mandatory on every new
server. In case of issues, an automated alert is generated and the email carries an option to accept / reject the
notification.
It is mandatory to accept the alert notification within 15 minutes of receipt and start working on the issue.
Currently, no ticket is created for such alert and resolved by Database Team.
It is desired to create incident ticket for each alert and record the investigation and resolution provided.
The escalation level and time interval for next level matrix is attached in the below link.

http://teamspace.merck.com/sites/edbmm/Database%20Information/Forms/AllItems.aspx?InitialTabId=Ribbon
%2EDocument&VisibilityContext=WSSTabPersistence

Note: The alert received may be from any technology of database services including MSSQL. Database Team has
to identify and correct personnel needs to work on it.

11.4 General Database Mail

Although Database Mail is being configured on all the newly built servers, there may be need to configure it on
some existing server (where it was not configured earlier). For that purpose, below method can be used.

Approval is needed to create a new Database Mail Account.

To send Database Mail, the User must be a member of the DatabaseMailUserRole database role in the msdb
database.
Steps Performed to Create an E-mail Operator:
 Connect to the instance using Microsoft SQL Management Studio
 Double Click SQL Server Agent
 Right Click on Operators and select New Operator
 Specify an operator Name, E-mail name, and click OK

Page 27 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

11.5 Login Failure Report

For some critical servers, business may want to monitor the failed login attempts. To monitor such cases and
report it to the intended parties, there is guideline that can be found at (Cabinets/IS/Global
Infrastructure/GDS/Team Folders/Database Services/Standards/SOPs/Scripts/DI-SOP-00021 - SQL Server Login
Failure Monitoring SOP).

11.6 Idera SQL Diagnostic Manager

Starting year 2014 Merck SQL Server Team adopted Idera SQL Diagnostic Manager tool to monitoring all SQL
Server environments. The existing instances (except instances are below version SQL2000 SP3) are all configured
in SQL DM Console. One of the steps for any new build or onboarding existing instance is to configure SQL DM
monitoring. To configure SQL DM monitoring please follow “WI-Configure_Idera_DM_Monitoring” in
Documentum or follow below URL
http://webtop.merck.com/webtop/drl/objectId/090030f980ef4240
To disable SQL DM monitoring please follow “WI-Disable SQL Server Monitoring In Idera” in Documentum or
follow below URL
http://webtop.merck.com/webtop/drl/objectId/090030f980ef6a87
If any need to contact the Idera vendor support please follow “WI-How to get IDERA Technical Support” in
Documentus or follow below URL
http://webtop.merck.com/webtop/drl/objectId/090030f980efbb86

12 Database SOX Auditing & Reporting

Currently there are no SQL database / Database Servers which are SOX compliance. However, when needed an
existing guideline can be referred at (Cabinets/IS/Global Infrastructure/GDS/Team Folders/Database
Services/Standards/SOPs/Scripts/DI-SOP-00019 – SQLSRV\Audit Configuration SOP Scripts).

There are some tables which are critical for its information and important to capture the changes. To configure such
critical tables for audit, please refer the guideline at (Cabinets/IS/Global Infrastructure/GDS/Team Folders/Database
Services/Standards/SOPs/Scripts/DI-SOP-00020 – SQLSRV Critical Table Auditing SOP Scripts).
Note: This can be obsolete from SQL 2008 onward as a feature of CDC (Change Data Capture) is ready available
there.

13 SQL Server Build

13.1 Database Acquisition Process

Following steps are followed before deploying a new database in the Merck environment:
For detailed steps, please refer below Work Instruction:
http://ssdba-central-prd.merck.com/pls/apex/f?p=10302:41:3061023822058244::NO::P41_DOC_ID:544

13.2 Database Profile Form

Database Profile Form (DPF) is used by CFDBAs for putting database build detail as per requirement gathered from
business users or application team. CFDBAs create Remedy and attach this form to the Engineering team in order to
generate a Database Build Document.

Page 28 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Below is DPF template:

http://ssdba-central-prd.merck.com/pls/apex/f?p=10302:41:3061023822058244::NO::P41_DOC_ID:542

You can get filled DPF from link below:

http://ssdba-central-prd.merck.com/pls/apex/f?p=10302:201:3061023822058244::NO

13.3 Default Database Parameters

If nothing is specifically mentioned in the Database Profile Form, the following default values are taken for
database parameters during generating Database Build Document. Please refer “Microsoft SQL Server
Engineering Standard”

13.4 APEX Database Request Process

There are defined Roles and Responsibilities for CFDBAs, Shared Services Manager, and HCL DBAs. Also, there
is a process guideline that needs to be followed. Please refer the document below for more detail.
http://ssdba-central-prd.merck.com/pls/apex/f?p=10302:41:3061023822058244::NO::P41_DOC_ID:763

Work order is created for building SQL Server. Detail of SQL Server installation can be found in Documentum
Checklist SQL Server Installation.doc (http://webtop.merck.com/webtop/drl/objectId/090030f980f15c31).
Please refer the SQL Server Standards section for the best practices followed by Merck.

Once installation is complete, IQOQ is performed on the new instance to check the server has been built as per
standard guideline. Please refer the IQOQ document
“http://webtop.merck.com/webtop/drl/objectId/090030f980ed8ef0” (MSSQL IQOQ Tech. Document) for more
information.

To generate report, go to the link below.

http://uswsapt10119/Reports_TEST01/Pages/Folder.aspx

Enter UserID and password as mentioned in Password list document and enter correct SQL Instance that was built
and report has to be generated.

13.5 Creating Service Account (Non-person Account)

Please note that there is a Self-Service available for Merck employees to create NORTHAMERICA domain non-
person accounts using the portal via https://accesscenter.merck.com/identity. Please have your business sponsor
follow the below process for Self-Service which is available for Merck employees to create non-person accounts
using the portal via https://accesscenter.merck.com/identity.
Steps for Self-Service:
1. Login with your ISID and network password.
2. Type npa in Catalog field then click Search.
3. Under Catalog Items click “Add to Cart” button.
4. Click “Checkout” button.
5. Select “Service” for “Account Type” from the drop down list
6. Enter unique account name you would create in “Account ID” field (follow the naming convention)
7. Enter default password “Merck123!” in “Password” field (the default password will be changed to standard
password later by instance creator)
8. Enter the same account name in “Display Name” field

Page 29 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

9. Enter “Service account to manage SQL Server on <server_name>” in “Description” field (enter
<server_name>a/b or a/b/c for cluster servers)
10. Check the “I have read and accept the Terms & Conditions stated above” check box at bottom of the page
11. Click “Save” button on top right.
12. Click “Submit”
13. You will get an email confirmation with your new non-person account information excluding the password.
Owners can submit the Outlook - Non-Person Account Management service request for managing a non-person
account as the self-service portal is limited to modifications of only changing NPA owners at this time.

Definitions of Non-Person Accounts:

 Custom Account: Custom Accounts are accounts created to provide specific process or automation functionality
beyond standard service or process NPA account capabilities.  They may map to a named specific user (the primary
owner).   For example, a custom account may be leveraged to provide mail-enabled functionality via Exchange for a
conventional NPA process account through the use of an email alias.  A Custom Account has historically been
called a Custom Business account. This may be used as a shared mailbox account, a conference room administrative
account, etc., and is capable of mailbox privileges—internal and external to Merck

 Service Account: Service Accounts are accounts created to provide specific functionality which but do not map to a
named specific user. Examples of service accounts are non-interactive accounts utilized for application
communications, daemon accounts used in an operating system to support or monitor system activities, and common
accounts shared by all members of the workforce performing a particular role or function. Service NPA accounts are
used in system automation scenarios involving service communications—not application process communications. 
A Service Account should ideally be leveraged by computer systems only to automate a programmatic, service, or
reporting function.  Exceptions to this use need to be evaluated and approved by Information Risk Mgmt

 Process Account: Process Accounts are created to provide specific functionality/access to an instance of an
application/database that is being executed but do not map to a named specific user. Process NPA accounts are used
in system automation scenarios involving intra- or inter-application process communications—not system
services/daemon communications.

 System Account: System Accounts are local system administrative accounts provided for a product or operating
system for the purpose of enabling specific role-based access. Typically these accounts provide administrative
access to applications and system services.  Please note that these accounts do not include the default product or
system accounts normally preconfigured with an OS or COTS application, but augment the default account
configurations

Page 30 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

 Test Account:  Test Accounts are accounts created to provide specific functionality but do not map to a named
specific user.  The accounts are utilized for the purposes of testing systems, applications, or enterprise access within
a dedicated test environment or domain

 Training Account:  Training Accounts are accounts created to provide specific functionality but do not map to a
named specific user.  The accounts are utilized for the purposes of training new users on a specific
application/system in a dedicated test/training environment.

14 Disaster Recovery / BCP

Class A servers are replicated at SAN level. The drives with binary files are not replicated and they are local to the
servers. It is important to keep both the servers (production and DR) at the same build level to make it possible to
switch to DR server without any conflict of version / build level.
Currently, there are only two Class Application in SQL Server environment.
There is only one Class-A DR environment supporting GSIP Application to participate the annual DR exercise.
Production Server Names:
usctapp50013a – 54.50.112.57 and usctapp50013b – 54.50.112.57
D/R Server Names:
uswsapr50013a – 54.16.144.13 and uswsapr50013b – 54.16.144.1
The detailed DR scenario and Recovery Process has been discussed and documented in “L5-DR PLAN CLASS A -
GSIP-MOONS.doc” in Documentum as below URL
http://webtop.merck.com/webtop/drl/objectId/090030f980f20d58

Other DR related Documents are available at \\usdata024\IT_Service_Continuity (you may need to require
permission to access).

Page 31 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Here is recovery flow for DR. Database Team / Application comes at the end (Level 5).

Note: Disaster Recovery Tests are performed annually for Class-A applications.
All levels of DR activities (Class A, Class B, etc.) are in the scope of 3A support.

15 SLA Management

15.1 SLA
Below are the SLA defined by MERCK for resolving the Support Request or Incidents.

Type of Tkts Response Time Resolution Time

Priority 1 15 Min 4 Hr.

Priority 2 15 Min 8 Hrs.

Priority 3 30 Min 24 Hrs.

Priority 4 60 Min 72 Hrs.

Page 32 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

15.2 Shift Timings

Suppor
t HCL will deliver round the clock coverage (24 X 7 X 365) for Merck in-scope environments
window
Steady State
Technology Shifts
Mon Tue Wed Thu Fri Sat Sun
st
India 1 Shift 1 1 1 1 1 * *
Shift
Rota India 2nd Shift 1 2 2 2 1 1 1
SQL Server India 3rd Shift 1 2 2 2 1 1 1
US Shift 1 1 1 1 1 * *
SQL Total 4 6 6 6 4 2 2

1st Shift 6.30 AM to 3.30 PM IST

2nd Shift 1:30 PM to 10:30 PM IST

3rd Shift 9.30 PM to 6.30 AM IST

Page 33 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

16 Scheduled Activities

16.1 Daily Activities

16.1.1 Daily Health Call (DHC)

There is daily health call at 8:00 AM EST and Track-Lead from every technology joins this call along with Onsite
Service Managers and Merck counterpart. This call is scheduled to discuss overall health status across the
technology, any on-going Priority ticket, downtime or priority incident / outage occurred after last Health Call.
Distribution list of James Fletcher…..join this call on behalf of Merck.

16.1.2 Shift Handover Process

There is overlapping of 1 hour between the shifts. This one hour will be utilized as handover time to provide
updates to the following shift in charge about any pending task / on-going issue or any kind of update that has to
be delegated / informed. The shift handover report can be found at
http://teamspace.merck.com/sites/edbmm/Offshore%20MOB/Forms/AllItems.aspx?RootFolder=%2Fsites
%2Fedbmm%2FOffshore%20MOB%2FShift
%20Handover&FolderCTID=0x01200073356873C1D3644FB6703BE1705015A4&View={29EE8451-B74A-
434F-89CA-8BA2017FF6AC}

16.1.3 Daily Backup Report and Meeting with Merck DBA

Backup status report is generated every day and necessary remediation action is taken as needed. Idera admin tool
is used from local machine / VDI machine (please note that Idera Admin tool comes with user license. However, it
cannot be installed on some server to be used by multiple users as it will violate the licensing agreement). Please
refer the document “Idera - SQL Backup Report Generation WI” for more information about how to install Idera,
configure and register servers, generate report and upload on backup Dashboard.

Once the backup status report is generated, it is uploaded to the in-house developed application called backup
dashboard.

Backup monitoring & reporting is part of daily DBA operations. Backup failures/issues are reported using a tool called
Apex (Back end Oracle tool) & TIBCO Spot Fire Dashboard.
 Daily at 4 PM EST, Backup Review meeting is conducted by James Fletcher, where the progress on the backup
tasks, issues, and other information’s related to backups are being discussed.
 Database Operations team is responsible to keep the dashboard clean without any issue. If any issues, it should have
valid reasons and proper assignment in the dashboard.
 Loka Reddy will be the contact for getting the access, doing any changes, or fixing any issues in Apex or
Dashboard.
 Production Database backups (FULL/INCR) should NOT be RERUN on the weekdays.
 To identify new incidents on day to daily basis, you can login to the APEX tool and sort it.
 Old Incidents status will get closed once if that particular entry meets its logical requirement. For example .Last full
failed category entry will be closed once we have successful backup, OEM Consecutive failure entry will be closed,
if we have consecutive 3 successful backups.

Page 34 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Following is the link which contains Work Instructions for using TIBCO Spot Fire Dashboard.
http://teamspace.merck.com/sites/edbmm/Database%20Backup%20Information/Forms/AllItems.aspx

Action to be taken:
Check the entries and fix the issues and make sure the next day report will be cleaned.

Report Mail Box:

Database Operations Resources <1datoperr@NorthAmerica.msx.merck.com>

Flow of Strategy for Backup Failures

Page 35 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

16.1.4 Daily Backlog Tracker / Automated E-mail from Remedy

There is an automated email sent from Remedy with the details of all open tickets with all priority and its status.
This report provides overall work progress.

Note: The DTR report will be delivered through e-mail to the intended audience.

For example: C:\Users\xxxxx\AppData\Local\Microsoft\Windows\Temporary Internet

Files\Content.Outlook\JBFHWFUZ\DBA - DailyTicketReport_Aug_06_2013.htm

16.2 Weekly Activities

16.2.1 Check for Available Disk Space

Currently there is monitoring in place to monitor the disk space proactively. As soon as the disk space utilization increases
more than 90%, database team analyses the situation and check if there is any unexpected backup file, other files, log file /
data file growth and if this can be fixed by removing, re-locating files within the group. In case there is no possibility to fix
it and disk space has to be increased, the same is forwarded to Merck DBA to look into it and arrange for additional disk
space. Please refer the section “SQL Server Storage Management” for more details. Disk space status report is available at
below location.
“http://usctap0417.merck.com/HPOV_Reports/Application_Reports_Group_MSSQL.htm”

Reports generated on a weekly and monthly basis are distributed to varies leads and uploaded to below location.
http://teamspace.merck.com/sites/InfSolEng/ISEDBEMS/ISEDB/Shared%20Documents/Database
%20Operations/Daily%20and%20Weekly%20Reports/SQL%20Server%20Reports.doc

16.2.2 Weekly Alarm Point Report

Please login into http://alerting.merck.com/ with standard credentials and go to Alerts tab in the right hand
side then choose Past alerts from the left side and select the results per page as desired and select the last week
dates(Monday to Sunday) and choose DBSS_DBA_PAGER and click on show past alerts. Report is sent to below
mentioned recipients and DLs.

To: Database Operations <1datopers@NorthAmerica.msx.merck.com>

Cc: CFDBA Resources

16.2.3 Operations Update with HCL

There is weekly call between Engineering, CFDBAs and HCL DBAs chaired by Merck Database Operations
Manager. In this meeting we discuss the status of new request coming from Business, in progress infrastructure
builds and any challenge or suggestion from the participants to improve the services. Currently meeting occurs
every Thursday from 7:30 AM to 8:30 AM

16.3 Yearly Activities

Patching: SQL Server patching is defined as Annual activity. For more details, please refer the section of SQL
Server patching.

Page 36 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

16.4 Audit Report

Currently there is no server which needs to be SOX compliant. Hence, the audit is out of scope until there is any
need. The process / guideline would be defined when needed. Accounts in SQL Server are validated every year for
the compliant servers. Member of server role and member of database level role are identified and recorded. Sa
account is exempted. For more detail, please refer the document DI-SOP-00018 SQLSRV Annual Account
Validation SOP

16.5 Ad-hoc Activities

As the title explains, it is ad-hoc activities and cannot be predicted.

17 Training

MyLearning provide the list of required training based on the curriculum assigned to the individual’s role. HCL has
identified Non Merck Persons to play the myLearning Report Runner role across 3A Group.  These individuals are
responsible to run myLearning reports & track course completions / overdue items.
Merck’s required training curriculum by role is maintain and updated by Merck Training Lead and is found at below
link:

IT Managed Services Partner Training Requirements

18 References
Reference Identification / Description Source / Location
SDLC-SOP-05-08: Global Services (GS) Service Request Fulfillment https://remedy-web.merck.com
SDLC-SOP-05-01: IT Incident Management https://sdlc.merck.com
SDLC-SOP-05-02: IT Problem Management https://sdlc.merck.com
SDLC-SOP-05-07: Administration and Control of Service Knowledge
Management Articles for Global Services (GS)
https://sdlc.merck.com
ITSM-SOP-GEN-000068 - GTS Configuration Management Process http://edms.merck.com
ITSOP-0006 - Qualification of Infrastructure Systems and Components http://edms.merck.com
ITSOP-0011: System Administration and Use of Enterprise Backup and Recovery
Systems
http://edms.merck.com
ITSOP-0014 - Account and Access Management for IRM&C Supported
Applications and Systems
http://edms.merck.com

ITSOP-0015 – IT Change Management Process – Pre-Authorized Changes http://edms.merck.com

ITSOP-0020 - IT Change Management Process – Planned and Unplanned
Changes
http://edms.merck.com
ITSOP-0018 - Information Security Threat and Vulnerability Management http://edms.merck.com
ITSOP-0019 - Information Security Investigation Management http://edms.merck.com
ITSOP-0021 - Information Security Incident Management http://edms.merck.com

Page 37 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

19 Resources

Resource Identification /
Source / Location
Description
Remedy https://remedy-web.merck.com
Documentum http://webtop.merck.com/
Teamsite http://teamspace.merck.com/sites/edbmm/_layouts/viewlsts.aspx?BaseType=1
Share Data Access http://sda.merck.com
Database Information e.g. Password http://teamspace.merck.com/sites/edbmm/Database
files location %20Information/Forms/AllItems.aspx
Location of SOPs for CFDBA 6
Documentum\Docbases\gdmsprd\IS\Global Infrastructure\GI Document
Database Services Library\Departmental View\DBA-Database\DBA-SOP
(Webtop.merck.com)

Location Work Instructions Documentum\Docbases\gdmsprd\IS\Global Infrastructure\GDS\Team

(Webtop.merck.com) Folders\Database Services\Standards\Work Instruction
Database Build Document http://teamspace.merck.com/sites/edbmm/Database%20Build%20Document
Repository %20Repository/Forms/AllItems.aspx
Database Operations (Need to http://teamspace.merck.com/sites/edbmm/Database
verify) %20Operations/Forms/AllItems.aspx
IQOQ Report Generation after
database creation and upgrade
http://webtop.merck.com/webtop/drl/objectId/090030f980dfe32d

http://teamspace.merck.com/sites/edbmm/Offshore%20MOB/Forms/AllItems.aspx?
On Boarding Document (Runbook, RootFolder=%2Fsites%2Fedbmm%2FOffshore%20MOB%2FOnBoarding
etc.) %20Documents&FolderCTID=0x01200073356873C1D3644FB6703BE1705015A4
&View={29EE8451-B74A-434F-89CA-8BA2017FF6AC}
http://teamspace.merck.com/sites/edbmm/Offshore%20MOB/Forms/AllItems.aspx?
RootFolder=%2Fsites%2Fedbmm%2FOffshore%20MOB%2FOnBoarding
On Boarding Training Checklist
%20Documents&FolderCTID=0x01200073356873C1D3644FB6703BE1705015A4
&View={29EE8451-B74A-434F-89CA-8BA2017FF6AC}
http://teamspace.merck.com/sites/edbmm/Offshore%20MOB/Forms/AllItems.aspx?
RootFolder=%2Fsites%2Fedbmm%2FOffshore%20MOB%2FOnBoarding
Changing Port numbers
%20Documents&FolderCTID=0x01200073356873C1D3644FB6703BE1705015A4
&View=%7b29EE8451-B74A-434F-89CA-8BA2017FF6AC%7d
http://teamspace.merck.com/sites/edbmm/Offshore%20MOB/Forms/AllItems.aspx?
RootFolder=%2Fsites%2Fedbmm%2FOffshore%20MOB%2FOnBoarding
Alarmpoint Escalations
%20Documents&FolderCTID=0x01200073356873C1D3644FB6703BE1705015A4
&View=%7b29EE8451-B74A-434F-89CA-8BA2017FF6AC%7d

Page 38 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
 Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

20 Appendixes

Appendix – A: Amazon RDS – Informational Guidelines for use at Merck

Purpose
The objective of this document is to provide items to consider when deciding on whether Amazon RDS for
SQL Server or Oracle is an appropriate database option for a given project.
Scope
This document is specifically intended to cover Amazon RDS for SQL Server and ORACLE .
Definitions

The following table contains terms and/or acronyms used in this document and the corresponding
definition.

Table 1: Definitions

Term/Acronym Definition

AD Active Directory
AWS Amazon Web Services
IOPS I/O Per Second
RDS Relational Data Service
VPC Virtual Private Cloud

Page 39 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Advantages of using Amazon RDS

Time to deliver
 The Amazon RDS service allows for rapid deployment of new database instances that can be
setup in minutes rather than the traditional time required for server setup, database instance
install/configuration and customization.
 Proof of concept projects can now quickly stand up and test and idea and increase chances of
successfully proving an idea in a timely fashion.

Scalability
 The Amazon RDS allow for on demand scaling. Projects can now scale the infrastructure up or
down depending on current need. In house solutions had generally provided for the highest use
scenario which in many cases drove project costs up significantly. Amazon RDS helps alleviate
some of the burden early on in a project when it is difficult to determine the proper sizing of a
database in terms of resource utilization.

Operational
 Patching ,maintenance, backups done by vendor

Cost
 Amazon RDS can also help reduce the overall project cost in numerous ways. Eliminating the
need to purchase, setup, and maintain servers and database instances along with the footprint
reduction in data centers can greatly reduce cost.
 The model for usage is pay for utilization. Projects that only require infrastructure for a specific
period of time such as acceptance testing can now stand up a database instance for a designated
timeframe and then shut down the instance when complete. Project will only pay for the
instance during the time is was in use.

Predictable Performance
 The Amazon RDS Service allows the project to set/determine the IOPS requirement of the
database application. Amazon guarantees IOPS thereby providing our clients with consistent
database performance.

Regional independence
 For projects that require a regional presence of the database instance, Amazon RDS provide for
locating services in the same region allowing for greater flexibility when deploying solutions.

Page 40 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Disadvantages of using Amazon RDS

Vendor dependence

 Loss of control over the software application and becomes dependent on the provider to
maintain, update and manage it. If something goes wrong, the user does not have direct access
to the software and must depend on the provider to fix the problem. If the provider is
unresponsive or unable to fix the problem quickly, the user can experience significant issues.
 May take longer to support newer product versions

Security/ Privacy and confidentiality areas

 By using a cloud system, Merck’s company sensitive data and information will be stored on
third-party servers. We rely on the provider security and encryption. If a breach of these
systems or procedures occurs for any reason, company’s private and confidential data may
become compromised. This could have many negative effects, and could cause legal problems
for Merck if third party confidential information (for example, customer information) is
compromised. Although VPC could can help mitigate this risk.

Technical Obstacles for SQL Server

 Amazon RDS for SQL Server does not support windows authentication. SQL authenticated
accounts are needed for application usage. Generally SQL Authentication is considered less
secure than windows authentication.
 Amazon RDS for SQL Server currently only supports static ports. Dynamic ports allows for
application code to exclude the port number in the connection string which provides greater
flexibility in code. Static ports will need to be included in application code.
 Loss of control over administrative functions thru normal SQL methods. Need to provide
training on Amazon procedures for administration.
 Merck monitoring tools not supported.
 SQL Server Enterprise Edition is not currently supported in the “license include” model. SQL
Enterprise Edition is only supported in the “bring your own license” model.

Technical Obstacles for Oracle

Page 41 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

 Segregation of data, log, and control files is not possible.

 No access to OS
 No allowance for transaction log backups although snapshots are possible.
 Feature set is limited

Good candidates

 Non critical app although this does not rule out critical applications from using.
 Application and DB Tiers split and both are hosted in cloud
 Standard database features
 Merck user based only (no third party)

Undesirable Candidate

 Shop Floor and building management systems not good candidate due to the nature of the
application requirement to be physically located within the building where the work is being
done.
 Applications that require AD authentication would not be considered good candidates for
Amazon RDS since windows authentication is currently not supported.
 Application requiring SQL Server Enterprise Edition but not in a position to fund the license
cost upfront.
 Colocation of database and application on same node
 Applications requiring third party access outside of the Merck network
 SQL Server projects requiring :
o Reporting Services
o Analysis Services
o Maintenance Plans
o Database Mail
o Distributed Queries (i.e., Linked Servers)
o Transparent Data Encryption
o Database Log Shipping
o Database Mirroring
o Windows Authentication

 Oracle projects requiring :

o Real Application Clusters (RAC)
o Real Application Testing
o Data Guard / Active Data Guard
Page 42 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

o Oracle Enterprise Manager Grid Control

o Automated Storage Management
o Database Vault
o Streams
o Java Support
o Locator
o Spatial
o Oracle XML DB Protocol Server
o Network access utilities such as utl_http, utl_tcp, utl_smtp, and utl_mail, are not
supported at this time.

RDS Options

Amazon RDS for SQL Server & Oracle

 Bring your own license - Microsoft’s License Mobility program allows customers who
already own SQL Server licenses to run SQL Server deployments on Amazon RDS. This
benefit is available to Microsoft Volume Licensing (VL) customers with SQL Server
licenses (currently including Standard and Enterprise Editions) covered by active
Microsoft Software Assurance (SA) contracts. Oracle also allows customers to bring an
existing license and utilize this model.
 License Included model - Under the "License Included" service model, you do not need
separately purchased Microsoft SQL Server licenses. "License Included" pricing is inclusive of
software, underlying hardware resources, and Amazon RDS management capabilities.
 On demand instance - On-Demand DB Instances for the License Included model let you pay for
compute capacity by the hour your DB Instance runs with no long-term commitments. This
may be a good option for projects requiring an instance for a very short period of time during a
POC or for a development /test instance which will only be used prior to go-live.
 Reserved instance – This option differs from the on-demand in that you pay upfront for a longer
term commitment such as 1 or 3 years. Amazon RDS provides three RDS Reserved Instance
types (Light, Medium, and Heavy Utilization Reserved Instances. See the following link for a
detailed description of each type and its appropriate use:
http://aws.amazon.com/rds/reserved-instances/
 Within each type of reserved instance you can further customize the type of instance class:

o Micro DB Instance: 630 MB memory, Up to 2 ECU (for short periodic bursts), 64-bit
platform, Low I/O Capacity, Provisioned IOPS Optimized: No

Page 43 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

o Small DB Instance: 1.7 GB memory, 1 ECU (1 virtual core with 1 ECU), 64-bit platform,
Moderate I/O Capacity, Provisioned IOPS Optimized: No

o Medium DB Instance: 3.75 GB memory, 2 ECU (1 virtual core with 2 ECU), 64-bit
platform, Moderate I/O Capacity, Provisioned IOPS Optimized: No

o Large DB Instance: 7.5 GB memory, 4 ECUs (2 virtual cores with 2 ECUs each), 64-bit
platform, High I/O Capacity, Provisioned IOPS Optimized: 500Mbps

o Extra Large DB Instance: 15 GB of memory, 8 ECUs (4 virtual cores with 2 ECUs each), 64-
bit platform, High I/O Capacity, Provisioned IOPS Optimized: 1000Mbps

o High-Memory Extra Large DB Instance 17.1 GB memory, 6.5 ECU (2 virtual cores with
3.25 ECUs each), 64-bit platform, High I/O Capacity, Provisioned IOPS Optimized: No

o High-Memory Double Extra Large DB Instance: 34 GB of memory, 13 ECUs (4 virtual

cores with 3,25 ECUs each), 64-bit platform, High I/O Capacity, Provisioned IOPS
Optimized: 500Mbps

o High-Memory Quadruple Extra Large DB Instance: 68 GB of memory, 26 ECUs (8 virtual

cores with 3.25 ECUs each), 64-bit platform, High I/O Capacity, Provisioned IOPS
Optimized: 1000Mbps
Pricing Models

 Utility based model for pricing (pay for what you use)
 On demand – flat hourly rate based on hours the instance is up and running
 Reserved – up-front yearly amount + lower hourly rate as compared to the on-demand. Less
expensive option for longer term instances.
 Per GB monthly charge
 For complete details pricing, please see : http://aws.amazon.com/rds/sqlserver/#pricing

Note: Distributed refreshes - Monthly charges for deploying a database in Amazon RDS are
computed based on multiple factors including network usage. Applications deployed in
heterogeneous locations (between AWS and Merck data centers) that need scheduled data
refreshes using MViews, DB links, etc. should be understood by the application teams.

Process for setup of instance

 EEA must be signed and approved

Page 44 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

 Provide application requirements to DBA team so that proper sized environment can be built.
Memory, CPU, IOPS and storage needs.

Operational Support

Standard Merck Operational Support model does not exist for Amazon deployed environments. 

While the support model is being developed, application teams need to understand the risk
associated with support before proceeding with Amazon Deployment.

Application teams must agree to perform end to end testing of their applications if migrating from
Merck Standard platform to Amazon RDS. To avoid functional inconsistencies, all SDLC tiers must
be migrated to the new platform, when an application is being deployed in Amazon.

For further information about the Amazon RDS offering for SQL Server:

http://aws.amazon.com/rds/oracle/

http://aws.amazon.com/rds/sqlserver/

AMAZON - AWS Cloud Links and Resources

Description Link
Quick Start Guide - http://ts1.merck.com/com/cloud/Wiki/Quick%20Access%20Guide%20-
New Users using AWS %20Merck%20Lab%20Cloud.aspx
in Merck Lab Cloud
Account
AWS Tagging http://ts1.merck.com/com/cloud/Wiki/Merck%20AWS%20Tagging
Standards %20Standards.aspx
Contact AWS http://ts1.merck.com/com/cloud/Wiki/Merck%20AWS%20Enterprise
Enterprise Support %20Support.aspx
AWS - Cost Estimator http://ts1.merck.com/com/cloud/Wiki/Merck%20AWS%20Calculator.aspx
for EC2 and RDS
resources
AWS Database http://webtop.merck.com/webtop/drl/objectId/090030f980ef2ee5
Documentation
IAM Password http://ts1.merck.com/com/cloud/Wiki/Merck%20AWS%20IAM
Maintenance %20Password%20Maintenance.aspx
Cloud Authorization http://merckcloud-auth.merck.com/
portal - To get access
using ISID to Lab and
Production AWS

Page 45 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.
Platform Support Plan
Database SQL Support Plan, 1.0
DEPARTMENT/DIVISION/SERVICE AREA – TOPIC/SUBJECT/USE:
Infrastructure Engineering & Development – Database Services

Portals
RDS SQL Server http://webtop.merck.com/webtop/drl/objectId/090030f980ef743d
Provisioning
RDS - User Guide for http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLSer
SQL Server ver.html
RDS - User Guide for http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Oracle.h
Oracle tml

Page 46 of 46
The information contained herein is the sole property of Merck and is PROPRIETARY. The recipient is responsible for
safekeeping, prevention of unauthorized appropriation, use or disclosure.