Enhanced Cyber Resilience Threat Detection With Ibm Flashsystem Safeguarded Copy and Ibm Qradar
Enhanced Cyber Resilience Threat Detection With Ibm Flashsystem Safeguarded Copy and Ibm Qradar
Enhanced Cyber Resilience Threat Detection With Ibm Flashsystem Safeguarded Copy and Ibm Qradar
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Executive summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Safeguarded Copy feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
IBM QRadar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Solution overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Control path use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Data path use case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Custom log source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
IBM QRadar sample rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Custom actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Author. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
QRadar deployment model availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Terms and conditions for product documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Applicability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Commercial use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Privacy policy considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
This document describes integrating IBM FlashSystem audit logs with IBM QRadar, and the
configuration steps for IBM FlashSystem and IBM QRadar. It also explains how to use the
IBM QRadar’s device support module (DSM) editor to normalize events and assign IBM
QRadar identifier (QID) map to the events.
Post IBM QRadar configuration, we review configuring Safeguarded Copy on the application
volumes by using volume groups and applying Safeguarded backup polices on the volume
group.
Finally, we demonstrate the use of orchestration software IBM Copy Services Manager to
start a recovery, restore operations for data restoration on online volumes, and start a backup
of data volumes.
Traditional approaches to data protection work well for their intended purposes, but are
inadequate to protect against cyberattacks, which can encrypt or otherwise corrupt your data.
Remote replication for disaster recovery replicates all changes (malicious or not) to the
remote copy.
Also, data that stored on offline media or the cloud can take too long to recover a widespread
attack. Large-scale recovery can take anywhere between days to weeks, which can lead to
substantial downtime for businesses.
The new Safeguarded Copy function for IBM FlashSystem and IBM SAN Volume Controller is
designed to help businesses recover quickly and safely from a cyberattack, which helps
reduce recovery to minutes or hours.
Detecting a threat before it starts can help speed recovery even more.
IBM Security™ QRadar is a Security Information and Event Management (SIEM) and threat
management system that monitors activities and looks for signs that can indicate the start of
an attack, such as logins from unusual IP addresses or outside business hours.
Now, IBM QRadar can proactively start Safeguarded Copy to create a protected backup at
the first sign of a threat.
If an attack occurs, IBM Copy Services Manager (the orchestration software) helps identify
the best Safeguarded backup to use and automates the process to restore data to online
volumes. Because a restore action uses the same snapshot technology, it is almost instant
and much faster than the use offline copies or copies that are stored in the cloud.
2 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Scope
The focus of this document is to describe how to proactively start Safeguarded Copy to create
an immutable backup at the first sign of a threat that is detected by IBM QRadar. It also
describes the use of Copy Services Manager orchestration software to recover or restore the
backup.
As part of early threat detection, several rules are presented and a sample Python script is
provided that was used to start the Safeguarded Copy process. The document also provides
several sample control path and data path use cases.
Customers and readers are encouraged create control path and data path use cases,
customized IBM QRadar rules, and custom response scripts that are best suited to their
environment. The use cases, rules, and Python script are seen as templates and cannot be
used as-is in an environment.
The solution that is featured in the document was created by using IBM QRadar release 7.4.2
and the Safeguarded Copy feature that was introduced in 8.4.2 software release for
FlashSystem 5100, 5200, 7200, 9100/R, 9200/R, and IBM SAN Volume Controller.
For the restore or recovery of Safeguarded Copy volumes, Copy Services Manager software
release 6.3.0 was used.
All components that are described in the document, such as IBM QRadar, IBM Copy Services
Manager, and IBM FlashSystem are in same network segment. More network planning is
required if these systems are in different networks.
For more information about IBM QRadar, Safeguarded Copy, and Copy Services Manager,
see “Resources” on page 46.
Introduction
Combining the capabilities of IBM Safeguarded Copy and IBM QRadar enables enterprises to
build comprehensive cyber resilience solutions. These solutions address the Protect,
Recover, and Detect functions of the NIST framework.
IBM FlashSystem can log all administration activities in the access logs, which includes all
storage objects access information. To identify and detect potential malicious access and for
compliance auditing purposes, such access logs must be integrated with the SIEM solution.
Similar to the IBM DS8000® Safeguarded Copy solution, Safeguarded Copy helps secure
data to prevent it from being compromised (accidentally or deliberately) and allows for
recovery from protected backups if a cyberattack occurs.
3
It also provides secure, point-in-time copies or snapshots of active production data that
cannot be altered or deleted (immutable copies), and that can later be used for identification,
repair, or replacement of data that was compromised by cyber or internal attack or corrupted
by system failures or human error.
The safeguarded backups or copies of data are protected with extra security provided that is
through unique user roles with dual management control (separation of duties).
The Safeguarded Copy solution on IBM® FlashSystem family and IBM SAN Volume
Controller storage systems integrates with IBM Copy Services Manager software, starting
with Copy Services Manager version 6.3.0.1, by using its automated, built-in copy and
retention scheduling, testing, and ease of recovery capabilities. IBM Copy Services Manager
also coordinates Safeguarded Copy function across multiple systems.
IBM QRadar
IBM QRadar is a leading SIEM solution that can monitor, inspect, detect, and derive insights
for identifying potential threats to the data that is stored on IBM Spectrum Scale-managed
systems. It is one of the most popular SIEM solutions on the market today.
The SIEM solution provides powerful cyber resilience and threat detection features, such as
centralized visibility, flexible deployment, automated intelligence, machine learning, and
proactive threat hunting.
The data management and storage features of IBM Spectrum Scale, combined with the log
analysis, deep inspection, and detection of threats that are provided by IBM QRadar, offer an
excellent platform for hosting unstructured business data, reducing the effect of cyber threats,
and increasing cyber resilience.
IBM QRadar can detect malicious patterns by using several data sources and analysis tools
and techniques, including access logs, heuristics, correlation with logs from other systems
(such as network logs or server logs), network flow, and packet data, and even unknown
threat vector detection by using IBM Watson for Security resources. Its open architecture
enables third-party interoperability so that many solutions can be integrated, which makes it
even more scalable and robust.
Prerequisites
This solution includes the following prerequisites:
• A user with Administrator privileges was created on the IBM FlashSystem® or centralized
authentication, such as LDAP or Active Directory. This user can be used by QRadar®
system to securely log on to storage system by using SSH to perform various actions. It is
suggested that a qradaradmin user is created for this task.
• The public key from a user on IBM QRadar is added to the qradaradmin user that is
defined on IBM FlashSystem to set up password less authentication between IBM QRadar
and IBM FlashSystem.
• The private key of the same user from IBM QRadar is added to the
/opt/qradar/bin/ca_jails/home/customactionuser/.ssh folder to authenticate
qradaradmin user by using the public key that is shared with IBM FlashSystem.
• The firewall rules between IBM QRadar and IBM FlashSystem are adjusted to allow traffic
on 514/tcp or 514/udp.
4 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
• To use IBM Copy Services Manager, a locally or externally identified user is required. The
use of Copy Services Manager allows greater flexibility to recover or restore data from
various backup sets. It offers great flexibility for Safeguarded Copy backup/restore
management because Copy Service Manager can connect to multiple storage systems.
CLI and GUI interfaces are available for Copy Services Manager.
This document does not discuss the installation and configuration of Copy Services
Manager software. For more information, see “Resources” on page 46.
Solution overview
The solution that is presented in this publication is shown in Figure 1.
Organizations can face threats in many ways, such as compromised user credentials by using
sphere fishing attack, or a rouge user within the organization cyberattacks, such as brute
force attempts or ransomware. Any of these threats pose grave risks to storage systems that
are used for storing the data.
For administrative tasks, IBM FlashSystem allows connectivity by using several methods,
including as web login, command-line, or API calls. Every action from every connection is
logged to IBM FlashSystems audit logs.
The IBM FlashSystem was configured to forward the audit logs to IBM QRadar by using the
mksyslogserver command. By forwarding the audit log event, every administrative action is
now logged and scrutinized for activities that are performed.
To simulate a brute force attack a 3-tier architecture application that involves a web server. an
application and database server was created. The application was also configured to log
actions by using the rsyslog Linux facility. The rsyslog configuration enables auditing of all
events (application or user-related) that occur on the system
5
Various log sources were defined on IBM QRadar to classify all incoming events. To
determine Control Path Events, a log source FS91K_Stoage_LS was used with the log
source type as IBM SAN Volume Controller. To determine authentication-related events
(GUI/CLI logins), a log source FS91K_Auth_LS was used with LinuxOS as the log source
type. The storage system name acted as an identifier for every event that was generated on
IBM.
The application events were forwarded by Linux rsyslog daemon running on the web server
and the generated events were classified as Gallery-LS log source events. For more
information about rsyslog daemon configuration, see “Appendix A” on page 45.
After the events are correctly classified based on the log sources, various rules were defined
to filter events from each category. As part of rule definition, the response definition was also
created ranging to register an offense to start Safeguarded Copy action by using custom
actions. Python scripts were created and uploaded as Custom action.
To cover the control path actions and data path actions, the use cases that are described next
were considered.
Use case 1
In this use case, an attempt that was made by an administrator or lower role to delete the
Safeguarded Copy (which is blocked and fails) raises an alert. Failed attempts to remove a
volume are logged in the audit log.
Use case 2
Administrator logins are detected outside business hours. Therefore, a Safeguarded Copy of
important volumes can be generated, or the administrator user can be blocked.
Use case 3
The same FlashSystem administrator logged from a different location or IP address at the
same time. Therefore, a Safeguarded Copy of important volumes can be generated or that
administrator user can be blocked and forcefully logged out.
6 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Data path use case
For the demonstration of data path use case, a simple 3-tier that is shown in Figure 2 was
used. The application allows users to upload images upon successful login, and stores
images on application server data volume.
To track user-specific images, a database table is maintained with an image ID for every user.
When a threat is detected on the application, such as a brute force login, the failed login
events are analyzed by IBM QRadar’s rules engine, and then, it starts the Safeguarded Copy
of the application and database volumes.
Lab setup
The data path use case that is shown in Figure 2 also shows the lab setup, on which the
solution was created and tested.
Note: To run the CLI or GUI commands, log on to IBM FlashSystem as a superuser or a
user with administrators privileges.
7
Figure 3 Creating Safeguarded Copy pool
8 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
ii. Assign the safeguarded policy to the volume group. The predefined policies are
shown in Figure 5.
iii. Figure 6 shows the newly defined safeguarded copy volume group with a safeguard
policy that is assigned to it. Two application volumes are also associated with the
safeguarded copy volume group. The volume group name SafeguardedCopy_2 is
used as parameter to the custom action that is defined in IBM QRadar.
9
◦ By using GUI (see Figure 7).
For more information about adding custom action scripts to IBM QRadar, see “Resources”
on page 46.
6. Create custom log source types to normalize the audit log events that are received in
QRadar into various log sources. For more information about how to normalize events by
using regular expressions and assigning IBM QRadar identifier (QID) and create a custom
log source, see “Custom log source” on page 11.
7. Define business-compliant rules. For more information, see “IBM QRadar sample rules”
on page 21.
8. Upload the custom action script. For more information about how a custom action script is
deployed in IBM QRadar, see “Custom actions” on page 30.
9. Generate a brute force attack on the web server to trigger multiple failure events.
10.Recover or restore storage volumes from the SafeguardedCopy backup by using the Copy
Services Manager interface.
10 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Custom log source
This section describes how to work with the audit events that are received from IBM
FlashSystem.
When log forwarding is enabled on IBM FlashSystem by using mksyslogserver, IBM QRadar
starts receiving the events. The received are show in the IBM QRadar’s Log Activity window.
IBM QRadar easily parses many types of log events and assigns the log source automatically.
In specific cases, when IBM QRadar cannot automatically parse the event, the received event
is listed as Unknown events, as shown in Figure 9.
A sample event that was received from IBM FlashSystem is shown in Example 1.
Event normalization is required to create the rules definition that is based on the information
that is contained in the payload. IBM QRadar’s DSM module offers excellent flexibility to
parse the events in many formats, including JSON or events with user-defined separator, as
shown in Example 1.
Figure 10 shows the how to open a group of events in the DSM editor.
11
While opening the Unknown log events, IBM QRadar prompts the user to select a suitable
Log Source Type. Many predefined log source types are available. Figure 11 shows IBM SAN
Volume Controller is chosen as Log Source Type for the events.
When you select multiple events and choose IBM SAN Volume Controller as the Log Source
Type, the DSM editor window looks the example that is shown in Figure 12.
Figure 12 DSM Editor view with multiple non-parsed and non-mapped events
The Parsing failed status in Log Activity Preview window indicates, the IBM QRadar was
unable to extract data for the event. In such cases, user intervention is required for providing
the regular expressions to extract the required data values.
For more information about system- and custom-defined user extensions and how to match
the required event values for those extensions, see Chapter 4, “Log source extensions”, in
IBM QRadar, DSM Configuration Guide.
12 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
It is also possible to select and open a single event in DSM editor, as shown in Figure 13.
Also, a regular expression is shown to extract data for the Event ID property.
Figure 13 DSM editor with Single event parsed but not mapped to any QID
Event ID is a system property and it is populated automatically when QRadar can parse the
event. In this case, the Event ID field is overridden to populate its value by using the
user-defined regular expression.
While entering the regular expression, the matched part from the event is automatically
highlighted in yellow. The green selection that is shown in Figure 13 indicates the successful
match of event data by using the regular expression. Also, when the match is successful, data
is seen in the Log Activity Preview window for the specific field.
It is possible to create a custom property to get a value for a specific field from the event.
Clicking the [+] sign-on event properties tab starts a wizard to help define custom property, as
shown in Figure 14. The new custom property definition is started by clicking Create New.
When the event is received from IBM FlashSystem, the field cluster_user helps identify the
user whose actions triggered the event.
13
Figure 15 shows the name, field type, and description that is chosen for the custom property.
The Enable for use in Rules, Forwarding Profiles and Search Indexing option must be
selected because this property is used later when the custom rules are defined.
Clicking Save closes dialog box, and returns the control to Custom Property Definition
window.
Choose Select to work with the cluster_user custom property in the DSM editor window.
Figure 16 shows the regular expression and the match for cluster_user custom property.
As shown in the Log Activity Preview window in Figure 16, values for cluster_user, Event
Category, and Event ID were extracted by providing correct regular expressions.
14 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
The properties that are defined to hold values that are matched from each type of audit
events are listed in Table 1. The table contains System and Custom defined properties. The
word common indicates that the property is used for multiple events. (The properties that
were listed Table 1 were used for a custom log source FS91K_Storage_LS).
Table 1 System and Custom properties for Storage Events for FS91K_Storage_LS
Property name Property Regular expression Capture Storage
type group event
All login events to storage were assigned to FS91K_Auth_LS Log Source. These events are
automatically parsed and required event categories were mapped by predefined LinuxOS Log
Source Type.
For every event that is received, the Event ID must have a value. When the event is not
automatically parsed, this property must be overridden to match the required value from event
as shown before.
After the required data from all events is correctly matched, QRadar attempts to assign a
QRadar ID (QID). Every QID includes a description to help understand the event category.
For the automatically parsed events, QRadar automatically assigns the QID. For the manually
parsed events, the QID must be assigned, which is done from DSM editor’s Event Mappings
tab.
15
As shown in Table 17, the Event ID and Event Category properties are populated with the
value rmvolume.
Clicking Choose QID at the bottom of the window opens a dialog box in which the correct
QID category can be selected (see Figure 18).
16 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
A QID can also be created from this window, if required. After the correct QID is located, it is
then selected. The selected value then is assigned to Event, as shown in Figure 19.
Now, the Event is correctly parsed and includes correct QID mapping. However, the existing
events properties are not changed. It is still shown as an Unknown event. Any future events
with a similar Event ID and Event Category are automatically assigned the same QID.
QRadar uses the QID information to coalesce events that belong to same category and
displays a count that indicates how many times the event occurred, as shown in Figure 20.
After event parsing and mapping is complete, a unique Log Source is defined to identify or
filter events. This Log Source is also used when custom rules are defined in QRadar.
17
To define a new Log Source, from the Admin tab, select the Log Source option, as shown in
Figure 21.
The new Log Source wizard is started by clicking New Log Source, as shown in Figure 22.
Then, select Single Log Source in next window, as shown in Figure 23.
18 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
A wizard is started from where system-defined values were selected (see Figure 24).
The next step involves choosing the Log Source (see Figure 25).
Then, the parameters for the Log Source are selected, as shown in Figure 26.
19
The final step in the configuration process is to define the identifier for the events. The name
of the storage system is listed here (see Figure 27).
The custom log source definition is now complete. QRadar now has enough information to
parse and map future events that are based on protocol parameters.
20 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Definition or changes to a Log Source requires a Deployment task, as shown in the Admin tab
(see Figure 29).
Figure 30 Sample compliance rules for creating and removing Safeguarded Copy volumes
21
To keep track of SafeguardedCopy volumes that were created on the storage system, a
reference set was defined in IBM QRadar by clicking Admin → Reference Set
Management → Add Reference Set (see Figure 31).
The reference set is populated with volume_id custom property of the mkvolume event. The
rules creation process is shown in Figure 32 - Figure 38 on page 25.
22 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Figure 34 Wizard Welcome window
23
Figure 36 Rule Test Stack Editor window
24 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Figure 38 Rule Summary window
The rule is run for every event that matches Create Volume QID and contains bk_ text as part
of event payload. Upon running, it adds the value of volume_id custom property to
SafeguardedCopy-Volume reference set that was defined.
Similarly, when a SafeguardedCopy volume removal event is detected, the user that triggered
the event is crosschecked against users that allowed the delete action on the storage. An
offense is generated when the unauthorized users are attempting volume removal.
When the same action is run by an authorized user, no offense is generated. In both cases,
the volume_id that is added by AddSafeguardedCopyVolume-To-Ref-Set rule is removed from
SafeguardedCopy-Reference Set.
25
Notice that a custom property rmvolume_id is defined for the rmvolume event that holds the
value of the volume that is being deleted. The rule’s summary is shown in Figure 39.
Rule Description
Rule Actions
Set Severity to 6
Set Credibility to 6
Rule Responses
Figure 40 shows the SafeguardCopy-Volumes Reference Set that is defined in IBM QRadar.
26 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Sample rules to capture control path actions
Figure 41 shows a rule to manage administrator logins that are detected outside of business
hours.
Rule Description
and when the event category for the event is one of the following Authentication.Auth
Server Login Succeeded, Authentication.Admin Login Attempt, Authentication.Host Login
Succeeded
Rule Notes
Rule Actions
Set Severity to 5
Set Credibility to 5
27
Figure 42 shows a rule that manages when the same user logs on from multiple locations or
IP addresses.
Rule Description
and when the event QID is one of the following (11750041) Auth Server Login Succeeded,
(11750336) Host Login Succeeded
and when at least 2 events are seen with the same Username and different Source IP in 5
minutes
Rule Notes
The rule attempts to detect multiple logins by same username but from different IP
addresses in 5 minutes
Rule Actions
Set Severity to 5
Set Credibility to 4
Force the detected Event to create a NEW offense, select the offense using Source IP
Annotate this offense with: Same user login from multiple locations
Annotate the Event with: Same user logged from multiple locations
28 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Sample rule to capture data path actions
Figure 43 shows a rule that applies multi-app login failures for a single username.
Rule Description
Apply Multiple App Login Failures for Single Username on events which are detected by
the Local system
and when the event QID is one of the following (50253724) Login failed via web
and when at least 10 events are seen with the same Username in 5 minutes
Rule Notes
Reports authentication failures for the same username
Rule Actions
Force the detected Event to create a NEW offense, select the offense using Username
Annotate this offense with: Multiple Login Failures for the Same User
Rule Responses
Dispatch New Event:
◦ Event Name: Multiple Login Failures for the Same User
◦ Event Description: Detected multiple (10) authentication failures for the same user
name in a 5 minute period.
◦ Severity: 4 Credibility: 7 Relevance: 7
◦ High-Level Category: Authentication
◦ Low-Level Category: User Login Failure
◦ Force the dispatched event to create a NEW offense, select the offense using
Username
Execute Custom Action
Rule Limiter
Respond no more than 3 times per 30 minutes per Rule
29
Custom actions
To add a custom action to IBM QRadar, choose the Admin option from the menu bar, and
then, select Custom Actions in the Data Sources section, as shown in Figure 44. Then,
click Add in the Custom Actions window.
Define the Custom Action, as shown in Figure 45. A Fixed property or Network Event
property also can be added. The Network Event property also can be a value that is extracted
from an event.
30 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Example 2 shows the sample Python script to start the Safeguarded Copy action.
Note: The script that is shown in Example 2 is provided as a sample reference and used
for demonstration purposes. Customers can modify it based on their needs.
import os
import re
import sys
import subprocess
import datetime
def usage():
msg = """
Usage: {0} <storage_system> <volume_group_name>
where
storage_system = storage system FQDN or IP address
volume_group_name = volume group on which safeguardedbackup policy has been
applied (case sensitive)
NOTE: Unset the simulation variable if the script has been exeuted in
simulation mode before.
1. export simulation=True
Once all the csv files are created, re-run the python script.
31
To get an accurate simulation refresh all the CSV files with latest data by
executing above commands on storage.
"""
sys.exit( 1 )
Parameters:
- IN
1. cli command to run
- OUT
1. output of cli command
"""
lst_cmd = []
lst_cmd.append( 'ssh' )
lst_cmd.append( '-o StrictHostKeyChecking=no' )
lst_cmd.append( remote_usr + "@" + svc_system )
lst_cmd.append( cli_cmd )
try:
# Command execution using subprocess
stdout = subprocess.check_output( lst_cmd,
universal_newlines = True,
shell = False )
if stdout != None:
return stdout
except KeyboardInterrupt:
print( "User abort ..\n" )
sys.exit( 1 )
except subprocess.CalledProcessError:
print( "Error connecting to remote host !! aborting !!! \n")
sys.exit( 1 )
Parameters:
32 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
- IN
1. Comma delimited detailed output of all mdisks available on system
- OUT
1. string value conaining name of the safeguarded pool
"""
if line != None:
try:
key = line.split(",")[0]
val = line.split(",")[1]
if key == 'name':
sgc_pool_name = val
return sgc_pool_name
except IndexError:
pass
return None
Parameters:
- IN
1. all_vdisks : Comma delimited detailed output of all vdisks
available on system
2. vol_group : Volume Group
3. sgc_pool_name : The SafeguardedCopy Pool defined on the system.
This is different
than volumegroup definiation with similar name.
- OUT
1. dictionary : Containing following record.
33
charasteristics as vol_id
"""
vols = {}
volgrp_matched = False
ctr = 0
if line != None:
try:
key = line.split(",")[0]
val = line.split(",")[1]
if key == 'id' :
vol_id = val
iogrp = val
size = val
volgrp_matched = True
pref_node = val
if volgrp_matched :
epoch = datetime.datetime.now().strftime('%s')
bk_vol_name = 'bk_' + str(epoch) + '_' + str(ctr)
mkvol = '-pool {0} -iogrp {1} -size {2} -unit b -preferrednode
{3} -name {4}' . format(sgc_pool_name, iogrp, size, pref_node, bk_vol_name)
vols[vol_id] = mkvol
ctr += 1
volgrp_matched = False
vol_id = ""
iogrp = ""
size = ""
volgrp_matched = False
pref_node = ""
34 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
except IndexError:
pass
return vols
Prameters:
- IN
1. Comma separated lines with newline
- OUT
String containing fcconsistency group name
"""
global simulation
35
print( "Step 1 : Creating Flashcopy Consistency group : {0} " .
format(fcconsistgrp_name) )
if not simulation:
runCli( mkfcconsistgrp_cmd )
else:
print( 'Simulating command => {0} \n' . format(mkfcconsistgrp_cmd) )
print( '-' * 30 )
print( '\n')
if not simulation:
# From above output retrieve the volume_id of the newly created volume
for line in output.strip().split("\n"):
x = line.split(" ")[2].replace('[','').replace('],','')
print( "Step 3 : Creating fcmap per source [{0}] and target [{1}] volume
" . format(vol_id,bk_vol_id) )
runCli( mkfcmap_cmd )
else:
36 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
mkfcmap_cmd = ( 'svctask mkfcmap -source {0} -target {1} -consistgrp {2}
-copyrate {3} ' \
. format( vol_id, bk_vol_id, fcconsistgrp_name, copyrate ) )
print( "Step 3 : Creating fcmap per source [{0}] and target [{1}] volume
" . format(vol_id,bk_vol_id) )
print( 'Simulating command => {0} \n' . format(mkfcmap_cmd) )
print( "Step 4: Starting copy sequence per volume using {0} flashcopy
consistency group" . format(fcconsistgrp_name) )
if not simulation:
runCli( startfcconsistgrp_cmd )
else:
print( 'Simulating command => {0} \n' . format(startfcconsistgrp_cmd) )
print( '-' * 30 )
print( '\n')
Parameters:
try:
f = open(csvfile,'r')
data = f.read()
f.close()
except FileNotFoundError:
return data
def main():
37
# before creating target volumes.
# Step 1 :
# - Get the first safeguard pool defined on the system
# - Get all the vdisks defined on the system
# - Get all the fcmaps defined on the system
if not simulation:
if not vdisks_per_volgroup:
print("Abort !! No volumes are protected with volume group {0}\n" .
format(vol_group) )
38 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
sys.exit(1)
if __name__ == '__main__':
argc = len(sys.argv) - 1
if simulation:
print(" *** Simulation {0} *** " . format(simulation) )
main()
39
Brute force login attack generation
This section describes how the attack that was used in our lab was created on the application
to simulate brute force login attempts.
Example 3 shows a sample script that was used to generate false logins for the applications.
The script performs a POST operation with username and password fields that are populated
with false data. When the login fails, a Login failed via web event is generated and sent to
IBM QRadar. All the Login failed via web events were tagged with the Gallery keyword to
uniquely identify them.
The series of Login failed via web events that are received by IBM QRadar act as a trigger
for running the multi-app-login-failure rule. This rule, in turn, starts the predefined custom
action mksgcbkup.py script. The script logs on to the IBM FlashSystem and runs a series of
operations to generate a Safeguarded Copy. This execution flow is shown in Figure 46 and
Figure 47 on page 41.
40 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Figure 47 Safeguarded Copy available as part of Cyber-Resiliency workflow
New or existing IBM FlashSystem customers who are interested in the use of the
Safeguarded Copy function that do not have Copy Services Manager can obtain this software
by using the IBM Copy Manager for IBM Spectrum Virtualize software bundle. Contact your
IBM account representative or IBM Support for more information about how to obtain the IBM
Copy Services Manager software.
This section discusses the use of Copy Services Manager software, which is stand-alone and
available free of charge. It is used to back up, recover, or restore the backups that are made
by using the SafeguardedCopy functions.
The Safeguarded Copy volumes are immutable in nature. Therefore, they cannot be mapped
to a host (even to view the data that is contained in them). A recovery or restore operation is
required to work with the data.
When recovery option is chosen, a volume is created and the data is copied from the
Safeguarded Copy volumes. After the data copy process is complete, the relation is broken.
The new volumes now include the data from Safeguarded Copy volumes and are available in
read/write mode.
When the restore option is chosen, the source volumes are overwritten with data from the
Safeguarded Copy volumes.
41
A local storage user or a centralized user must be available for connection to the storage
system. For our lab setup, a local storage user, csmadmin, was used for authentication, as
shown in Figure 48.
Figure 49 shows the FlashSystem that was registered in Copy Services Manager.
For the Copy Services Manager, an active session with storage system is required to start a
backup, restore, or recover, as shown in Figure 50.
42 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
A list of backups that are available for restoring or recovering can be obtained by selecting
one of the sessions that are listed, as shown in Figure 51.
The recovery or restore operation is possible on backup sets that are listed with a Yes value in
the Recoverable column. The Recover Backup Info tab lists all the backups that were restored
or recovered, as shown in Figure 52.
Summary
The solution that is described in this paper shows the integration of IBM QRadar and IBM
FlashSystem SafeguardedCopy feature. This integration helps with early threat detection and
creating an instantaneous immutable copy of a single volume or group of volumes.
IBM QRadar can receive events from various sources, normalizes them, and uses the data
from these events to run various rules to detect any type of anomaly. The rules also are used
to trigger a wanted response on the detected threat.
The IBM FlashSystem Safeguarded Copy feature is used to create immutable copies of
volumes that are based on a defined safeguarded policy or in an ad hoc manner.
The solution that is described in this paper also shows control and data path use cases and
the rules that are associated with them.
The paper also describes the configuration steps to enable IBM FlashSystem Safeguarded
Copy feature, event processing in IBM QRadar, and the use of Copy Services Manager to
restore or recover the wanted volumes.
43
The other artifacts that are provided in this paper, such as Python script or IBM QRadar
sample rules, were tested in the IBM Lab. No guarantee is given that these artifacts work as
when they are deployed in customer environment. Readers are encouraged to create their
own response scripts or rules by reviewing the samples scripts and rules that are provided in
this paper.
Author
Shashank Shingornikar is a Storage Solutions Architect with IBM Systems, ISDL Lab Pune,
India, for the past 12 years. He has worked extensively with IBM Storage products, such as
IBM Spectrum Virtualize, IBM FlashSystems, and IBM Spectrum Scale building solutions that
combine Oracle and Redhat OpenShift features. Currently, he is working on demonstrating
cyber resilience solutions with IBM QRadar and IBM Storage Systems. Before joining IBM,
Shashank worked in The Netherlands on various high availability, Disaster Recovery, cluster,
and replication solutions for database technologies, such as Oracle, MSSQL, and MySQL.
Acknowledgments
The author wishes to thank the following people for their support in the production of this
publication:
Oiza Dorgu
Yves Santos
Julio Cesar Hearnandez
Sandeep Patil
Pradip A. Waykos
Hemant Kantak
Hemanand Gadgil
Chris Daniel
Ajinkya Nanavati
Mandar Vaidya
IBM Systems
Sridhar Muppidi
Adam Frank
Boudhayan Chakrabarty
Ashish M Kothekar
Praphullachandra Mujumdar
Prateek Jain
IBM Security
44 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Appendix A
This section describes the configuration that was created for rsyslog daemon on the web
server.
A configuration file that is specific to the application was created in the /etc/rsyslog.d folder
with the configuration options that are shown in Figure 53 (the rsyslog version that was used
for the configuration was rsyslog-8.24.0-57).
module(load="imfile" PollingInterval="5")
input(type="imfile"
File="/var/log/gallery_error.log"
Tag="gal"
Severity="error"
Facility="local4")
45
Resources
For more information, see the following resources:
• Configuring User Roles on IBM FlashSystem:
http://www.ibm.com/docs/en/flashsystem-9x00/8.4.0?topic=overview-user-roles
• Adding custom actions to IBM QRadar:
http://www.ibm.com/docs/en/qsip/7.4?topic=tasks-adding-custom-actions
• IBM Copy Services Manager:
https://www.ibm.com/docs/en/csm
46 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
47
48 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Notices
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
Redbooks (logo) ® IBM FlashSystem® IBM Watson®
DS8000® IBM Security™ QRadar®
IBM® IBM Spectrum®
The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.
Other company, product, or service names may be trademarks or service marks of others.
50 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Terms and conditions for product documentation
Permissions for the use of these publications are granted subject to the following terms and conditions.
Applicability
These terms and conditions are in addition to any terms of use for the IBM website.
Commercial use
You may reproduce, distribute and display these publications solely within your enterprise provided that all
proprietary notices are preserved. You may not make derivative works of these publications, or reproduce,
distribute or display these publications or any portion thereof outside your enterprise, without the express
consent of IBM.
Rights
Except as expressly granted in this permission, no other permissions, licenses or rights are granted, either
express or implied, to the publications or any information, data, software or other intellectual property
contained therein.
IBM reserves the right to withdraw the permissions granted herein whenever, in its discretion, the use of
the publications is detrimental to its interest or, as determined by IBM, the above instructions are not being
properly followed.
You may not download, export or re-export this information except in full compliance with all applicable
laws and regulations, including all United States export laws and regulations.
This Software Offering does not use cookies or other technologies to collect personally identifiable
information.
If the configurations deployed for this Software Offering provide you as customer the ability to collect
personally identifiable information from end users via cookies and other technologies, you should seek your
own legal advice about any laws applicable to such data collection, including any requirements for notice and
consent.
For more information about the use of various technologies, including cookies, for these purposes, see IBM’s
Privacy Policy at http://www.ibm.com/privacy and IBM’s Online Privacy Statement at
http://www.ibm.com/privacy/details in the section entitled “Cookies, Web Beacons and Other Technologies,”
and the “IBM Software Products and Software-as-a-Service Privacy Statement” at
http://www.ibm.com/software/info/product-privacy.
Notices 51
52 Enhanced Cyber Resilience Threat Detection with IBM FlashSystem Safeguarded Copy and IBM QRadar
Back cover
August 2021
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Please recycle
ISBN 0738459941
REDP-5655-00