Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

Alibaba Cloud Apsara Stack Solution: Version 2018/4/9

Download as pdf or txt
Download as pdf or txt
You are on page 1of 39

Alibaba Cloud

Apsara Stack Solution

Version 2018/4/9 I
Index
Index ..................................................................................................................................................... II
1 Executive Summary ........................................................................................................................ 1
2 Solution Overview .......................................................................................................................... 2
2.1 Solution Benefit ................................................................................................................ 2
2.2 Solution Architecture ........................................................................................................ 2
3 Apsara Stack Solution Description ................................................................................................. 2
3.1 Apsara Stack Core Capabilities ........................................................................................ 2
3.1.1 Apsara Operating System ...................................................................................... 2
3.1.2 Network Architecture ............................................................................................ 3
3.1.3 Security Architecture ............................................................................................. 4
3.1.4 Deployment and Control System (Tianji) ............................................................. 5
3.1.5 Unified O&M management system ....................................................................... 5
3.2 Compliance Security Solution .......................................................................................... 5
3.2.1 Cloud-based classified protection and compliance................................................ 5
3.2.2 Classified protection implementation process ....................................................... 8
3.3 ECS Service ...................................................................................................................... 8
3.3.1 Service Description................................................................................................ 8
3.3.2 Key Features of ECS ............................................................................................. 9
3.3.3 Security ................................................................................................................ 10
3.4 VPC ................................................................................................................................ 11
3.4.1 Service Description.............................................................................................. 11
3.4.2 Key Features of VPC ........................................................................................... 12
3.4.3 Security ................................................................................................................ 13
3.5 OSS ................................................................................................................................. 14
3.5.1 Service Description.............................................................................................. 14
3.5.2 Key Features of OSS ........................................................................................... 15
3.5.3 Security ................................................................................................................ 16
3.6 SLB ................................................................................................................................. 16
3.6.1 Service Description.............................................................................................. 16
3.6.2 Key Features of SLB ........................................................................................... 17
3.6.3 Security ................................................................................................................ 18
3.7 RDS ................................................................................................................................ 18
3.7.1 Service Description.............................................................................................. 18
3.7.2 Key Features of RDS ........................................................................................... 19
3.7.3 Security ................................................................................................................ 19
3.8 Alibaba Cloud Security .................................................................................................. 20
3.8.1 Service Description.............................................................................................. 20
3.8.2 Key Features of Alibaba Cloud Security ............................................................. 21
4 Business Continuity and Disaster Recovery Solution .................................................................. 28
4.1 Apsara Stack Intra-city Disaster Recovery Solution ...................................................... 28
4.2 SLA ................................................................................................................................. 29
4.3 Apsara Stack Service DR Architecture........................................................................... 29

Version: 2018/4/9 II
1 Executive Summary

1.1 Customer Information


1.2 Project Background

1
2 Solution Overview
Alibaba cloud proposed solution includes the following product:
Ÿ Apsara Stack Enterprise Edition
Alibaba Cloud’s Apsara Stack Enterprise Edition solution is a comprehensive private
cloud platform based on Alibaba Cloud’s distributed architecture that enables enterprise
customers to deploy and operate Alibaba Cloud services in their on-premises data center.

2.1 Solution Benefit


Apsara Stack Enterprise Edition private cloud solution can help the customer rapidly realize
the benefits of private cloud IT infrastructure solutions. We provide:
Ÿ Hyper-scale Cloud
Apsara Stack supports hyper scale cluster systems containing more than 6,000 servers per
region to meet the requirements of operating large business systems and applications visited
by a high number of users. Users can efficiently operate and manage those resources,
applications, and data on the Apsara Stack platform through a unified management portal,
which supports multi-tenancy mode.
Ÿ Comprehensive Alibaba Cloud Services On-premises
Apsara Stack provides comprehensive cloud services such as hardware, computing, storage,
network, data base service and Alibaba security products in data center, and can be deployed
in either hybrid or private cloud mode, to meet the requirements of various applications and
solutions.
Ÿ Reliable and Secure
Apsara Stack incorporates end-to-end security design and disaster recovery mechanisms to
ensure a high standard of security and reliability. Combined with strong data analysis ability
and a professional operations team, Apsara Stack IT security products provide users with total
data security protection. Alibaba Cloud has passed multiple international certifications
including ISO 27001 Information Security Management and Trusted Cloud certification.
Ÿ Professional Services
Provides customers with professional services including workload migration, live event
support, application consulting, and technology management to ensure secure and stable cloud
business

2.2 Solution Architecture


Apsara Stack Enterprise Edition private cloud solution includes:
Ÿ Data center hardware deployment
Provides hardware architecture design, planning and deployment.
Ÿ Apsara Operating System
Apsara Operating System is the infrastructure layer to support for computing, storage,
network, resource scheduling, hardware management and resource abstraction. The
supported capabilities include hardware management and virtualization, security

2
management, resource management, large-scale distributed file systems, job scheduling,
and coordination services.
Ÿ Cloud Service Layer
Service Layer provides multiple cloud services to meet customer requirements including
SLB (load balancing service), ECS (computing service), OSS (storage service), VPC
(tenant network), RDS for MySQL (relational database service).
Ÿ Security Framework
Apsara Stack Security provides security solutions for cloud platform and cloud product
at the design level. For tenant security, Alibaba Cloud Security provides Basic Edition
and Advanced Edition. The Basic Edition is composed of three main function modules:
network traffic monitoring system, host intrusion protection system (Server Guard), and
security auditing. The Advanced Edition includes all the functions of the Basic Edition,
in addition to DDoS Cleaning, Cloud Firewall, WAF, Situation Awareness, and other
functions.
Distributed deployment/Unified management and O&M
The unified O&M management system includes the cloud service consoles and O&M
monitoring console. The consoles enable you to perform the operations such as account
management, distribution of cloud services resources, alerts handling, system upgrading,
and audit management.
Ÿ Backup and disaster recovery solution
Apsara Stack offers utilities and resource to build scalable, durable and secure backup
and restore solution to meet RTO, RPO, data retention and compliance requirements
which, based on network conditions, can achieve RTO and RPO in a matter of minutes
or hours. In additional, Apsara Stack provides cloud-based intra-city disaster recovery
solution to ensure business continuity. Business Continuity Management Center aka.
BCMC is a management utility to switch failure data center with low RPO to disaster
data center in Apsara Stack
Figure 2-1 Apsara Stack Overview Architecture

3
4
Apsara Stack Enterprise

Version: 2018/4/9
Apsara Stack Enterprise

3 Apsara Stack Solution Description

3.1 Apsara Stack Core Capabilities


3.1.1 Apsara Operating System
The Apsara Operating System provides computing support for storage, computing, scheduling, and
other aspects of upper-level services. The modules that the Apsara Operating System contains the
following main features:
Ÿ hardware management services
Provide resource management services for underlay hardware and interact with upper-level services
Ÿ Distributed file systems
Provide a massive, reliable, high availability, scalable data storage service that aggregates the storage
capabilities of the nodes in a cluster.
Ÿ Task scheduling
Provide automatically scheduling for tasks running in Apsara Operating System such as ECS
provisioning, Storage creation, cluster etc. Task scheduling is intelligent engine based on multiple
schedule policies and make resource management smart.
Ÿ Cluster monitoring
Monitor the resource status of Apsara Stack clusters and performance indicators of application
services. Cluster monitoring captures and reports errors and abnormal events timely.
Ÿ Deployment
Provide the operation and maintenance personnel with the deployment and configuration
management for the entire Apsara platform and support online expansion of clusters as well as
service upgrade.
Figure 3-1 Apsara Stack Operating System Architecture

Version: 2018/4/9
Apsara Stack Enterprise

3.1.2 Network Architecture


Apsara Stack physical network is designed to be flexible, high bandwidth, easy extensible and
reliable and meets enterprise-level business requirements.
The network architecture has the following basic characteristics:
Two-layer physical network is provisioned in Apsara Stack architecture. Redundancy ASW is
deployed in fundamental layer to connect to underlying hardware as well as aggregates traffic into
DSW layer via Layer-3 protocol.
Figure 3-2 Apsara Stack Network Architecture

Version: 2018/4/9
Apsara Stack Enterprise

3.1.3 Security Architecture


Apsara Stack offers a security architecture in two layers, that is, the platform layer and user layer.
The security architecture of platform layer, including the underlying platform and cloud product
security architecture, emphasizes the control over the system. And the security architecture of user
layer emphasizes the user-level security policy.
Figure 3-3 Apsara Stack Security Architecture

Version: 2018/4/9
Apsara Stack Enterprise

3.1.4 Deployment and Control System (Tianji)


Tianji provides a unified deployment, verification, authorization and control framework for cloud
service. The framework provides a unified access mechanism and dependency management for cloud
services.
Ÿ The resource library stores deployment configuration and execution files for all cloud services
and dependent components.
Ÿ provide access control capabilities for cloud services and support multi-tenant isolation.
Ÿ Leverage interface gateway to provides a secure API management.
Ÿ Log services provide log records collection, store, retrieval, .
Ÿ The control module monitors the health status of cloud services.

3.1.5 Unified O&M management system


The unified O&M management system includes the cloud service consoles and O&M monitoring
console. The consoles enable user to perform the operations such as account management, resources
allocation, alerts handling, system upgrade and audits.
Figure 3-4 Apsara Stack Unified O&M management system architecture

3.2 Compliance Security Solution


3.2.1 Cloud-based classified protection and compliance

Ÿ Shared compliance responsibilities


The Alibaba Cloud platform and cloud tenant systems should be respectively rated and assessed.
Assessment conclusions of the Alibaba Cloud platform can be reused by tenant systems in
assessment.
Alibaba Cloud provides the following contents:
Version: 2018/4/9
Apsara Stack Enterprise

Ø Classified protection archival filing certificate of the Alibaba Cloud


Ø Key pages of the Alibaba Cloud assessment report
Ø Sales license of the Alibaba Cloud shield
Ø Description of partial test items of the Alibaba Cloud
Detailed interpretations on shared responsibility are as follows:
Ø Alibaba Cloud is the unique cloud service provider in China to participate in and pass
demonstration of classified cloud computing protection standard. The public cloud and
e-Government cloud pass class III protection archival filing and assessment. The financial cloud
passes class IV protection archival filing and assessment.
Ø When tenant systems on Alibaba Cloud pass classified protection assessment, physical security,
partial network security and security management conclusions can be used and Alibaba Cloud
can provide explanations according to conclusion reuse rules issued by the supervision
authorities.
Ø Complete security technologies and management architecture of the Alibaba Cloud platform
and Alibaba Cloud Security protection system facilitate tenants to pass classified protection
assessment better.
Ÿ Figure 3-5 Shared compliance responsibilities

Version: 2018/4/9
Apsara Stack Enterprise

Ÿ Classified protection and compliance ecology


Current conditions of classified cloud-based protection
Ø Most tenants do not know classified protection.
Ø Most tenants do not know how to start with classified protection.
Ø Most tenants are not good at communication with supervision authorities.
Ø Security systems lag behind business development.
To facilitate cloud-based systems to quickly pass classified protection assessment, the Classified
Protection and Compliance Ecology is established on Alibaba Cloud to provide one-stop classified
protection and compliance solution.
Work division of classified protection:
Ø Alibaba Cloud: Integrate capabilities of service agencies and provide security products.
Ø Consulting firm: Provide technical support and consulting services in the whole flow.
Ø Assessment agency: Provide assessment services.
Ø Public security authorities: Provide archival filing auditing, supervision and check services.
Figure 3-6 Classified protection and compliance ecology

Version: 2018/4/9
Apsara Stack Enterprise

3.2.2 Classified protection implementation process


The classified protection implementation process is shown as below:
Figure 3-7 Classified protection implementation process

3.3 ECS Service


3.3.1 Service Description
ECS is computing service deployed over Apsara Stack certificated hardware and hypervisor platform
with extraordinary stability and performance. ECS provides comprehensive functions more than just
a computing pool. Compared with other computing pool solution, the ECS has the benefits in the
following aspects:
Ÿ Elastic Computing
Elasticity of Alibaba Cloud is a combination of elastic computing, storage, network, and the
elasticity to redesign business architecture.
Ÿ Completed Control
Complete control of ECS instances including root access and the ability to interact with them. The
ECS instances can managed and controlled by using web service APIs.
Ÿ Flexible Cloud Hosting Services
ECS allows user to select a configuration of memory, CPU, instance storage, and the boot partition
size that is optimal for the choice of operating system and application.
Ÿ Integration
Version: 2018/4/9
Apsara Stack Enterprise

ECS is integrated with most Apsara Stack services such as OSS, RDS, and VPC) to provide a
complete, secure solution for computing, query processing, and cloud storage across a wide range of
applications.
Ÿ Reliability
Large-scale redundancy architectures guarantee the availability of running instances and the
reliability of data stored in cloud disks. High data reliability and service availability. Instance
availability up to 99.95% and cloud disk data reliability no less than 99.9999999% Automatic
downtime migration and data backup. Automatic downtime migration and automatic snapshot
backup (manual configuration of snapshot policies required) make data recovery simple
Ÿ Security
Server security through Alibaba Cloud Server Guard provides such features as interception against
brute-force password attacks, Trojan scans, remote logon reminders, and anti-intrusion against
high-risk vulnerability repair.
Ÿ Monitoring
CloudMonitor guarantees service security through a range of real time alert and notification services.

Figure 3-8 ECS Architecture

3.3.2 Key Features of ECS

Ÿ Flexible Instance Configuration

Version: 2018/4/9
Apsara Stack Enterprise

Supports multiple instance generations, dozens of instance types (ranging from 1-core 1 GiB to
32-core 128 GB).
Ÿ Multiple regions creation
Allows instance creation in all regions
Ÿ Abundant image resources
Provides various image resources, including public images, custom images, and shared images,
allowing quick operating system deployment and applications without installation
Ÿ Abundant image resources
Supports multiple Windows and Linux operating systems.
Ÿ Multiple storage methods
Provides three types of data storage disks (Basic Cloud Disks, Ultra Cloud Disks, and SSD Cloud
Disks) and I/O-optimized instances.
Ÿ Convenient management
Provides multiple management methods, including the console, VNC, and APIs, ensuring complete
control.
Ÿ Multiple level resource monitoring
Site Monitoring: Provides statistics-collection, monitoring, and alert notifications for availability
and response time for services including http, ping, dns, tcp, udp, smtp, pop and ftp
Cloud Service Monitoring: Provides cloud service monitoring and alert notifications, as well as a
custom monitoring service to allow users to customize their personalized monitoring needs
Alert and Contact Management: Provides uniform batch management for alert policies and alert
notifications through a range of channels including text message, email, and interface callback

3.3.3 Security
Ÿ Image security
Ø Regular fixing of high-risk vulnerabilities
Ø Built-in host intrusion prevention software
Ÿ Hot upgrades
Ø Hot upgrades for Linux kernel hosts
Ø Hot upgrades for Hypervisor
Ÿ Tenant isolation
Ø Hypervisor is isolated from the CPUs, memory, and storage of different virtual machines.
Ø Tenant networks are isolated through VPCs and security groups.
Ø All stored data are cleared after memory and storage are released.
Ÿ Reliability
Ø Distributed redundant storage system ensures data reliability.
Ø Quick backup and rollback is provided based on disk snapshots.
Ø Point-in-time Recovery is provided based on failover deployment.
Ø Smart resource scheduling is provided based on online migration.
Version: 2018/4/9
Apsara Stack Enterprise

Ø Availability reaches 99.95%.


Ÿ Network security
Ø VPC (based on VxLAN) isolation
Ø Virtual firewalls with status detections, divided security domains
Ø Protection against IP and MAC spoofing and Address Resolution Protocol (ARP) spoofing
Ø Protection against network sniffing
Ÿ Host security
Ø Tenants have maximum permissions and Alibaba Cloud does not have any login permission.
Ø Linux supports SSH Key certificates to prevent brute-force attacks.
Figure 3-9 ECS server security architecture

3.4 VPC
3.4.1 Service Description
Virtual Private Cloud (VPC) is a private network service established in Apsara Stack. VPCs are
logically isolated from other virtual networks. VPCs allow you to launch and use Apsara Stack
resources in your VPC. You have full control over your VPC. For example, you can select its IP
address range, further segment your VPC into subnets, as well as configure route tables and network
gateways. Additionally, you can connect VPCs with an on-premises network using a physical
connection or VPN to form an on-demand customizable network environment. This allows you to
smoothly migrate applications to the cloud with little effort. the VPC has the benefits in the
following aspects:
Ÿ Network Elasticity
In Apsara Stack VPC, all network configurations and offline IDC configurations can be the same,
and more possibilities are allowed. Interconnection and security domain isolation between data
centers can be realized, and all network configurations and planning in the VPC are flexible.
Ÿ Software Defined Network (SDN)
SDN provides customized network configurations to extend and manage complex network
infrastructure easily as well as provides complete traffic isolation between cloud tenants.
Ÿ Interconnection connection methods
Version: 2018/4/9
Apsara Stack Enterprise

Apsara Stack provides more secure method to set up interconnection of different DCs.
Ÿ Internet Access
VPC Gateway to provide secure method for ECS instance accesses the Internet in VPC including EIP
associated or NAT gateway
Figure 3-10 VPC architecture

3.4.2 Key Features of VPC


Ÿ Private IP Address Range
The IP address range can be specified by user for each VPC. Alibaba Cloud VPC provides the
following IP address ranges for use.
Table 3-1 CIDR

VPC CIDR Block The number of available private IPs (excluding system reserved IPs)
192.168.0.0/16 65,532
172.16.0.0/12 1048,572
10.0.0.0/8 16,777,212
Ÿ VSwitch
VSwitch is a basic network device in a VPC. It is used for connecting different cloud product
instances. After creating a VPC, you can create one or more subnets in the VPC by creating
VSwitches. Different VSwitches in a VPC can communicate with each other through the intranet. A
VPC contains at least 1 VSwitch and can be up to 24 VSwitches.
Ÿ CIDR Block
When creating a VSwitch, the private IP address range of the VSwitch must be specified in the form
of Classless Inter-Domain Routing (CIDR) block.
Note the following when specifying the VSwitch CIDR block:

Version: 2018/4/9
Apsara Stack Enterprise

Ø The CIDR block of the VSwitch can be the same as that of the VPC to which it belongs, or a
subset of the VPC CIDR block.
Ø The size of the subnet mask for the VSwitch can be /16 to /29, and the VSwitch CIDR block can
provide 8 to 65536 IP addresses.
Ø The first and last three IP addresses are reserved by the system.
Ø Consider the number of cloud instances to be created in the VSwitch. Up to 15000 instances can
be created in a VPC.
Ø If the VSwitch has to communicate with a VSwitch in another VPC or a local network, make
sure the CIDR block of the VSwitch does not conflict with that of the resource to connect.
Ÿ VRouter
A VRouter is a hub in the VPC that connects all VSwitches in the VPC and serves as a gateway
device that connects the VPC to other networks. VRouter routes the network traffic according to the
configurations of route entries.
Ÿ Route Entry
Each entry in a route table is a route entry. A route entry specifies the next hop address for the
network traffic destined to a CIDR block. It has two types of entries: system route entry and custom
route entry. There are two types of route entries:
Ø System route entry is a type of route entry with the destination CIDR block is added by the
system when you create a VPC. This allows for communication between cloud product
instances in the VPC. Additionally, a route entry is added for each VSwitch by the system when
you create a VSwitch. The destination CIDR block of this system route entry is the CIDR block
of the VSwitch.
Ø Custom route entry is route entry that you add to route specific traffic to a specified destination.

3.4.3 Security
Ÿ Security isolation
Different VPCs are isolated by tunnel IDs. Using VSwitches and VRouters, VPC can be segmented
into subnets similar to that in the traditional network environment. Different cloud servers in the
same subnet communicate with each other by using VSwitch, and cloud servers in different subnets
within a VPC communicate with each other by using VRouters. Layer 2 networks between different
VPCs are isolated. ECS instances within a VPC use a security group firewall to control the network
access.
Currently, the Virtual Private Cloud (VPC) in Alibaba Cloud does not comes with a dedicated
resource access management policy. Resource access management in the VPC relies on the access
control capabilities of each cloud product. For example, resource access management for ECS is
implemented using security groups, and that for SLB and RDS is implemented using whitelists.
Ÿ Access control
Ø ECS Security Group
A security group is a virtual firewall that provides the stateful packet inspection feature. Security
groups are used to set network access control for one or more ECSs. An important means of security
isolation, security groups are used to divide security domains on the cloud.
Ø SLB Whitelist
To configure the whitelist, add the user’s IP addresses or the cloud service IP addresses inside the
VPC to be accessed over SLB to the access management whitelist of SLB.
Ø RDS Whitelist
Version: 2018/4/9
Apsara Stack Enterprise

Using the whitelist feature of ApsaraDB for RDS, the user can customize IP addresses that are
allowed to access the RDS. All access from unspecified IP addresses are denied. When using the
RDS products in a VPC, add the IP address of the ECS to the whitelist for the required RDS so that
the ECS can visit the RDS instance.

3.5 OSS
3.5.1 Service Description
Alibaba Cloud Object Storage Service (OSS) is an easy-to-use storage pool service that enables you
to store, backup and archive large amounts of data on the cloud. OSS acts as an encrypted central
repository from where files can be securely accessed in the cloud. OSS is a massive, secure,
cost-effective and highly reliable cloud storage service. Compared with traditional user-created
server storage, OSS has many outstanding advantages in the reliability, security, cost, and data
processing capabilities. OSS enables you to store and retrieve unstructured data including text files,
images, audios, and videos. The benefit of OSS as below:
Ÿ Reliability
Ø Three copies of objects stored in OSS
Ø 99.95% designed service availability and 99.999999999% designed data persistence.
Ø Automatic scaling without affecting external services and automatic redundant data backup
Ÿ Flexibility
Ø Using multiple ways such as standard RESTful APIs, SDKs, client tools and a dedicated
console to manage massive volumes of data from website or application
Ø Provides multiple streams of data writing and reading
Ø Deletion of expired/old data objects in batches or transfer to low-cost archiving service
Ø Accelerates development speed and reduces development costs by providing C language
SDKs for direct connection with OSS.
Ÿ Scalable
Offers scalable and unlimited object based storage capacity and supports high concurrency
Ÿ Data Processing Capabilities
Ø Provides high throughput read-and-write access.
Ø Supports multiple file formats such as jpg, png, bmp, gif, web page, tiff etc.
Ø Content delivery acceleration with OSS as origin site and ensures stability with no retrieve
from the origin bandwidth restrictions.
Ÿ Image Processing
Supports thumbnails, cropping, watermarking, compression, format conversion and other image
processing functions for stored images.

Figure 3-11 OSS Architecture

Version: 2018/4/9
Apsara Stack Enterprise

3.5.2 Key Features of OSS


Ÿ Object
An object is a discrete unit of data. An object’s lifecycle starts from when it has been successfully
uploaded, and ends when it has been deleted. During an object’s lifecycle, its information cannot be
changed. If uploading an object with a duplicate name in a bucket, it overwrites the existing one.
Therefore, unlike the file system, OSS does not allow users to modify only part of an object.
An object is composed of:
Ø Metadata, known as Object Meta, which is a key-value pair that expresses the object’s
attributes, such as its last modification time and size, and user-defined information.
Ø User data, known as Data.
Ø A unique object name, known as a Key.
Ÿ Bucket
A bucket is a virtual division of object storage that, unlike file systems, manages objects in a flat
structure. Bucket properties are as follows:
Ø All objects must belong to a bucket, and during an object’s lifecycle it remains directly
affiliated with the corresponding bucket.

Version: 2018/4/9
Apsara Stack Enterprise

Ø The user can create multiple buckets and each of buckets can contain an unlimited number
of objects.
Ø The user can configure the attributes of a bucket for region, object access control and object
lifecycle management.

3.5.3 Security
Ÿ Tenant isolation
Ø Tenant data are separated with tags.
Ø The service access layer uses symmetric encryption key authentication technology to
identify users.
Ÿ Reliability
Ø Distributed redundant storage ensures data reliability.
Ø Availability reaches 99.9%.
Ÿ Access Control
Ø Access control is implemented through the access control list (ACL).
Ø Access is controlled based on Resource Access Management (RAM) authorization policies.
Ÿ Encrypted transmission
Ø Supports SSL transmission encryption
Ø Supports encrypted storage on the server side

3.6 SLB
3.6.1 Service Description
Alibaba Cloud Server Load Balancer is a traffic distribution control service. It distributes the
incoming application traffic among multiple ECS (Elastic Compute Service) instances according to a
scheduling algorithm and listening rules. By setting a virtual IP address, Server Load Balancer
service virtualizes the ECS instances located in the same region into a high-performing and highly
available application service pool. Client requests are distributed to the cloud server pool according
to the defined listening rules. This increases the fault tolerance of your applications. Server Load
Balancer checks the health status of the ECS instances in the cloud server pool and automatically
isolates any ECS instances with an abnormal status. This resolves the single point of failure (SPOF)
problem and improves the overall service capability. The benefits of SLB as below:
Ÿ High Availability
Server Load Balancing automatically distributes traffic across multiple targets – Apsara Stack ECS
instances, containers and IP addresses – in a single data center or multiple data centers.
Ÿ Secure
SLB works with VPC to provide robust security features, including integrated certificate management
and SSL decryption. Together, they give the flexibility to centrally manage SSL settings and offload CPU
intensive workloads from applications.
Ÿ Elastic
Elastic Load Balancing is capable of handling rapid changes in network traffic patterns. Additionally,
deep integration with Auto-scaling service ensures sufficient application capacity to meet varying
levels of application load without requiring manual intervention.
Ÿ Flexible
Version: 2018/4/9
Apsara Stack Enterprise

SLB allows user to use IP addresses to route requests to application targets. This offers flexibility in
how to virtualize the application targets, allowing user to host more applications on the same
instance.
Ÿ Robust monitoring
SLB allows user to monitor applications and their performance in real time with CloudMonitor
metrics, logging, and request tracing. This improves visibility into the behavior of the applications,
uncovering issues and identifying performance bottlenecks in the application stack at the granularity
of an individual request.

Figure 3-12 SLB architecture

3.6.2 Key Features of SLB


Ÿ Protocol
Alibaba Cloud provides both layer-4 (TCP and UDP) and layer-7 (HTTP and HTTPS) load
balancing services.
Ÿ Health check
Through health check on backend ECS instances, Server Load Balancer can automatically block
abnormal ECS instances and distribute requests to them when they become normal.
Ÿ Session persistence
Server Load Balancer supports session persistence. You can set listening rules to forward a session
request from a client to the same backend ECS instance during the session lifecycle.
Ÿ Scheduling algorithms
Server Load Balancer supports the following scheduling algorithms:
Ø Round robin: Requests are distributed across the backend ECS servers sequentially.
Ø Weighted least connections (WLC): The servers with a higher weight value will receive a larger
percentage of live connections at any one time. If weights are the same, the system directs
network connections to the server with the least established connections.
Version: 2018/4/9
Apsara Stack Enterprise

Ÿ Access control
Set a whitelist to control which IP addresses can access Server Load Balancer.
Ÿ Certificate management
Server Load Balancer service provides Certificate Management for the HTTPS protocol listening.
With Certificate Management, you do not need to upload certificates to backend ECS instances.
Deciphering is performed on Server Load Balancer to reduce the CPU overheads of backend ECS
instances.
Ÿ Instance type
You can choose to create an Internet or Intranet Server Load Balancer service. The system will
assign a public IP address or private IP address accordingly.
Ÿ Management methods
Server Load Balancer instances can be managed via various methods, such as the Server Load
Balancer console, Open API, and SDK.

3.6.3 Security
Ÿ DDoS Attack Protection
Combined with Alibaba Cloud Security, Server Load Balancer can defend against up to 5 Gbps
DDoS attacks, such as HTTP flood and SYN flood attacks.
Ÿ SLB Whitelist
whitelist to control which IP addresses can access the load balancing service
Ÿ Server certification
Server Load Balancer supports load balancing HTTPS applications and provides a certificate
management function. Use server certificate from Alibaba Cloud Security Certificate Service, or
from other service providers and upload certificate to the Server Load Balancer certificate
management system. This option is only available for HTTPS listeners

3.7 RDS
3.7.1 Service Description
ApsaraDB for RDS (Relational Database Service) is a stable and reliable online database service, and
it also supports elastic scaling function. Based on the Apsara distributed system and
high-performance storage of ephemeral SSD, It offers a complete set of solutions for backup,
recovery, monitoring, migration, disaster recovery, and troubleshooting database operation and
maintenance. ApsaraDB for MySQL proves to have excellent performance and throughput.
ApsaraDB for MySQL also offers a range of advanced functions including optimized read/write
splitting, data compression, and intelligent optimization. The benefits of ApsaraDB for RDS as
below:
Ÿ Single Deployment
RDS specifications can be customized through the APIs. RDS generates the specified instance
immediately.
Ÿ Ease of management
Alibaba Cloud is responsible for ensuring the normal operation of RDS through routine maintenance
and management, such as hardware/software fault processing and database update patches. The

Version: 2018/4/9
Apsara Stack Enterprise

customer can independently perform database addition, deletion, restart, backup, recovery, and other
management operations in the Alibaba Cloud console.
Ÿ Effortless migration
RDS is used similarly to the native database engine, meaning that it is easy to transfer the
pre-existing knowledge and skills to RDS management. Data can be migrated to RDS using the
commercial off-the-shelf data import and export tools with minimal labor cost required.
Ÿ On-demand upgrades
Along with changes in the database load and data storage capacity, it is flexible to adjust the
instance types, and RDS will not interrupt the data link service during the upgrade period.
Ÿ Transparent and compatible
The use method of RDS is the same as that of the native database engine. In addition, RDS is
compatible with other programs and tools. Data can be migrated to RDS using a data import and
export tool with minimal labor required.

3.7.2 Key Features of RDS


Ÿ Data link service
The data link service mainly provides data operations, including adding, deleting, modifying, and
querying table structures and data.
Ÿ High-availability service
The high-availability service is mainly designed to ensure the availability of the data link service. It
is also responsible for handling internal database exceptions.
Ÿ Backup service
The backup service supports offline data backup, dump, and recovery.
Ÿ Monitoring service
The monitoring service tracks the status of services, networks, operating systems, and instances on
which RDS depends operate properly.
Ÿ Migration service
The migration service helps you migrate data from self-built databases to RDS.

3.7.3 Security
Alibaba Cloud RDS offers a variety of security reinforcement features to secure user data, including
but not limited to:
Ÿ IP address whitelist
provides the IP address whitelist feature to implement access control for network security
Ÿ Virtual Private Cloud (VPC)
A VPC is a private network environment in the public cloud, which strictly isolates users’ network
packets with the underlying network protocol and implements access control at Layer 2. By using the
VPN or private line, use a custom RDS IP address segment of the VPC to resolve IP address
conflicts, and access RDS instances from both server and Alibaba Cloud ECS instance.
Ÿ Secure Sockets Layer (SSL)
RDS provides Secure Sockets Layer (SSL) for MySQL. Use the server root certificate provided by
RDS to verify whether the database service with the target IP address and port is provided by RDS,
which can effectively prevent man-in-the-middle attacks. To guarantee security and validity, RDS
enable and update the SSL certificates for servers.

Version: 2018/4/9
Apsara Stack Enterprise

Ÿ Storage backup and recovery


Ø Backup
RDS regularly backs up database data to guarantee data integrity, reliability, and restorability.
RDS provides the following two backup functions: data backup and log backup
Ø Recovery
Data recoverability is a key indicator of database reliability. RDS provides the following three
recovery functions: recover by backup set ID, recover by time, overwrite and recover
The recovery functions are closely related to the backup functions:
o The early time for data recovery depends on the previous data backup, which is related to
the data backup frequency and the data expiration policy.
o The late time for data recovery depends on the next log backup, which is closely related to
the log volume.
o The speed of data recovery depends on the data backup frequency and is also closely
related to the log volume.

3.8 Alibaba Cloud Security


3.8.1 Service Description
Alibaba Cloud Security is a key component that Alibaba Cloud uses to protect the security of
customers' business systems. It takes advantage of the strong data analysis capabilities of Alibaba
Cloud's cloud computing and the professional Alibaba Cloud security operations team, to provide a
multi-level integrated cloud security solution for data centers. Alibaba Cloud Security of Apsara
Stack provides security solutions for cloud platform and cloud product at the design level. For tenant
security, Alibaba Cloud Security provides Basic Edition and Advanced Edition.
Ÿ The Basic Edition is composed of three main function modules: network traffic monitoring
system (Beaver), host intrusion protection system (Server Guard), and security auditing.
Ÿ The Advanced Edition includes all the functions of the Basic Edition, in addition to DDoS
Cleaning, Cloud Firewall, WAF, Situation Awareness, and other functions. Together with
Alibaba Cloud's professional security operations services, Alibaba Cloud Security provides
users with an all-in-one security assurance product, including intrusion protection, security
audits, situational awareness, and centralized management.
Figure 3-13 Alibaba Cloud Security Architecture

Version: 2018/4/9
Apsara Stack Enterprise

Table 3-2 Detailed list of Alibaba Cloud Security functions

3.8.2 Key Features of Alibaba Cloud Security


Ÿ Pioneer in cloud security
Alibaba Cloud Security protects over 37% of Chinese websites. Each day, Alibaba Cloud Security
defends against more than half of the large-traffic volume attacks in China. On a daily basis, Alibaba
Cloud Security identifies and defends against 35,000 malicious IP addresses. Over the past year,
Alibaba Cloud Security has helped users fix over 1.4 million vulnerabilities.
Version: 2018/4/9
Apsara Stack Enterprise

Figure 3-14 Alibaba Cloud processes massive volumes of Internet data

Ÿ Network traffic monitoring


The network traffic monitoring module is a self-developed Alibaba Cloud Security product able to
monitor attacks within milliseconds. By performing in-depth analysis on the traffic packets mirrored
from the Apsara Stack portal, it can detect various attacks and abnormal behaviors in real time and
coordinate with the other protection modules to implement defenses. The network traffic monitoring
module provides a wealth of information output and basic data support throughout the Alibaba Cloud
Security defense system.
Table 3-3 Network traffic monitoring

Module Function Description


Network Traffic DDoS attack Uses traffic mirroring to detect DDoS attacks that bypass detection at
monitoring Detection the cloud boundary.
Traffic statistics Calculates the traffic usage of cloud products to generate
traffic diagrams.
Network-layer web Conducts network-layer interception and bypass blocking for
attack interception common web attacks based on embedded web matching rules.
IP blacklist Blocks bypass TCP attacks from IP addresses on the black list.
Malicious host Recognizes and warns against malicious hosts within the cloud
recognition computing platform.
Ÿ Host Intrusion Protection (Advanced Edition)
The Host Intrusion Protection (Advanced Edition) module deploys agents on ECS instances to
collect data for centralized detection and analysis on the server-end. The host intrusion protection
(Advanced Edition) module has the following functions:
Table 3-4 Host Intrusion Protection

Module Function Description


Host intrusion Webshell detection Rule matching and dynamic analyses are employed to precisely detect
Protection and removal and remove backdoors and Trojans on ECS instances.
Basic malicious file Relying on the malicious file sample library, the module detects,
detection and removal removes, and isolates malicious files and processes on servers.
Remote login alerts The module analyzes and records users' normal login locations, to
identify common login regions (precise to the city). If suspected
non-administrator system logins are detected, it sends out text message
alerts.
Suspicious account By analyzing server administrator behavior information, the module

Version: 2018/4/9
Apsara Stack Enterprise

Detection can detect suspicious accounts and provide corresponding warnings.


Brute force The module can detect and intercept brute-force password cracking
password cracking attempts in real time, and monitor brute-force cracking behaviors for
interception SSH, RDP, FTP, MySQL, SQL Server, and other common services in
Windows and Linux environments.
Vulnerability scan Based on host scanning, the module discovers host vulnerabilities and
provides methods for repairing them.
Vulnerability repair The module can repair for ECS applications and fix some high-risk
Windows system vulnerabilities with the one-click solution. This
feature applies to web application vulnerability fixes, system file fixes,
and other vulnerability fixes.
Security baseline The module collects audit records for all operations performed by the
host.
Ÿ Vulnerability Scanning
Vulnerability Scanning is a self-developed security module designed to scan web applications built
on ECS instances for vulnerabilities. This module is developed on the basis of the stateless scan
technology. In coordination with the network traffic security module, it relies on a combination of
dynamic detection and static matching scanning modes to provide you with automated, high
performance, and precise web vulnerability scanning capabilities. The vulnerability scanning module
has the following functions:
Table 3-5 Vulnerability Scanning

Module Function Description


Vulnerability Common web Scans SQL injection, XSS, file inclusion, code execution,
analysis Vulnerability scanning information leaks, and other vulnerabilities.
Exclusive third-party Scans vulnerabilities in common third-party web application
vulnerability scanning components, such as Discuz, WordPress, DedeCms, and PHPCms.
Weak password scanning Supports weak password scanning for common systems, applications,
and databases, including RDP, FTP, SSH, MySQL, SQL Server, and
MongoDB.
Malicious webpage link Scans webpages to discover maliciously implanted leeching links.
Scanning

Ÿ Security auditing
The security auditing module is an integrated solution based on the cloud computing platform. This
module meets the basic requirements for information system security classified protection. It
operates on the physical server level, the network equipment level, and the cloud computing platform
application level to provide behavior log collection, storage, analysis, and alarm functions. The
security auditing module provides the following functions:
Table 3-6 Security Auditing

Module Function Description


Security Network Auditing Audits logins and operations on cloud platform network devices.
Audit Physical server Audits logins and operations on cloud platform physical server.
Auditing Audits calls to the platform's internal APIs and command execution
operations.
Cloud platform Audits the calls and operations of each of the cloud platform's internal APIs.

Version: 2018/4/9
Apsara Stack Enterprise

auditing
Cloud product Audits logins on ECS instances.
auditing Audits operations on ApsaraDB for RDS database.
Audits operations on MaxCompute instances.

Ÿ DDoS Cleaning
Alibaba Cloud relies on its self-developed, large-scale, distributed operating system and more than a
decade of defense experience to provide a wide range of cloud platform users with its Alibaba Cloud
Security DDoS attack protection product, designed and developed based on its cloud computing
architecture. The DDoS cleaning module provides the following functions:
Table 3-7 DDoS Cleaning

Module Function Description


DDoS Cleaning Massive volume DDoS Effectively defends against SYN Flood, ACK Flood, ICMP Flood,
cleaning UDP Flood, NTP Flood, SSDP Flood, DNS Flood, HTTP Flood,
and HTTP Flood attacks.
Application layer Protects the application layer against DDoS attacks.
DDoS protection Uses re-certification, identity recognition, verification
codes, and other techniques to precisely distinguish
between malicious access and authentic access. This
provides defenses against HTTP Flood attacks on
websites and games.
Elastic scaling Coordinates with the Alibaba Cloud Anti-DDoS Service
to expand DDoS attack protection capabilities to a
maximum of over 1,000 Gbit/s.

Ÿ Web Application Firewall


Web Application Firewall (WAF) is an Alibaba Cloud's proprietary website security protection
product and can protect website applications against attacks against common web vulnerabilities.
Such attacks can be SQL injection, XSS, and other common web application attacks, or HTTP Flood
attacks and other attacks that affect website availability by consuming resources. At the same time,
WAF allows user to develop precise protection policies for the business characteristics of your
website, so as to filter the malicious web requests against website. The WAF module secures the
business traffic on HTTP and HTTPS websites. On the WAF management interface, user can import
certificates and private keys at discretion for all link business encryption, and eliminate the
possibility of traffic monitoring. Meanwhile, this also satisfies security protection needs for HTTPS
services.
The WAF module's protection system is divided into two main parts:
Ø Alibaba Cloud's big data analysis platform: As a threat intelligence library and a website
reliability model relying on Alibaba Cloud's core big data capabilities, this module allows
user to easily distinguish between normal and abnormal traffic.
Ø Security policies focusing on attack profile matching and statistical analysis: WAF's
built-in general protection rules can usually detect and block common web vulnerability
attacks among OWASP's top 10 attacks. Moreover, it provides precise protection capabilities

Version: 2018/4/9
Apsara Stack Enterprise

that allow user to customize your own protection policies for your website. These policies
allow user to filter specified malicious web request traffic.
The WAF module also supports rule sorting in protection scenarios and allows user to adjust the
relationships between precise protection and other security protection policies. Precise protection
allows user to add custom protection policies, which always are given the highest priority during
request matching.
Table 3-8 Web Application Firewall

Module Function Description


Web Application Protection against The module defends against common OWASP threats
Firewall common web , and provides high, medium, and low protection
attacks rules policies to meet the needs of different website
businesses regarding common HTTP GET and POST
requests, in order to defend against SQL injection,
XSS, Webshell uploads, command injection, invalid
HTTP protocol requests, attacks on common web
server vulnerabilities, unauthorized access to core
files, and path traversal. It also provides backdoor
isolation protection, defense scans, and other security
protections.
HTTP Flood attacks Controls frequent access from a single source IP
defense addresses, provides redirect jump verification, and
determines if access requests come from a human or
machine.
Uses a combination of precise access control filters to
control requests with abnormal Referrer and User-Agent
fields, protects against massive slow request attacks, and identifies
abnormal response codes, abnormal IP
access, and abnormal URL distributions.
Fully utilizes Alibaba Cloud's advantages in big data security to
establish threat intelligence and trusted access analysis models.
This allows you to quickly identify malicious traffic.
Precision access Provides a friendly configuration console interface and
control supports condition combinations for common HTTP fields,
including IP, URL, Referrer, and User-Agent. This allows you to
create powerful precision access control policies that support
scenarios such as anti-leeching and website management
background protection.
Integrated with the security modules for protection against
common web attacks and HTTP Flood attacks, WAF establishes
comprehensive multi-layer protection to easily distinguish between
trusted and malicious traffic based on your actual needs

Ÿ Cloud Firewall

Version: 2018/4/9
Apsara Stack Enterprise

The cloud firewall module is Alibaba Cloud's proprietary cloud access control system developed to
meet needs of east-west traffic micro-isolation in a cloud computing environment. The cloud firewall
module provides the following functions:
Table 3-9 Cloud Firewall

Module Function Description


Cloud Micro isolation Effectively defends against SYN Flood, ACK Flood, ICMP Flood, UDP Flood,
Firewall NTP Flood, SSDP Flood, DNS Flood, HTTP Flood, and HTTP Flood attacks.
Visualization Protects the application layer against DDoS attacks.
Uses re-certification, identity recognition, verification
codes, and other techniques to precisely distinguish
between malicious access and authentic access. This
provides defenses against HTTP Flood attacks on
websites and games.
Role + Tag Developed on the basis of the IP-free (intellectual property free) asset
based asset definition, this module is more aligned to your business, for role/tag-based
definitions asset management.
Distributed No traffic routing is required. The distributed architecture helps you eliminate
architecture the troubles caused by traffic routing through the cloud computing platform's
centralized firewall once for all, and can be adapted for various virtualization
environments.
Ÿ Situation Awareness
Situation Awareness is a big data security analysis platform that uses machine learning and
data modeling to find potential infiltration and attack risks. From the attacker's perspective, it
effectively captures 0day vulnerability attacks mounted by advanced attackers, new virus attacks,
and ongoing security attacks. It also effectively presents this information, keeping aware of business
security in a visual way. This solves the problem of data leaks due to cyberattacks and allows user to
discover the hacker's identity using the tracing service. Situation Awareness provides the following
features:
Ø Big data threat analysis
Situation Awareness uses big data analysis methods along with intelligent machine learning
and molding analysis to detect new threats and security trends facing data center cloud computing
users, including attacks on web applications, brute-force system cracking, hacker intrusions, and
application and host layer vulnerabilities.
Ø Big screen presentation
Situation Awareness intuitively presents the results of big data threat analysis as graphs on a big
screen. This serves as a tool supporting cloud computing platform security decision making.
function modules is shown in the diagram below:
Figure 3-15 Architecture of Situation Awareness

Version: 2018/4/9
Apsara Stack Enterprise

Version: 2018/4/9
Apsara Stack Enterprise

4 Business Continuity and Disaster Recovery Solution

4.1 Apsara Stack Intra-City Disaster Recovery Solution


At Alibaba cloud, we work hard to make sure that our services are always available to our customer
when the customer need them. Forces beyond our control sometimes impact us in ways that cause
unplanned service disruptions. Apsara Stack offers end-to-end intra-city disaster recovery Solution to
ensure continuity, recoverability, and availability of data accesses, protect production systems
against irrecoverable damage due to various natural and man-made disasters, and deliver
enterprise-level disaster recovery services. Compared from traditional DR solution on virtual
machine level failover, Apsara Stack disaster discovery solution is focusing on data and business
high availability. We recommend that business data is stored in Apsara Stack data services such as
RDS or OSS rather than in ECSs, and the stateless application is deployed in ECS cluster. By
leveraging high availability capability of RDS and OSS in dual sites, the data can be recovered
rapidly when failure occurs.
Figure 4-1 Intra-city Disaster Recovery Overall Architecture

Version: 2018/4/9
Apsara Stack Enterprise

4.2 SLA
SLA is essentially the promise the customer about how long a system will remain unavailable during
an emergency. Apsara Stack Disaster Recovery solution SLAs are made up of Latency of Round
Trip Time, Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs):
Ÿ Latency of Round Trip Time is less than 0.6ms
Ÿ RTO is the measure of how long the systems can be offline during a disaster. This is the amount
of time it would take to bring the standby systems online in DR solution. The metric of RTO
begins with switchover and ends to online of all services, and promises 10 minutes in Apsara
Stack solution.
Ÿ RPO is a measure of the amount of data that can be lost to a disaster. The RPO will be the point
to which the cloud services will have all data up to that point recovered. The details of service
RTO lists as below:
Table 11-1 RPO for Apsara Stack Services

Service RPO (Forced Description


switchover)
SLB 0 minute When failure occurs in primary site, network
traffic is automatically switched to DR site in
seconds
RDS At the second level In general, Switchover is at the second level.
(MySQL two Note, No RTO guarantee from Apsara Stack from
Replicas the following:
mode) Bandwidth does not meet data sync requirements
and the amount of data that is getting written
suddenly is huge when recovery
RDS 0 minute This solution is designed to have Four data copies
(MySQL four stored in dual data center (2+2 deployment) and
Replicas guarantees strong consistency of data. Whether
mode) failure of primary or secondary DC, switchover of
recovery will be triggered forcedly.
OSS At the minute level In general, Switchover is at the minute level.
Note, No RTO guarantee from Apsara Stack from
the following:
Bandwidth does not meet data sync requirements
and the amount of data that is getting written
suddenly is huge when recovery

4.3 Apsara Stack Service DR Architecture

Ÿ SLB Disaster Recovery architecture

Version: 2018/4/9
Apsara Stack Enterprise

Failover of SLB is designed in an active/active mode for disaster recovery. Master and API is the
controller to serve the management of SLB service. Master&API controls SLB service switchover
automatically at the second level when the failure of primary data center.
Figure 4-2 SLB Intra-City Disaster Recovery Architecture

Ÿ OSS Disaster Recovery Solution


Bucket Cross-Region Replication enables automatic and asynchronous replication of objects across
buckets in different OSS data centers, which synchronizes changes to objects in the source bucket to
the target bucket. This feature could be a boon to customers looking for cross-site disaster recovery
for their buckets or data replication. Objects in the target bucket are precise copies of objects in the
source bucket. They have the same object name, metadata, and content.
Figure 4-3 OSS Intra-City Disaster Recovery Architecture

Version: 2018/4/9
Apsara Stack Enterprise

Ÿ ECS Controller Disaster Recovery Solution


ECS Controller is core service to perform the management of ECS instances such as task scheduling
ECS lifecycle management , resource monitoring and so on. The components of ECS controller are
designed with high availability across data centers. Even though one region is broken, business
continuity of ECS services is still guaranteed without any interruption.
Figure 4-4 ECS Controller Intra-City Disaster Recovery Architecture

Version: 2018/4/9
Apsara Stack Enterprise

Ÿ RDS Disaster Recovery Solution

Mode 1:
RDS for MySQL cluster is configured to master/slave mode. Master node is deployed in primary site
and slave node is deployed in secondary site. The data is replicated from primary to secondary data
center by using MySQL binlog semi-synchronous replication. This solution is a classic disaster
recovery configuration from MySQL and guarantees the metric of RPO at the second level.
Figure 4-5 Mode-1 Architecture

Mode 2:
RDS for MySQL cluster is configured to 4 replica nodes mode. One master and one slave nodes are
deployed in primary site, another two slave nodes are deployed in secondary site. The data is
replicated in all nodes by using MySQL binlog semi-synchronous replication. The data must be
replicated in all the slave nodes before transaction is committed. This replication mechanism
guarantees strong consistency of the data and transactions and service RPO is 0 minute.
Figure 4-6 Mode-2 Architecture

Version: 2018/4/9
Apsara Stack Enterprise

Ÿ BCMC (Business Continuity Management Console)


BCMC is a utitlity of business continuity for switching business from failure data center to
reduntancy data center in Apsara Stack. BCMC is supporting the switchover of core services
including SLB, RDS, OSS
Ø One-click switchover from failure DC to reduntency
Ø Manage disaster recovery of service by BCMC console
Ø A set of configuration scripts deployed in dual data centers for disaster recovery management
Ø Provide the functions of status check, serivce monitoring, log tracking

Version: 2018/4/9

You might also like