Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
Expanding and Deepening
NTT DOCOMO’s private cloud
NTT DOCOMO Inc. Jun Ishii
Kojiro Amano
VirtualTech Japan Hiromichi Ito
DOCOMO, INC All Rights Reserved
Jun Ishii
o Research Engineer, NTT DOCOMO
o Developer, operator and technical consultant in NTT
DOCOMO private cloud
Hiromichi Ito
o CTO, VirtualTech Japan
o One of the first members of proposing OpenStack Bare
Metal Provisioning (currently called "Ironic")
Kojiro Amano
o Research Engineer, NTT DOCOMO
o Security consultant in NTT DOCOMO private cloud
About us
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
Expanding strategy of
our private cloud
Scale-up
strategy
DOCOMO, INC All Rights Reserved
o One year after launched our private cloud,
it goes larger and larger!
Jun. 2015 Oct. 2016 Mar. 2017
Number of DCs 1 2 4
Number of HWs 50 300 900
Cores 1500 10000 Over 35000
DOCOMO, INC All Rights Reserved
o Why could we expand our cloud so fast ?
o Main Strategy : Forest and Tree
Make a forest
Fill in trees
DOCOMO, INC All Rights Reserved
o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update
DOCOMO, INC All Rights Reserved
o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update
DOCOMO, INC All Rights Reserved
o Decided to migrate a large scale in-house system
 Update whole system due to HW EOL
o OpenStack-based cloud has many strengths.
 Three years TCO is superior to an on-premises
 Reduce 22% TCO, CAPEX & OPEX
 Distributed architecture is compatible with cloud.
 REST interfaces are suitable for maintaining systems.
 Feasibility of migration/replication between long distance
 L2GW, details are mentioned in later.
DOCOMO, INC All Rights Reserved
o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update
DOCOMO, INC All Rights Reserved
o Fast deployment methods by normalization
 DRY : Don't repeat yourself
o How to
 deploy
 set up [compute node, swift storage node] more
 deal with hardware trouble
 ansible, KB (See our Tokyo summit presentation)
o Only take one month to deploy new DC
 From after racking and cabling ends till finish first QA test
 Over 300 nodes, HW configuration settings & OpenStack install are just
finished in 10 days by 5 operators.
DOCOMO, INC All Rights Reserved
o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update
DOCOMO, INC All Rights Reserved
o Novel challenges… to satisfy various users' will, there are many
difficulty and many know-how.
 Add functions
 L2GW
 GPU instance
 Reduce time to construct and manage users' systems
 Reference model
 Security update
DOCOMO, INC All Rights Reserved
o Three important details
 Decided to migrate a large scale in-house system
 Obtain budget for making a forest
 Fast deployment methods by normalization
 Enable to create the forest quickly
 Novel challenges for various users
 Enrich the forest to plant various trees
• L2GW
• GPU instance
• Reference model
• Security update
These reasons enable our private cloud so FaT !!
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
How deepening our private cloud
Enrich
for users
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
L2 Gateway
for
connecting existing large-scale networks
and inter-cloud networking.
I'm good at
connecting.
DOCOMO, INC All Rights Reserved
○ Overview
 Our user has large scale existing network and proprietary computer
systems.
– This network system has the great ability that provides Layer-2 connectivity
to nationwide.
– This proprietary computer system side does not have enough flexibility.
 REST API
 Service mobility
 They decided to migrate to OpenStack on this renewal timing.
 Network system side migration must be minimal.
 Our user requested new two network services.
– Connect the tenant network between the two datacenters
– Connect instance and existing equipment with the layer-2
DOCOMO, INC All Rights Reserved
○ Before
DC 1
RTT 20-40ms
WAN
(Nationwide)
DC 2
dedicated
equipment
dedicated
equipment
dedicated
equipment
dedicated
equipment
proprietary
computer
system
proprietary
computer
system
DOCOMO, INC All Rights Reserved
○ After
DC 1
RTT 20-40ms
WAN
(Nationwide)
DC 2
dedicated
equipment
dedicated
equipment
dedicated
equipment
dedicated
equipment
L2
GW
RT
L2
GW
RT
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF)
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF).
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
We choose "Region" zoning model.
In "Region" model all service is separated correctly.
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF).
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
Our base OpenStack deployment model avoid SPOF already.
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF).
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
Existing system's IP addressing and routing architecture can
deploy on the overlay network.
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF).
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
NAT is must technique for floating IP and connecting the external network.
But, This system does not request the floating IP address function.
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF).
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
Our base OpenStack deploy model is using L3 ECMP fabric and VXLAN.
So, We choose VXLAN Layer 2 Gateway(L2 Gateway).
DOCOMO, INC All Rights Reserved
○ Our user's requirements
 High Availability
– Do not share control services between data center.
– Avoid Single-Point-Of-Failure (SPOF).
 Technical limitations
– Do not change IP addressing and routing architecture.
– Do not use Network Address Translation(NAT).
– Must connect instance and existing equipment by Layer 2.
 Performance
– Total throughput requirement is several dozen Gbps.
– Average packet size is smaller than general private clouds.
– Several hundred VLAN must connect.
Software based VXLAN L2 Gateway does not match short packet workload.
So, We choose using hardware based VXLAN L2 Gateway.
DOCOMO, INC All Rights Reserved
○ Equipment selection
 Hardware VTEP
– Modern L3 switch chipset has Hardware VTEP gateway functions.
 Intel FM6000, Broadcom Trident II, Trident II+, Tomahawk
– We tried to examine both Intel FM6000 and Broadcom Trident II based L3
network switch.
– Finally, we compared three vendors' L3 switches. (A, D, J)
 Comparison result
– Vendor A's L3 switch can support VXLAN within a Multi-Chassis LAG
(MLAG) deployment. Other vendors can not. (as of June 2016)
– All vendors' L3 switches cleared performance criteria.
– All vendors OVSDB protocol support has some issues.
We choose vendor A's L3 switch. Because they support MLAG.
DOCOMO, INC All Rights Reserved
○ Software test and proof-of-concept
 Test target
– Neutron networking-l2gw
 API's and implementations to support L2 Gateways in Neutron.
– Networking-l2gw provides "L2GW Service Plugin" and "L2GW Agent".
 "L2GW Service Plugin" provides L2GW API services.
 "L2GW agent" controls L3 switch by OVSDB protocol.
 Test results
– Several minor bugs (Already fixed by the community.)
– Missing of features that is required for the production environment.
 SSL support (Already implemented by the community.)
 Handling Mcast_Macs_Remote table (We created modified patch for
vendor A based on community patch, not merged yet.)
DOCOMO, INC All Rights Reserved
○ Controller Node
Neutron Server
L2GW Service
Plugin
API
ML2
L3
Nova
Keystone
Glance
Cinder
Horizon
ML2 L2POP
Compute Node
ML2 OVS
Agent
Open vSwitch
VTEP
Virtual
Switch
OVSDB
Server
ML2 L2POP
A’s Management Virtual appliance
Network Node
L2GW
Agent
ML2 OVS Agent
Open vSwitch
VTEP
L3
Agent
A’s
Hardware VTEP
WAN
Hardware
VTEP
Virtual Router
VLAN VLAN
OVSDB
Server
OVSDB
protocol
OVSDB
Server
Control
DOCOMO, INC All Rights Reserved
○ Result of pilot tests with as scale as production environment
 Hardware L2GW side
– OVSDB Server crash issues
 When inserting a large number of record at one time, OVSDB server has
crashed. (This issue already fixed by the vendor.)
 Networking-L2GW side
– We encountered several critical bugs.
 But It is hard to reproduce.
 When hit these bugs, L2GW agent stopped.
– L2GW agent recovery from a crash state is terrible.
 L2 gateway agent always syncs state between neutron database and
OVSDB.
 Unfortunately, when L2GW agent crashed or stalled, these two databases
sometimes lost sync.
 So, We must re-register L2GW connections manually when met these bugs.
DOCOMO, INC All Rights Reserved
o Network trouble occurred without missing a week.
 The L2GW agent is unstable.
 The l2gw agent does not work correctly after few days when users run a long
test.
• That test includes continuous instance creating and deleting.
• That test includes continuous CRUD testing for neutron virtual network
port.
 The instance could not communicate another region instance and existing
equipment.
DOCOMO, INC All Rights Reserved
o We decided to not use Networking-l2gw in the production
environment for the time being.
 We could not reproduce connection troubles between OVSDB and the
L2GW agent that occurred a weekly.
 We could not fix all critical bugs that we encountered.
 In our environment, a port status issue occurred.
 That issue will cause L2GW agent problem.
 We would not like to use the l2population.
 The l2population does not have enough scalability yet.
Keep to a delivery date.
 Delivery date delay. aka. Our project death.
DOCOMO, INC All Rights Reserved
o Our solution
 We created a manual procedure to manage L2GW.
 Manage an OVSDB by CLI
• "vtep-ctl" command
 Manage an OVS flow table by CLI
• "ovs-ofctl" command
 We created an automation system based on the manual procedure.
 The upper management system calls the system.
This system is working correctly now.
DOCOMO, INC All Rights Reserved
o Results
 We provide stable L2GW which is connecting the two regions instances
and existing equipment.
 We passed all test criteria provided from the client.
 Gets excellent service flexibility by the OpenStack.
 The existing network configuration was kept that our user requested.
o Next challenges of our L2GW project
 Fix known issues of networking-l2gw.
 We would like to provide OpenStack API for managing L2GW.
 Fix scalability issue of l2population.
 Investigate EVPN for expanding L2GW services.
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
GPU instance
Machine learning
on OpenStack
DOCOMO, INC All Rights Reserved
Nvidia Tesla M40
for CUDA/cuDNN
DOCOMO, INC All Rights Reserved
o Deploy
 To deploy GPU nodes, not only PCI pass-through functions but also
IOMMU functions must be enabled.
 Enable PCI pass-through
 Whitelist, alias, flavor
 See OpenStack wiki*
 Enable IOMMU
 Add grub settings to grub.conf
 GRUB_CMDLINE_LINUX_DEFAULT=“$GRUB_CMDLINE_LINUX_DEFAUL
T intel_iommu=on”
 dmseg | grep –e DMAR –e IOMMU
* https://wiki.openstack.org/wiki/Pci_passthrough
DOCOMO, INC All Rights Reserved
o Operate : Take care of flavor memory size
 As a result of our verification, IOMMU allocates all memory resources
when instances are launched.
 If you set flavor memory size large enough and launch maximum number
of instances, OOM-killer might kill the qemu process.
 Swapping doesn't work well because IOMMU allocate memory too fast.
Normal compute node
Mem space
IOMMU-enabled
compute node
Mem space
HostOS
HostOS
Instance
A
Instance B
Instance A Instance B
Allocated
arter
ballooning
DOCOMO, INC All Rights Reserved
o A workarounds to this problem are to take enough margin for
host OS.
 Reduce flavor memory size
 Sometimes too uncomfortable for GPU users
 Set reserved_host_memory_mb in nova.cfg to large size
 Also affect other flavors
 Decrease maximum number of instances on per host
→Any other solutions? Help us!
DOCOMO, INC All Rights Reserved
o How should we offer GPU flavor to in-house users?
o GPU with OpenStack, pros/cons
As a point of Pros. Cons.
Virtualization More stable than container Some GPU card trouble needs host
reboot
Immutableness Fast deploy, fast PDCA cycle Difficulty fair sharing GPU resources
Preparation
before
run machine
learning
Can provide device driver and
CUDA pre-installed image file
Need to follow new version, new
combination of driver, guest OS, CUDA
ver, library ver…
Cooperate with GPU instance users is important for private cloud providers
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
Reference Model
DOCOMO, INC All Rights Reserved
Aim to migrate some of in-house APP to our cloud
Problem
 Strict security policies:over 100 guidelines for system architecture
when users reconstruct APP on our cloud
 A lot of efforts are required to meet the policies.
 Nice if predefined models are provided:)
Monitoring
Security
vulnerabilityCertificate
Remote Access
Identification and
authentication IDS/IPS
Log
Encryption
Server Network Storage Operation
Firewall
Redundancy
Backup
Configuration
management
… … … …
DOCOMO, INC All Rights Reserved
“Reference Model” on our cloud
 System architecture based on many of security policies
 Sets of OSS stacks that have been heavily tested on our project
Our cloud
Web three-tier
model
Heat template
• Web basic model
• Web three-tier model
Input Template file into heat
Virtual router
Virtual network
Jump server
Proxy server
Web basic
model
Virtual router
Virtual network
LB/Web server
Jump server
Proxy Server LB/Web server
DB ServerWEB/LB
Jump
…...
AP server
Images
Mechanism
DOCOMO, INC All Rights Reserved
System architecture of Reference Model
WEB/LB WEB/LB
DB DB
Storage Storage
Internet
AP AP
Backup
Public-NW 192.168.10.0/24
Private-NW 192.168.20.0/24
Management-NW 192.168.30.0/24
SSL termination SSL termination
LB
VI
P
DB
VI
P
VPN
SSL -VPNProxy/NT
P
Monitoring
Storage
LB-HA-NW
DB-HA-NW
DB-repl-NW
End User
HTTPS
Operator
VPN
DOCOMO, INC All Rights Reserved
WEB(Apache)/LB(LVS w/ Ultramonkey) server
 HTTPS, dummy certificates installed by default
 WAF for IDS/ IPS
 The key point is to not only install, but also complete the
default setting about security.
 Why didn’t we use LBaaS_v1 ?
 LBaaS_v1(juno) doesn’t satisfy with use cases of our users.
 Required to
• set security group to LB(LBaaS_v2:not yet)
• terminate SSL at LB(LBaaS_v2:done)
• provide sorry page (LBaaS_v2:not yet)
DOCOMO, INC All Rights Reserved
 VPN(OpenVPN) server
 SSL-VPN for secure remote access
 Tools for operation of SSL-VPN, such as create and revoke certificates
 Why didn’t we use VPNaaS_v1 ?
 The algorithm for authentication in IKE phase1 accepted sha1, which
will be encryption losing safety assurance.
 VPNaaS in recent version “Newton” accepted sha256.
DOCOMO, INC All Rights Reserved
The covered areas by “Reference Model”
 60% of security policies
Future
 Update this model by adding the missing parts about security policies
 Aims to cover 100%
Monitoring
Security
vulnerabilityCertificate
Remote Access
Identification and
authentication IDS/IPS
Log
Encryption
Server Network Storage Operation
Firewall
Redundancy
Backup
Configuration
management
… … … …
Copyright©2016 NTT DOCOMO, INC. All rights reserved.
Security Update
DOCOMO, INC All Rights Reserved
 Current daily operation about vulnerabilities
 Most of the operation is manual.
 Check vulnerabilities
 Risk assessment of vulnerabilities
 Management TODO list
 Update our cloud
 Caused
 Human error
• Forget to check vulnerabilities
 Time:1hours/day
 More important operation of security as our cloud expands
 Nice if these operations can be automated:)
DOCOMO, INC All Rights Reserved
 Current Operation
 We proposed how to be automatic through processes.
 Testing to enable us to reduce human error
Check
vulnerabilities
Risk
assessment
Management
TODO list
Update
our cloud
Semi-
automatic
Operation
By the script checking
package-version related
with vulnerabilities
By checking
vendor site
By Excel
Semi-
automatic
By making ansible
playbook
DOCOMO, INC All Rights Reserved
 Check vulnerabilities
 CVE & CVSS
 CVE: attached ID to vulnerabilities
 CVSS: score to vulnerabilities
 API “CVE-search” is used for check
 Github: https://github.com/cve-search/cve-search
DOCOMO, INC All Rights Reserved
 Risk Assessment
 Key point
 CVSS risk assessment is not always match with our environment.
 Important
 The usage and version of package(→script)
 Whether the host can be internal NW or not.
 Vulnerability that guest OS can invade host OS
 Need to re-evaluate the CVSS score for each host regarding its
environment
DOCOMO, INC All Rights Reserved
 Management TODO list
 Do not forget vulnerabilities which have high risk until the patch of the
vulnerability is applied.
 Important
 Even if the CVSS score is low, it will sometimes become high score
in our environment.
 Need to check the same vulnerabilities continuously
DOCOMO, INC All Rights Reserved
 Update Our cloud
 Semi-automatic Procedure
 Manual interventions are required only for check points
 Consider the influence on users’ instance
 Future
 Apply our proposed way in the test environment at first
 Extend our tools for user’s self check
Live Migration
users’ instance
Security
Update
Return back
user’s instance
Check point①
The normality
of User’s APP
Check point②
The normality of
Openstack function
Check point③
The normality of
User’s APP
DOCOMO, INC All Rights Reserved
Thank you for listening!

More Related Content

NTTドコモ様 導入事例 OpenStack Summit 2016 Barcelona 講演「Expanding and Deepening NTT DOCOMO's Private Cloud」 - OpenStack最新情報セミナー(2016年12月)

  • 1. Copyright©2016 NTT DOCOMO, INC. All rights reserved. Expanding and Deepening NTT DOCOMO’s private cloud NTT DOCOMO Inc. Jun Ishii Kojiro Amano VirtualTech Japan Hiromichi Ito
  • 2. DOCOMO, INC All Rights Reserved Jun Ishii o Research Engineer, NTT DOCOMO o Developer, operator and technical consultant in NTT DOCOMO private cloud Hiromichi Ito o CTO, VirtualTech Japan o One of the first members of proposing OpenStack Bare Metal Provisioning (currently called "Ironic") Kojiro Amano o Research Engineer, NTT DOCOMO o Security consultant in NTT DOCOMO private cloud About us
  • 3. Copyright©2016 NTT DOCOMO, INC. All rights reserved. Expanding strategy of our private cloud Scale-up strategy
  • 4. DOCOMO, INC All Rights Reserved o One year after launched our private cloud, it goes larger and larger! Jun. 2015 Oct. 2016 Mar. 2017 Number of DCs 1 2 4 Number of HWs 50 300 900 Cores 1500 10000 Over 35000
  • 5. DOCOMO, INC All Rights Reserved o Why could we expand our cloud so fast ? o Main Strategy : Forest and Tree Make a forest Fill in trees
  • 6. DOCOMO, INC All Rights Reserved o Three important details  Decided to migrate a large scale in-house system  Obtain budget for making a forest  Fast deployment methods by normalization  Enable to create the forest quickly  Novel challenges for various users  Enrich the forest to plant various trees • L2GW • GPU instance • Reference model • Security update
  • 7. DOCOMO, INC All Rights Reserved o Three important details  Decided to migrate a large scale in-house system  Obtain budget for making a forest  Fast deployment methods by normalization  Enable to create the forest quickly  Novel challenges for various users  Enrich the forest to plant various trees • L2GW • GPU instance • Reference model • Security update
  • 8. DOCOMO, INC All Rights Reserved o Decided to migrate a large scale in-house system  Update whole system due to HW EOL o OpenStack-based cloud has many strengths.  Three years TCO is superior to an on-premises  Reduce 22% TCO, CAPEX & OPEX  Distributed architecture is compatible with cloud.  REST interfaces are suitable for maintaining systems.  Feasibility of migration/replication between long distance  L2GW, details are mentioned in later.
  • 9. DOCOMO, INC All Rights Reserved o Three important details  Decided to migrate a large scale in-house system  Obtain budget for making a forest  Fast deployment methods by normalization  Enable to create the forest quickly  Novel challenges for various users  Enrich the forest to plant various trees • L2GW • GPU instance • Reference model • Security update
  • 10. DOCOMO, INC All Rights Reserved o Fast deployment methods by normalization  DRY : Don't repeat yourself o How to  deploy  set up [compute node, swift storage node] more  deal with hardware trouble  ansible, KB (See our Tokyo summit presentation) o Only take one month to deploy new DC  From after racking and cabling ends till finish first QA test  Over 300 nodes, HW configuration settings & OpenStack install are just finished in 10 days by 5 operators.
  • 11. DOCOMO, INC All Rights Reserved o Three important details  Decided to migrate a large scale in-house system  Obtain budget for making a forest  Fast deployment methods by normalization  Enable to create the forest quickly  Novel challenges for various users  Enrich the forest to plant various trees • L2GW • GPU instance • Reference model • Security update
  • 12. DOCOMO, INC All Rights Reserved o Novel challenges… to satisfy various users' will, there are many difficulty and many know-how.  Add functions  L2GW  GPU instance  Reduce time to construct and manage users' systems  Reference model  Security update
  • 13. DOCOMO, INC All Rights Reserved o Three important details  Decided to migrate a large scale in-house system  Obtain budget for making a forest  Fast deployment methods by normalization  Enable to create the forest quickly  Novel challenges for various users  Enrich the forest to plant various trees • L2GW • GPU instance • Reference model • Security update These reasons enable our private cloud so FaT !!
  • 14. Copyright©2016 NTT DOCOMO, INC. All rights reserved. How deepening our private cloud Enrich for users
  • 15. Copyright©2016 NTT DOCOMO, INC. All rights reserved. L2 Gateway for connecting existing large-scale networks and inter-cloud networking. I'm good at connecting.
  • 16. DOCOMO, INC All Rights Reserved ○ Overview  Our user has large scale existing network and proprietary computer systems. – This network system has the great ability that provides Layer-2 connectivity to nationwide. – This proprietary computer system side does not have enough flexibility.  REST API  Service mobility  They decided to migrate to OpenStack on this renewal timing.  Network system side migration must be minimal.  Our user requested new two network services. – Connect the tenant network between the two datacenters – Connect instance and existing equipment with the layer-2
  • 17. DOCOMO, INC All Rights Reserved ○ Before DC 1 RTT 20-40ms WAN (Nationwide) DC 2 dedicated equipment dedicated equipment dedicated equipment dedicated equipment proprietary computer system proprietary computer system
  • 18. DOCOMO, INC All Rights Reserved ○ After DC 1 RTT 20-40ms WAN (Nationwide) DC 2 dedicated equipment dedicated equipment dedicated equipment dedicated equipment L2 GW RT L2 GW RT
  • 19. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF)  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect.
  • 20. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF).  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect. We choose "Region" zoning model. In "Region" model all service is separated correctly.
  • 21. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF).  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect. Our base OpenStack deployment model avoid SPOF already.
  • 22. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF).  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect. Existing system's IP addressing and routing architecture can deploy on the overlay network.
  • 23. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF).  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect. NAT is must technique for floating IP and connecting the external network. But, This system does not request the floating IP address function.
  • 24. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF).  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect. Our base OpenStack deploy model is using L3 ECMP fabric and VXLAN. So, We choose VXLAN Layer 2 Gateway(L2 Gateway).
  • 25. DOCOMO, INC All Rights Reserved ○ Our user's requirements  High Availability – Do not share control services between data center. – Avoid Single-Point-Of-Failure (SPOF).  Technical limitations – Do not change IP addressing and routing architecture. – Do not use Network Address Translation(NAT). – Must connect instance and existing equipment by Layer 2.  Performance – Total throughput requirement is several dozen Gbps. – Average packet size is smaller than general private clouds. – Several hundred VLAN must connect. Software based VXLAN L2 Gateway does not match short packet workload. So, We choose using hardware based VXLAN L2 Gateway.
  • 26. DOCOMO, INC All Rights Reserved ○ Equipment selection  Hardware VTEP – Modern L3 switch chipset has Hardware VTEP gateway functions.  Intel FM6000, Broadcom Trident II, Trident II+, Tomahawk – We tried to examine both Intel FM6000 and Broadcom Trident II based L3 network switch. – Finally, we compared three vendors' L3 switches. (A, D, J)  Comparison result – Vendor A's L3 switch can support VXLAN within a Multi-Chassis LAG (MLAG) deployment. Other vendors can not. (as of June 2016) – All vendors' L3 switches cleared performance criteria. – All vendors OVSDB protocol support has some issues. We choose vendor A's L3 switch. Because they support MLAG.
  • 27. DOCOMO, INC All Rights Reserved ○ Software test and proof-of-concept  Test target – Neutron networking-l2gw  API's and implementations to support L2 Gateways in Neutron. – Networking-l2gw provides "L2GW Service Plugin" and "L2GW Agent".  "L2GW Service Plugin" provides L2GW API services.  "L2GW agent" controls L3 switch by OVSDB protocol.  Test results – Several minor bugs (Already fixed by the community.) – Missing of features that is required for the production environment.  SSL support (Already implemented by the community.)  Handling Mcast_Macs_Remote table (We created modified patch for vendor A based on community patch, not merged yet.)
  • 28. DOCOMO, INC All Rights Reserved ○ Controller Node Neutron Server L2GW Service Plugin API ML2 L3 Nova Keystone Glance Cinder Horizon ML2 L2POP Compute Node ML2 OVS Agent Open vSwitch VTEP Virtual Switch OVSDB Server ML2 L2POP A’s Management Virtual appliance Network Node L2GW Agent ML2 OVS Agent Open vSwitch VTEP L3 Agent A’s Hardware VTEP WAN Hardware VTEP Virtual Router VLAN VLAN OVSDB Server OVSDB protocol OVSDB Server Control
  • 29. DOCOMO, INC All Rights Reserved ○ Result of pilot tests with as scale as production environment  Hardware L2GW side – OVSDB Server crash issues  When inserting a large number of record at one time, OVSDB server has crashed. (This issue already fixed by the vendor.)  Networking-L2GW side – We encountered several critical bugs.  But It is hard to reproduce.  When hit these bugs, L2GW agent stopped. – L2GW agent recovery from a crash state is terrible.  L2 gateway agent always syncs state between neutron database and OVSDB.  Unfortunately, when L2GW agent crashed or stalled, these two databases sometimes lost sync.  So, We must re-register L2GW connections manually when met these bugs.
  • 30. DOCOMO, INC All Rights Reserved o Network trouble occurred without missing a week.  The L2GW agent is unstable.  The l2gw agent does not work correctly after few days when users run a long test. • That test includes continuous instance creating and deleting. • That test includes continuous CRUD testing for neutron virtual network port.  The instance could not communicate another region instance and existing equipment.
  • 31. DOCOMO, INC All Rights Reserved o We decided to not use Networking-l2gw in the production environment for the time being.  We could not reproduce connection troubles between OVSDB and the L2GW agent that occurred a weekly.  We could not fix all critical bugs that we encountered.  In our environment, a port status issue occurred.  That issue will cause L2GW agent problem.  We would not like to use the l2population.  The l2population does not have enough scalability yet. Keep to a delivery date.  Delivery date delay. aka. Our project death.
  • 32. DOCOMO, INC All Rights Reserved o Our solution  We created a manual procedure to manage L2GW.  Manage an OVSDB by CLI • "vtep-ctl" command  Manage an OVS flow table by CLI • "ovs-ofctl" command  We created an automation system based on the manual procedure.  The upper management system calls the system. This system is working correctly now.
  • 33. DOCOMO, INC All Rights Reserved o Results  We provide stable L2GW which is connecting the two regions instances and existing equipment.  We passed all test criteria provided from the client.  Gets excellent service flexibility by the OpenStack.  The existing network configuration was kept that our user requested. o Next challenges of our L2GW project  Fix known issues of networking-l2gw.  We would like to provide OpenStack API for managing L2GW.  Fix scalability issue of l2population.  Investigate EVPN for expanding L2GW services.
  • 34. Copyright©2016 NTT DOCOMO, INC. All rights reserved. GPU instance Machine learning on OpenStack
  • 35. DOCOMO, INC All Rights Reserved Nvidia Tesla M40 for CUDA/cuDNN
  • 36. DOCOMO, INC All Rights Reserved o Deploy  To deploy GPU nodes, not only PCI pass-through functions but also IOMMU functions must be enabled.  Enable PCI pass-through  Whitelist, alias, flavor  See OpenStack wiki*  Enable IOMMU  Add grub settings to grub.conf  GRUB_CMDLINE_LINUX_DEFAULT=“$GRUB_CMDLINE_LINUX_DEFAUL T intel_iommu=on”  dmseg | grep –e DMAR –e IOMMU * https://wiki.openstack.org/wiki/Pci_passthrough
  • 37. DOCOMO, INC All Rights Reserved o Operate : Take care of flavor memory size  As a result of our verification, IOMMU allocates all memory resources when instances are launched.  If you set flavor memory size large enough and launch maximum number of instances, OOM-killer might kill the qemu process.  Swapping doesn't work well because IOMMU allocate memory too fast. Normal compute node Mem space IOMMU-enabled compute node Mem space HostOS HostOS Instance A Instance B Instance A Instance B Allocated arter ballooning
  • 38. DOCOMO, INC All Rights Reserved o A workarounds to this problem are to take enough margin for host OS.  Reduce flavor memory size  Sometimes too uncomfortable for GPU users  Set reserved_host_memory_mb in nova.cfg to large size  Also affect other flavors  Decrease maximum number of instances on per host →Any other solutions? Help us!
  • 39. DOCOMO, INC All Rights Reserved o How should we offer GPU flavor to in-house users? o GPU with OpenStack, pros/cons As a point of Pros. Cons. Virtualization More stable than container Some GPU card trouble needs host reboot Immutableness Fast deploy, fast PDCA cycle Difficulty fair sharing GPU resources Preparation before run machine learning Can provide device driver and CUDA pre-installed image file Need to follow new version, new combination of driver, guest OS, CUDA ver, library ver… Cooperate with GPU instance users is important for private cloud providers
  • 40. Copyright©2016 NTT DOCOMO, INC. All rights reserved. Reference Model
  • 41. DOCOMO, INC All Rights Reserved Aim to migrate some of in-house APP to our cloud Problem  Strict security policies:over 100 guidelines for system architecture when users reconstruct APP on our cloud  A lot of efforts are required to meet the policies.  Nice if predefined models are provided:) Monitoring Security vulnerabilityCertificate Remote Access Identification and authentication IDS/IPS Log Encryption Server Network Storage Operation Firewall Redundancy Backup Configuration management … … … …
  • 42. DOCOMO, INC All Rights Reserved “Reference Model” on our cloud  System architecture based on many of security policies  Sets of OSS stacks that have been heavily tested on our project Our cloud Web three-tier model Heat template • Web basic model • Web three-tier model Input Template file into heat Virtual router Virtual network Jump server Proxy server Web basic model Virtual router Virtual network LB/Web server Jump server Proxy Server LB/Web server DB ServerWEB/LB Jump …... AP server Images Mechanism
  • 43. DOCOMO, INC All Rights Reserved System architecture of Reference Model WEB/LB WEB/LB DB DB Storage Storage Internet AP AP Backup Public-NW 192.168.10.0/24 Private-NW 192.168.20.0/24 Management-NW 192.168.30.0/24 SSL termination SSL termination LB VI P DB VI P VPN SSL -VPNProxy/NT P Monitoring Storage LB-HA-NW DB-HA-NW DB-repl-NW End User HTTPS Operator VPN
  • 44. DOCOMO, INC All Rights Reserved WEB(Apache)/LB(LVS w/ Ultramonkey) server  HTTPS, dummy certificates installed by default  WAF for IDS/ IPS  The key point is to not only install, but also complete the default setting about security.  Why didn’t we use LBaaS_v1 ?  LBaaS_v1(juno) doesn’t satisfy with use cases of our users.  Required to • set security group to LB(LBaaS_v2:not yet) • terminate SSL at LB(LBaaS_v2:done) • provide sorry page (LBaaS_v2:not yet)
  • 45. DOCOMO, INC All Rights Reserved  VPN(OpenVPN) server  SSL-VPN for secure remote access  Tools for operation of SSL-VPN, such as create and revoke certificates  Why didn’t we use VPNaaS_v1 ?  The algorithm for authentication in IKE phase1 accepted sha1, which will be encryption losing safety assurance.  VPNaaS in recent version “Newton” accepted sha256.
  • 46. DOCOMO, INC All Rights Reserved The covered areas by “Reference Model”  60% of security policies Future  Update this model by adding the missing parts about security policies  Aims to cover 100% Monitoring Security vulnerabilityCertificate Remote Access Identification and authentication IDS/IPS Log Encryption Server Network Storage Operation Firewall Redundancy Backup Configuration management … … … …
  • 47. Copyright©2016 NTT DOCOMO, INC. All rights reserved. Security Update
  • 48. DOCOMO, INC All Rights Reserved  Current daily operation about vulnerabilities  Most of the operation is manual.  Check vulnerabilities  Risk assessment of vulnerabilities  Management TODO list  Update our cloud  Caused  Human error • Forget to check vulnerabilities  Time:1hours/day  More important operation of security as our cloud expands  Nice if these operations can be automated:)
  • 49. DOCOMO, INC All Rights Reserved  Current Operation  We proposed how to be automatic through processes.  Testing to enable us to reduce human error Check vulnerabilities Risk assessment Management TODO list Update our cloud Semi- automatic Operation By the script checking package-version related with vulnerabilities By checking vendor site By Excel Semi- automatic By making ansible playbook
  • 50. DOCOMO, INC All Rights Reserved  Check vulnerabilities  CVE & CVSS  CVE: attached ID to vulnerabilities  CVSS: score to vulnerabilities  API “CVE-search” is used for check  Github: https://github.com/cve-search/cve-search
  • 51. DOCOMO, INC All Rights Reserved  Risk Assessment  Key point  CVSS risk assessment is not always match with our environment.  Important  The usage and version of package(→script)  Whether the host can be internal NW or not.  Vulnerability that guest OS can invade host OS  Need to re-evaluate the CVSS score for each host regarding its environment
  • 52. DOCOMO, INC All Rights Reserved  Management TODO list  Do not forget vulnerabilities which have high risk until the patch of the vulnerability is applied.  Important  Even if the CVSS score is low, it will sometimes become high score in our environment.  Need to check the same vulnerabilities continuously
  • 53. DOCOMO, INC All Rights Reserved  Update Our cloud  Semi-automatic Procedure  Manual interventions are required only for check points  Consider the influence on users’ instance  Future  Apply our proposed way in the test environment at first  Extend our tools for user’s self check Live Migration users’ instance Security Update Return back user’s instance Check point① The normality of User’s APP Check point② The normality of Openstack function Check point③ The normality of User’s APP
  • 54. DOCOMO, INC All Rights Reserved Thank you for listening!

Editor's Notes

  1. By the way, Provider network can provide layer 2 connectivity to external devices too. But, our model can not deploy on large FLAT L2 network that required by provider network. So, We chooses L2 Gateway.
  2. Short packet workload with using commodity server based solution requires large number node for fulfilling the target performance.
  3. http://blog.ipspace.net/2015/09/vxlan-hardware-gateway-overview.html 性能値を載せるか検討
  4. https://github.com/openstack/networking-l2gw https://bugs.launchpad.net/networking-l2gw https://review.openstack.org/#/q/project:openstack/networking-l2gw SSL support https://review.openstack.org/#/c/334744/ Mcast_Macs_Remote table https://review.openstack.org/#/c/202495/
  5. 後で大幅に直す
  6. 大幅に直す
  7. Test on production size environment 稼動前テスト探す!pilot?
  8. 謝っている雲秀 Network trouble occurred without missing a week in customer's long run test phase. The l2gw agent does not work correctly after few days when customers long run a test. That test create and delete neutron port many times.
  9. Keep to a delivery date
  10. By using the L2GW, the existing network configuration was kept that our user requested.
  11. 16分: execute
  12. Our cloud expanded as we presented. Next, We would like to present how to plant various tree to free space, which means how to migrate various in-house APP to our cloud. When we aimed to migrate users, we had big problem. Our company have over one hundred guidelines for security governance, and a lot of efforts are required to meet the policies when users reconstruct APP on our cloud. For example, this picture is part of all security policies, and various point of view, Redundancy log, certificate and so on. In order to reduce users task, we wonder predefined models are provided
  13. We propose predefined models, called Referenence model. This is system architecture based on many on security rules, and then deployed server are included sets of Open Source software stacks that heavily tested on our project. Now, doing quick demo.(確認appleのdemo movieで言ってること)
  14. This picture display system architecture of Reference model. This is covered with several security policies, such as dividing network segment, enforcing SG. And, Each server meets needed security guidelines. For example, we will explain about WEB/LB server and step server.
  15. We explain about WEB/Load balancer server. Our reference model uses apache for web and LVS that is software loadbalancer by default. And, some of security policies are set by default, such as dummy certificates for HTTPS and WAF basic rules. By the way, we didn’t use LBaaS_v1. this is because LBaaS_v1, which we use juno, doesn’t satisfy use cases of our users. It means that Users required to set SG to LBaaS, or use sorry page and so on , and then it was difficult to satisfy these requirement by LBaaS_v1. That’s why we decided to use LVS
  16. We explain about VPN server. In order to secure remote access, (this server is prepaied,) and installed OpenVPN for SSL-VPN by default. And, default setting can be done, and we prepaied tools for operation of SSL-VPN. Same as LBaaS, we didn’t use VPNaaS_v1. this is because VPNaaS_v1, which we use juno, use only sha1, and then sha1 is under encryption compromise . That’s why we decided to use openvpn. we hear VPNaaS in neuton have sha256. Sha1
  17. Finally, about security policies, Reference model can covered 60% of security policies In this picture, covered area is changed pink color. For the future, we will update this model by adding the lack parts about security policies, aim to 100%.
  18. From now we will explain about security update. As our cloud expands, more security threats. Current manual operation is a lot of time, and can cause human error such as forget to check vulnerabilities Nice if we have automatic operation to vulnerabilities
  19. This shows current operations about vulnerability. Almost all of process is manual, it can make us a lot of time and human error. Especially, in this first step, we checked vendor site, and in second step, we assessed the risk of vulnerability one by one. Even if we have the scripts checking package-version, it won’t be enough. That’s why we hoped this operation can be automatic.
  20. now doing quick demo(検証中をデモで見せます). First, detection of vulnerabilities. Until now, we have found vulnerabilities by checking vendor site. (それに対して。の英語)this process can be based on CVE & CVSS. CVE means attached ID to vulnerabilities, and CVSS means score to vulnerability. As this score is high, the risk is high That’s why it is important to check CVE, and then we used API CVE-search for detection. This is demo. By this API once by day(1日に1回), the latest vulnerabilities can be found and shown in screen.
  21. Second, risk assessment. Even is CVSS is high, the risk of the vulnerability in our cloud me be low/none. For example, if the version is not related, risk will be none, Or, if the host in internal NW is targeted, The risk will be low. On the contrary, in case of vulnerability that guest OS can invade host OS, it is important to check heavyiliy. That’s why it is needed to re-evaluate the CVSS score for each host regarding its environment. Now Demo. We re-evaluate CVSS from the point of view about this important part, and shows S/A/B which are used in our security governance.
  22. Third, Management TODO list. Of cource, do not forget vulnerabillities until patch are applied. Moreover, it is important to check CVSS of same vulnerabilities. This is because it will be high score as analysis proceeds. Demo shows. In this screen, we manage TODO list as “status”. For example, updatable means this vulnerability will be applied. (逆もあるで) We updated CVE&CVSS by updating API, and then if CVSS become change, the vulnerabilities will be re-evaluated and shows screen. Now ultra short demo short quick
  23. Fourth, Update Our cloud. In this process, we must care about the influence on users’ instance. Then,, it is necessary to have manual interventions only for check points. It means that there are three processes in update process大まかに言って such as live migration, security update, return back. And, we had check points between three processed. For example, we try to confirm the normality users’ app after live migration, the nomarity of openstack function after update. Thus, we decided semi-automatic procedure currently. For the future, we try to automatic update hosts which don’t exist user Moreover, extend our security tools for users’ self-check, Thank you for your listening. Any question. 手のひらを向けて、please