Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Gerrit Code Review migrations step-by-step

•Download as PPTX, PDF•
•
Luca Milanesio
Luca Milanesio

The document discusses different types of migrations ranging from trivial to ultrahazardous. It describes approaches for migrating Gerrit installations of varying complexity based on factors like team size, number of repositories, and distance of migration. Blue/green deployments are recommended for complex migrations to minimize risk and downtime. Automating migrations with Ansible and proper planning including staging and testing are emphasized to enable smooth upgrades.

1 of 39
0 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 0 Gerrit migrations Smoothly and Step by step Luca Milanesio Gerrit Code Review Maintainer GerritForge
1 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 1 About GerritForge Founded in the UK HQ in London Committed to OpenSource + Sunnyvale CA
2 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 2 Gerrit migrations smoothly step-by-step https://www.futurelearn.com/courses/migration-facts
3 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 3 Everything changes and nothing stands still Heraclitus of Ephesus Plato in Cratylus (360 BCE)
4 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 4 What if you do not want to change? https://stream.org/our-world-is-changing-too-fast/ The world around you is changing anyway
5 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 5 Example: Chrome 76 and Gerrit v2.13
6 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 6 Migration dimensions: distance Who has migrated to another home? to another state? to another country?
7 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 7 Migrating to another house Get a man with a van he moves everything you pack your bags One day … easy, done.
8 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 8 Migrating to another state Get interstate moving Plan well in advance Pack the essential, throw the rest Plan for storage Change utilities companies More days needed … more pain
9 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 9 Migrating to another country Global movers Get paperwork sorted: VISA etc. Pack your bags, sell the rest Study a foreign language Start a fresh new life Months needed … expensive, painful
10 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 10 Migration dimensions: people Who is migrating with you just your partner? with the kids? the whole family with relatives?
11 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 11 Small teams migrations Easy coordination, small costs • Plan a date • Get a car • Move https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/
12 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 12 Large teams migrations Coordinate people and high costs • Lots of advance planning • Get proper transport • Move in stages https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/
13 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 13 Migrations categories Size Distance trivial complex risky ultrahazardous
14 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 14 Common high-level change management process 1. Staging as a copy of production 2. Migrate in staging 3. Verify success, resolve problems 4. Migrate in production
15 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 15 Trivial migrations Small distance: +1 version (e.g. 2.14 to 2.15) Small team: < 100 users Small number repos: < 100 Small repos size: < 100MBytes Small number of refs: < 10k refs
16 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 16 Trivial migrations Stop + migrate + start, as per release notes 1. Read release-notes 2. Shutdown Gerrit 3. Backup repos, index, cache and db 4. Upgrade plugins 5. Gerrit init + reindex 6. Startup Gerrit
17 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 17 Complex migrations Small distance: +1 version (e.g. 2.14 to 2.15) Large team: 100s or 1000s of users Large number repos: 100s or 1000s of repos OR/AND Large repos size: GBs in size Large number of refs: 100ks of refs
18 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 18 Complex migrations: requirements 1. High SLA (99.9% or higher) = 8h outage /year 2. Minimize risk 3. Enforce consistency 4. Manage people expectations 5. Impact analysis on other tools (Jenkins, Jira, etc)
19 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 19 Complex migrations: planning • Resources allocation budget, people, boxes, storage, CPUs • High level schedule of the phases 1. Health-check of current setup 2. Development and preparation 3. Testing and bug-fix 4. Execution 5. Clean-up
20 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 20 Complex migrations: health- check What are the current system metrics? Do I have the right configuration? Are the repos and changes consistent? How is the system performing? What is the status of heap utilization? Are the repos GCed?
21 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 21 Complex migrations: health- check Consequences of migrating an unhealthy system? • Re-indexing errors • Unexpected configuration behavior • Unexpected performance issues • Inability to manage the migrated system • Very long migration times
22 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 22 Complex migrations: preparation Simulate in staging what will happen during cutover • Measure exact timings • Prepare scripts and configs for migration • Prepare acceptance tests • Test, fix, measure, iterate
23 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 23 Gerrit master (Blue) Gerrit master (Green) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Reduced risk and downtime • Setup HAProxy and HA Plugin • Upgrade one node at a time • Test upgraded node before opening traffic
24 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 24 Gerrit blue (Ver. N) Gerrit green (Ver. N) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Begin of the rollout • Both blue and green active • RO plugin inserted • Backup of the Repos/ReviewDb RO Plugin RO Plugin
25 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 25 Gerrit blue (Ver. N) Gerrit green (Ver. N) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Upgrade on green • Shutdown Gerrit green • Run init and upgrade plugins • Start Gerrit green • Online reindex + validation RO Plugin RO Plugin
26 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 26 Gerrit blue (Ver. N) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Gerrit green live on Ver. N+1 • Move traffic to Gerrit green • Observe metrics • Identify and address issues • Wait for online reindex to finish RO Plugin
27 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 27 Gerrit blue (Ver. N+1) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Gerrit blue to Ver. N+1 • Shutdown Gerrit blue • Upgrade war and plugins on blue • Copy indexes from green to blue • Startup Gerrit blue • Delta reindex RO Plugin
28 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 28 Gerrit blue (Ver. N+1) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Migration completed • Remote RO plugin on blue • Gerrit blue + green active • All indexes migrated
29 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 29 Risky migrations Large distance: 2+ version (e.g. 2.11 to 2.16) Small team: < 100 users Small number repos: < 100 Small repos size: < 100MBytes Small number of refs: < 10k refs
30 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 30 Risky migrations Test migrations for all releases from A to B 1. Read release-notes of *all* releases between A and B 2. Create a staging Gerrit 3. Apply the migration of all releases, one at a time 4. Identify problems, apply fixes, restart from 2.
31 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 31 Risky migrations Stop + migrate (all from A to B) + start 1. Shutdown Gerrit 2. Backup repos, index, cache and db 3. For every release from A to B: • Upgrade plugins • Gerrit init + reindex 4. Startup Gerrit
32 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 32 Ultrahazardous migrations https://company.sbb.ch/en/media/background-information/gotthard-base-tunnel.html
33 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 33 Automating Gerrit migrations with Ansible
34 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 34 Enabling a smooth migration  Migration is an issue that comes up all of the times and is clearly the major headache in managing Gerrit  Reasons for migrating: compatibility, added  Migration is scary because brings DISRUPTION and OUTAGE on Gerrit, vital in the dev life of organizations
35 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 35 Enabling a smooth migration  Gerrit has evolved to a point where the technology and tooling allow  limited risk  nearly zero down time migration.  With a proper planning in conjunction with the use of HA and well managed test / prod separation we can migrate without lowering the level of service we are currently providing to our providing to our organization
36 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 36 GerritForge smooth migration proposal 1. Pre-migration Health check • Identify installation possible weak spots • Identify some quick wins • Determine level of maturity needed to 2. Provide a migration plan • Timeline of the migration process • Discussed and agreed with the client 3. Assist the HA setup and migration in the test environment (*), T&C apply. 4. Three months included of Gerrit Enterprise Support • Allows full testing and possibly moving on All of the above for $15,000 Offer time-limited to Dec 31st 2019 (*) First 16h included, standard rate for extra hours. Travelling expenses excluded.
37 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 37 Poll: what are your plans for migrating? Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/
38 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 38 Wants to know more? GerritForge.com/contact

More Related Content

Gerrit Code Review migrations step-by-step

  • 1. 0 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 0 Gerrit migrations Smoothly and Step by step Luca Milanesio Gerrit Code Review Maintainer GerritForge
  • 2. 1 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 1 About GerritForge Founded in the UK HQ in London Committed to OpenSource + Sunnyvale CA
  • 3. 2 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 2 Gerrit migrations smoothly step-by-step https://www.futurelearn.com/courses/migration-facts
  • 4. 3 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 3 Everything changes and nothing stands still Heraclitus of Ephesus Plato in Cratylus (360 BCE)
  • 5. 4 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 4 What if you do not want to change? https://stream.org/our-world-is-changing-too-fast/ The world around you is changing anyway
  • 6. 5 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 5 Example: Chrome 76 and Gerrit v2.13
  • 7. 6 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 6 Migration dimensions: distance Who has migrated to another home? to another state? to another country?
  • 8. 7 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 7 Migrating to another house Get a man with a van he moves everything you pack your bags One day … easy, done.
  • 9. 8 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 8 Migrating to another state Get interstate moving Plan well in advance Pack the essential, throw the rest Plan for storage Change utilities companies More days needed … more pain
  • 10. 9 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 9 Migrating to another country Global movers Get paperwork sorted: VISA etc. Pack your bags, sell the rest Study a foreign language Start a fresh new life Months needed … expensive, painful
  • 11. 10 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 10 Migration dimensions: people Who is migrating with you just your partner? with the kids? the whole family with relatives?
  • 12. 11 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 11 Small teams migrations Easy coordination, small costs • Plan a date • Get a car • Move https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/
  • 13. 12 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 12 Large teams migrations Coordinate people and high costs • Lots of advance planning • Get proper transport • Move in stages https://driving-tests.org/beginner-drivers/how-to-pack-a-car-for-vacation/
  • 14. 13 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 13 Migrations categories Size Distance trivial complex risky ultrahazardous
  • 15. 14 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 14 Common high-level change management process 1. Staging as a copy of production 2. Migrate in staging 3. Verify success, resolve problems 4. Migrate in production
  • 16. 15 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 15 Trivial migrations Small distance: +1 version (e.g. 2.14 to 2.15) Small team: < 100 users Small number repos: < 100 Small repos size: < 100MBytes Small number of refs: < 10k refs
  • 17. 16 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 16 Trivial migrations Stop + migrate + start, as per release notes 1. Read release-notes 2. Shutdown Gerrit 3. Backup repos, index, cache and db 4. Upgrade plugins 5. Gerrit init + reindex 6. Startup Gerrit
  • 18. 17 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 17 Complex migrations Small distance: +1 version (e.g. 2.14 to 2.15) Large team: 100s or 1000s of users Large number repos: 100s or 1000s of repos OR/AND Large repos size: GBs in size Large number of refs: 100ks of refs
  • 19. 18 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 18 Complex migrations: requirements 1. High SLA (99.9% or higher) = 8h outage /year 2. Minimize risk 3. Enforce consistency 4. Manage people expectations 5. Impact analysis on other tools (Jenkins, Jira, etc)
  • 20. 19 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 19 Complex migrations: planning • Resources allocation budget, people, boxes, storage, CPUs • High level schedule of the phases 1. Health-check of current setup 2. Development and preparation 3. Testing and bug-fix 4. Execution 5. Clean-up
  • 21. 20 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 20 Complex migrations: health- check What are the current system metrics? Do I have the right configuration? Are the repos and changes consistent? How is the system performing? What is the status of heap utilization? Are the repos GCed?
  • 22. 21 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 21 Complex migrations: health- check Consequences of migrating an unhealthy system? • Re-indexing errors • Unexpected configuration behavior • Unexpected performance issues • Inability to manage the migrated system • Very long migration times
  • 23. 22 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 22 Complex migrations: preparation Simulate in staging what will happen during cutover • Measure exact timings • Prepare scripts and configs for migration • Prepare acceptance tests • Test, fix, measure, iterate
  • 24. 23 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 23 Gerrit master (Blue) Gerrit master (Green) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Reduced risk and downtime • Setup HAProxy and HA Plugin • Upgrade one node at a time • Test upgraded node before opening traffic
  • 25. 24 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 24 Gerrit blue (Ver. N) Gerrit green (Ver. N) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Begin of the rollout • Both blue and green active • RO plugin inserted • Backup of the Repos/ReviewDb RO Plugin RO Plugin
  • 26. 25 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 25 Gerrit blue (Ver. N) Gerrit green (Ver. N) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Upgrade on green • Shutdown Gerrit green • Run init and upgrade plugins • Start Gerrit green • Online reindex + validation RO Plugin RO Plugin
  • 27. 26 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 26 Gerrit blue (Ver. N) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Gerrit green live on Ver. N+1 • Move traffic to Gerrit green • Observe metrics • Identify and address issues • Wait for online reindex to finish RO Plugin
  • 28. 27 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 27 Gerrit blue (Ver. N+1) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Gerrit blue to Ver. N+1 • Shutdown Gerrit blue • Upgrade war and plugins on blue • Copy indexes from green to blue • Startup Gerrit blue • Delta reindex RO Plugin
  • 29. 28 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 28 Gerrit blue (Ver. N+1) Gerrit green (Ver. N+1) HAproxy HAproxy HA Plugin HA Plugin Shared NFS Complex migrations: blue/green rollout Migration completed • Remote RO plugin on blue • Gerrit blue + green active • All indexes migrated
  • 30. 29 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 29 Risky migrations Large distance: 2+ version (e.g. 2.11 to 2.16) Small team: < 100 users Small number repos: < 100 Small repos size: < 100MBytes Small number of refs: < 10k refs
  • 31. 30 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 30 Risky migrations Test migrations for all releases from A to B 1. Read release-notes of *all* releases between A and B 2. Create a staging Gerrit 3. Apply the migration of all releases, one at a time 4. Identify problems, apply fixes, restart from 2.
  • 32. 31 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 31 Risky migrations Stop + migrate (all from A to B) + start 1. Shutdown Gerrit 2. Backup repos, index, cache and db 3. For every release from A to B: • Upgrade plugins • Gerrit init + reindex 4. Startup Gerrit
  • 33. 32 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 32 Ultrahazardous migrations https://company.sbb.ch/en/media/background-information/gotthard-base-tunnel.html
  • 34. 33 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 33 Automating Gerrit migrations with Ansible
  • 35. 34 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 34 Enabling a smooth migration  Migration is an issue that comes up all of the times and is clearly the major headache in managing Gerrit  Reasons for migrating: compatibility, added  Migration is scary because brings DISRUPTION and OUTAGE on Gerrit, vital in the dev life of organizations
  • 36. 35 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 35 Enabling a smooth migration  Gerrit has evolved to a point where the technology and tooling allow  limited risk  nearly zero down time migration.  With a proper planning in conjunction with the use of HA and well managed test / prod separation we can migrate without lowering the level of service we are currently providing to our providing to our organization
  • 37. 36 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 36 GerritForge smooth migration proposal 1. Pre-migration Health check • Identify installation possible weak spots • Identify some quick wins • Determine level of maturity needed to 2. Provide a migration plan • Timeline of the migration process • Discussed and agreed with the client 3. Assist the HA setup and migration in the test environment (*), T&C apply. 4. Three months included of Gerrit Enterprise Support • Allows full testing and possibly moving on All of the above for $15,000 Offer time-limited to Dec 31st 2019 (*) First 16h included, standard rate for extra hours. Travelling expenses excluded.
  • 38. 37 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 37 Poll: what are your plans for migrating? Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/
  • 39. 38 Gerrit User Summit 2019 – GerritForge, Inc. – Sunnyvale CA GerritForge.com 38 Wants to know more? GerritForge.com/contact