Continuous Integration on Steroids
•
1 like•490 views
This document discusses continuous integration techniques at a company called HERE. It covers how HERE uses Jenkins for continuous integration, scaling Jenkins to handle a large number of builds per day, common issues encountered with Jenkins and plugins, and recommendations for optimizing continuous integration workflows. Specifically, it emphasizes the importance of considering scalability, reducing flakiness, minimizing memory usage, caching, and avoiding expensive failed builds. The presenter also encourages open source collaboration to improve build tools and workflows.
Report
Share
Continuous Integration on Steroids
- 1. Continuous Integration on Steroids Akbashev Alexander Highload++ | November 07, 2016
- 3. Agenda 01. CI in HERE 02. Monitoring 03. Scalability 04. Jenkins 05. Nightmares Plugins 06. Morale 07. Q&A
- 5. Every change goes through validation pipeline Gerrit Gerrit Plugin Pre-submit Trigger Pre-submit Trigger Build Build Build Build Build Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests
- 6. Feedback goes from tests back to Gerrit Gerrit Gerrit Plugin Pre-submit Trigger Pre-submit Trigger Build Build Build Build Build Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests
- 7. Feedback comes from every pipeline Gerrit Gerrit Plugin Pre-submit Trigger Pre-submit Trigger Build Build Build Build Build Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests Tests
- 8. Numbers 100k+ builds per day ~1.5k concurrent builds 1.3-2.5k executors • Each “build” is execution of one build/ test job • Total number correlates with number of commits • Number of builds is not so important as number of commits • Big throughput is extremely important • Morning commit • Before lunch • “Last attempt for today” • Raised on-demand • Health checks • Jenkins strategy is not optimized for cloud
- 10. Collects information about every build in system Groovy Event Listener Plugin Jenkins build Fluentd InfluxDB Grafana
- 11. Collects information about every build in system Groovy Event Listener Plugin Jenkins build Fluentd InfluxDB Grafana
- 12. JVM stats are the best “canary” Groovy Event Listener Plugin Jenkins build Fluentd InfluxDB Grafana Jenkins JVM
- 14. 03 Scalability
- 15. What do we want to achieve?
- 16. What do we want to achieve? Keep feedback time (< 20 min.)
- 17. What do we want to achieve? Keep feedback time (< 20 min.) Test as much as possible
- 18. What do we want to achieve? Keep feedback time (< 20 min.) Test as much as possible … with debug symbols
- 19. What do we want to achieve? Keep feedback time (< 20 min.) Test as much as possible … with debug symbols … and code coverage information
- 20. What do we want to achieve? Keep feedback time (< 20 min.) Test as much as possible … with debug symbols … and code coverage information and on physical devices
- 21. How to scale Increase number of executors Minimize job execution time Smart testing
- 22. How to increase number of executors? EC2 Plugin TestDroid
- 23. How to minimize job execution time
- 24. How to minimize job execution time Split tests by type
- 25. How to minimize job execution time Split tests by type Parallel execution
- 26. How to minimize job execution time Split tests by type Parallel execution Node as cache storage
- 27. How to minimize job execution time Split tests by type Parallel execution Node as cache storage Shared compiler cache
- 28. How to minimize job execution time Split tests by type Parallel execution Node as cache storage Shared compiler cache Profiling!
- 29. 04 Jenkins
- 30. Is Jenkins so slow or we are doing something wrong?
- 31. Is Jenkins so slow or we are doing something wrong? Jenkins is ok.
- 32. Is Jenkins so slow or we are doing something wrong? Jenkins is ok. But…
- 33. Surprise #1 Rotation costs a lot
- 34. Surprise #2 It works much better with nginx less jenkins.access.log | tail -n1000 | grep urt="-" | wc -l 407
- 35. Surprise #3 Some buttons are very dangerous
- 36. Surprise #3 Some buttons are very dangerous
- 38. What can you find in heap dump of OOM-Killed Jenkins?
- 39. What can you find in heap dump of OOM-Killed Jenkins? Console logs
- 40. Console logs Should be less than X MB Verbose output goes to file “>” and “tee” are amazing!
- 41. What can you find in heap dump of OOM-Killed Jenkins? Console logs
- 42. What can you find in heap dump of OOM-Killed Jenkins? Console logs Build history
- 43. Build history 2000 entities or 3 days Efficient rotator
- 44. What can you find in heap dump of OOM-Killed Jenkins? Console logs Build history
- 45. What can you find in heap dump of OOM-Killed Jenkins? Console logs Build history Build artifacts
- 46. Build artifacts Push to S3 directly from slaves Don’t store anything on master
- 48. Limit of number of builds 20K
- 49. Groovy Event Listener Plugin all events synchronized groovy compilation fixed since 1.010 (Mar 10, 2016)
- 50. Limit of number of builds 40K
- 51. Warnings Plugin Just another parser of console log parseConsole is “deprecated” parseFile is allowed 0 warnings are very appreciated :)
- 52. Limit of number of builds 60K
- 53. Timestamper Plugin Tail needs not only “tail” fixed since 1.8.5 (Aug 31, 2016)
- 54. Limit of number of builds 60K
- 55. EC2 Plugin Full list of all images in AWS fixed since 1.35 (Jun 30, 2016)
- 56. Limit of number of builds 90K
- 57. Robot Framework Plugin Green chart costs 100 times more Replaced by xUnit Plugin
- 58. Limit of number of builds 120K
- 59. Build Failure Analyzer Plugin One regexp One stream One thread PR-57 is not accepted yet
- 60. Limit of number of builds 140K
- 61. Cleanup Workspace Plugin `ü` breaks everything PR-29 is not accepted yet
- 62. 06 Morale
- 64. Final recommendations Think about scalability at first place
- 65. Final recommendations Think about scalability at first place Flakiness could be a huge problem
- 66. Final recommendations Think about scalability at first place Flakiness could be a huge problem Reduce memory allocations
- 67. Final recommendations Think about scalability at first place Flakiness could be a huge problem Reduce memory allocations Cache as much as possible
- 68. Final recommendations Think about scalability at first place Flakiness could be a huge problem Reduce memory allocations Cache as much as possible Failing builds can be expensive
- 69. Workflow Slowness? Profile! Fix! Contribute!
- 70. Open source collaboration Let’s make our life better ;)
- 71. Full list of our contributions related to this talk • Jenkins • ccache • clcache • EC2 Plugin • S3 Plugin • FluentD Plugin • BuildRotator Plugin • Groovy Event Listener Plugin • Timestamper Plugin • Robot Framework Plugin • Build Failure Analyzer Plugin • JVM GC Log Plugin for FluentD
- 72. Thank you Contact Akbashev Alexander GitHub: Jimilian E-mail: alexander.akbashev@here.com