TIAD - DYI: A simple orchestrator built step by step
•
4 likes•2,811 views
This document provides step-by-step instructions for building a simple orchestrator. It begins by setting up the basic components including RabbitMQ as a messaging broker and Celery workers. Step 1 demonstrates executing a simple AWS resource by adding a task to the queue. Step 2 adds a MongoDB database to store resources outside of the queue. Step 3 builds a service level on top by allowing resources like AWS instances and Docker containers to be orchestrated together through a YAML file.
TIAD - DYI: A simple orchestrator built step by step
- 1. DIY: A simple orchestrator built step by step TIAD PARIS 2015
- 2. ChallengesHi, I’m Ahmet Demir Also known as @ahmet2mir Opensource Enthusiast @D2SI
- 4. All IT want
- 6. When we try to assemble them http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823
- 8. What we orchestrate ? IT Process: disaster recovery, restart process Infrastructure: Build a VM, upgrade it Application Deployment: Deploy an application on x nodes Developpements: Continuous integration, Continuous Delivery
- 9. State of the art
- 10. Eliminate "server management deployment" FusionDirectory - http://www.fusiondirectory.org m23 - http://m23.sourceforge.net/PostNuke-0.750/html/index.php Fabric - http://www.fabfile.org/ Archipel - http://archipelproject.org/ AppScale - https://github.com/AppScale/appscale OpenBuild - http://openbuildservice.org/ CompatibleOne - http://compatibleone.org/ Opsi - http://www.opsi.org
- 11. Eliminate config manager Puppet - https://puppetlabs.com/ Chef - https://chef.io SaltStack - http://saltstack.com/ Ansible - http://www.ansible.com/home CFengine - http://cfengine.com/
- 12. Eliminate "server task oriented" Rundeck - http://rundeck.org/
- 13. Eliminate Only Cloud oriented Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki Openstack Heat - https://github.com/openstack/heat
- 14. Finally 1/3 Terraform - https://terraform.io/ Terraform provides a flexible abstraction of resources and providers. This model allows for representing everything from physical hardware, virtual machines, and containers, to email and DNS providers #Go
- 15. Finally 2/3 Salt Cloud - https://salt-cloud.readthedocs.org Salt cloud is a public cloud provisioning tool. Salt cloud is made to integrate Salt into cloud providers in a clean way so that minions on public cloud systems can be quickly and easily modeled and provisioned. #Python
- 16. Finally 3/3 OpenScore - http://openscore.github.io/ Flow-based orchestration engine to manage deployed applications Allows you to rapidly automate your DevOps use cases using ready-made workflows. Based on Java and HP OO engine. #Java
- 17. OpenStack Heat based: Resource based, CRUD, Graph dependency, describe in Yaml Terraform: Execution Plan, Diff on infra and all heat features, describe in “json like” format Salt Cloud: Easy modules integration but no plan, no graph OpenScore: Service in Yaml, no plan but runbook featured Features
- 18. API access ? How to scale ? If I had more than 100 requests in same time ? Where can I store all my services ? How can I track ? Is modules integration easy ? What about lifecycle ? Versioning the service ? Multi-user ? Organization > 100 peoples ? High Availability ? But Try to solve or ...
- 19. Do It Yourself
- 21. Simple to maintain Think tomorrow Infrastructure agnostic Easy development Don't forget Windows users :) Fundamentals
- 22. NetworkStorage Compute Orchestrator Referential Config Manager Catalog HP MP, OpenCart... Monitoring Nagios... CI / CD Jenkins, Bamboo ... Service Designer Big Picture
- 23. Engine API Resource runner Resource 1 Resource 2 Resource 3 Resource 4 Run 1 & 2 in parallel Then run 3 Then run 4 Service Designer & Orchestrator
- 24. Components
- 25. If you think to something, you will find your answer in Python Both of Devs and Ops can use Python and use it quickly No compilation and not "magic" Write less and do more Limits the number of times your brain has to do a "context switch." Helps productivity The language
- 26. YAML is simple to read and write Many projects use it: Heat, Fig etc... Service Description: YAML
- 27. Many tasks management Easy to scale Dedicate queue to some tasks (DMZ, Windows...) Asynchronous task queue/job Real-time operation, but supports scheduling Tasks are executed concurrently Used in production systems to process millions of tasks a day Tasks elery
- 28. Messaging broker Common platform to send and receive messages Safe place to live until received Can hits one million messages per second Message Delivery
- 29. rapid and agile web development No migrations Not a relational database Schemaless Database
- 32. Step 0: Build workers and broker Rabbit MQ Celery 1 Celery 2 Flower GUI
- 33. orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo orion ~ $ cd demo orion demo $ cd 00_celery_rabbit # start celery orion 00_celery_rabbit $ fig -f celery.yml -p demo up -d # start rabbitmq orion 00_celery_rabbit $ fig -f rabbitmq.yml -p demo up -d # create rabbitmq user orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_user myuser mypassword orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_vhost myvhost orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*" # gui for flower and rabbitmq orion 00_celery_rabbit $ fig -f flower.yml -p demo up -d # Visit http://localhost:5555 Step 0: Demo
- 34. Step 1: Execute a simple resource Rabbit MQ Celery 1 Celery 2 AWS Instance Resource add in queue Flower GUI
- 35. Celery 1 Wrapper AWS Wrapper Docker Docker CRUD AWS CRUD Pure Python Package deployed with pip Wrapper to simplify execution
- 36. orion demo $ cd 01_aws_resource # virtual python environment orion 01_aws_resource $ virtualenv . orion 01_aws_resource $ source bin/activate orion 01_aws_resource $ pip install -r requirements.txt # add credentials in client_provider.py # edit client_provider with some values (optionnal) … # run the resource orion 01_aws_resource $ python client.py Step 1: Demo
- 37. Step 2: Store outside Rabbit MQ Celery 1 Celery 2 Flower GUI MongoDBApplication
- 38. orion demo $ cd 02_db_provider # virtual python environment orion 02_db_provider $ virtualenv . orion 02_db_provider $ source bin/activate orion 02_db_provider $ pip install -r requirements.txt # start mongodb orion 02_db_provider $ fig -f mongodb.yml -p demo up -d # add credentials in add_*.py orion 02_db_provider $ python add_aws.py orion 02_db_provider $ python add_docker.py # visit http://localhost:8080 Step 2: Demo
- 39. Application Step 3: Service level Rabbit MQ Celery 1 Celery 2 Service AWS Instance Resource Flower GUI Docker Resource 1 2 MongoDB
- 40. orion demo $ cd 03_app # virtual python environment orion 03_app $ virtualenv . orion 03_app $ source bin/activate orion 03_app $ pip install -r requirements.txt # run the service orion 03_app $ python teecli.py create sample01.yml Step 3: Demo
- 41. Commit API Commit Admin GUI Commit Referential Management Commit all modules and wrappers Build a service catalog Role based access to the app http://teerex.io (soon) https://github.com/ahmet2mir/teerex (soon) Next steps