Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

R meetup 20161011v2

•
•
Niels Ole Dam
Niels Ole Dam

Presentation from the CopenhagenR - useR Group Meetup at 
IT University of Copenhagen on Oct. 11 2016 on how to automatically deploy web applications built in R to a Cloud server (here DigitalOcean) using open source Docker with GitHub and basic Continuous Integration (here CircleCI) for automated testing and deployment. 
Presenter: Niels Ole Dam, Things in Flow Excerpt from the invitation to the meetup: Niels will talk about his favorite R-setup and will demonstrate how R, combined with some nice DockeR and Github tricks, can help even small teams and companies leverage the power of modern cloud computing. Niels uses R on a daily basis in his work as an independent consultant and he will share his thoughts on DockeR at the next meetup. Subjects covered: - How to setup and use RStudio, Docker, Docker Compose locally and with GitHub intgration. - How to setup and use Continuous Integration (CI) with automated testing and deployment to DigitalOcean using CircelCI and with reuse of the same docker-compose.yml file locally and remotely. - Tips and tricks on how to setup a good workflow. - Introduction to all the technologies and tools used. There are lots of clickable links in the pdf-version of the slides. Code for the setup demonstrated can be found at: https://github.com/thingsinflow/r-docker-workflow An accompanying clickable flowdiagram can be found at: http://bit.ly/R-Docker-workflow Enjoy! :-)

1 of 46
Download to read offline
Things in Flow | Helping teams and individuals to get things done in flow Things in Flow How to autodeploy your R solution to DigitalOcean using Docker, GitHub and CircleCI CopenhagenR - useR Group Meetup
 IT University of Copenhagen, Oct. 11 2016
 Presenter: Niels Ole Dam, Things in Flow Things in Flow | Helping teams and individuals to get things done in Things in Flow
Things in Flow About me • Physics and Communications at Roskilde University • Previously: • Project Manager at Experimentarium science center • Science Journalist at Illustrated Science magazine • Digital Business Developer at Bonnier Publications • Project Manager and Drupal Teamlead at Peytz & Co • Games Producer at Cape Cph • Since 2014: • Independent consultant at Things in Flow • I collect tools and help people get their stuff done digitally and with the right amount of efficient processes so that the work still feels right 2
Things in Flow Coursera courses to recommend • Data Science Specialisation by Johns Hopkins University • I did 8 of them a couple of years ago – they where great. • Process Mining by Eindhoven University of Technology • I did this one too – and I’m actually an official Disco partner. • By The Way: • I’m currently looking for a good Process Mining case. • Please contact me if you have that kind of log file data and business itch that needs scratching. 3 Tip: All underlined links are clickable in the pdf-version of this slideshow – click to get more info on the subjects.
Things in Flow Why I chose R • Feel the raw data under my fingertips (again) • Visualie raw data • Deliver prototypes to colleagues/clients • Deliver widely available solutions to clients • Bonus: Test new tools without polluting my computer 4
Things in Flow Why I chose Docker • Production ready for teams and small companies • Makes it possible to follow the Continuous Integration (CI) paradigme in my own production • Open Source – dedication and low cost • Docker with R is Awesome! :-) 5
Hang on! A lot of steps and things to grasp but easy to execute in practice ;-)
Things in Flow Ingredients to shop for • GitHub account: • Free if Open Source – otherwise starting at $7/mo • Docker Hub account (optional): • Free if Open Source and only 1 private account, otherwise starting at $7/mo • CircleCI: • Free if Linux and only 1 container – otherwise starting at $39-50/mo • R + RStudio locally: • Free • Cloud Server: • Variable, typically starting at $5/mo • Total: • Starting with Free (Open Source) or $7/m (1 private project) + Cloud server 7
GitHub + RStudio
Things in Flow GitHub: How it works • Clone code from GitHub • Commit your own changes • Git keeps track on additions and removals automatically • Push your code back to GitHub 9 Source: betterexplained.com
Things in Flow Using Git with RStudio • Create a repo at GitHub • Clone the repo to a local RStudio project • Commit to GitHub regularly using RStudio Use branching for trying out alternative solutions: • Open a terminal via the RStudio-link and run: git checkout -b [branchname] • …do your stuff with local commits… • Switch to Master, open a terminal via the RStudio-link and run: git merge --no-ff [branchname] • …the branch is now merged back into master with all it’s Git history… 10
Things in Flow Demo 1: GitHub + RStudio • Connect RStudio project with GitHub • Make a RStudio Shiny App project • Make some code changes • Push changes to GitHub repo • Optional: Make a local branch 11
Things in Flow New GitHub repo 12 Copy and use for RStudio
Things in Flow Create new RStudio project 13
Things in Flow RStudio is now connected with GitHub 14
Docker
Things in Flow Docker: How it works 16 Traditional Virtual Machines: Full OS for each VM Docker: OS code is partly shared
Things in Flow The ideal Docker workflow 17 Source: docker.com
Things in Flow Another take on the same 18 Source: terranillius.com
Things in Flow Using Docker • Install and start Docker for Mac (or Windows) • Verify that Docker is running • Search for Docker images on Docker Hub • … either via http://dockerhub.com • …or via Kitematic (installed as part of Docker for Mac) • Start your first Docker container i.e. hello-world-nginx • You are now running Docker locally 19
Things in Flow Demo 2: Deploy Shiny App on local VM • Start Docker for Mac (or Windows) • Verify that Docker is now running • Add Dockerfile to project • Add docker-compose.yml file to project • Open terminal via RStudio-link • Run: docker-compose up -d • Shiny App is now running on local Docker VM • …and can be accessed via http://localhost 20
Things in Flow Docker cheatsheet docker run <container> <command> docker run -v LOCAL_DIR:CONTAINER_DIR <container> <command> docker exec -it <container_id> <command> docker images docker logs <container_id> docker-compose up -d --build docker-compose start docker rmi [imageID] docker ps [-a] docker run ubuntu /bin/echo 'Hello world' 21
Things in Flow Manage data in containers • If you have some persistent data that you want to share between containers, or want to use from non-persistent containers, it’s best to create a named Data Volume Container, and then to mount the data from it.
 
 (from docker.com) 22
Things in Flow Using Compose is basically a three-step process 1. Define your app’s environment with a Dockerfile so it can be reproduced anywhere. 2. Define the services that make up your app in docker- compose.yml so they can be run together in an isolated environment. 3. Lastly, run docker-compose up and Compose will start and run your entire app.
 
 (from docker.com) 23
Things in Flow Compose can mange the whole lifecycle • Start, stop and rebuild services (start, stop, up, down) • View the status of running services • Stream the log output of running service (logs) • Run a one-off command on a service (run)
 
 (from docker.com) 24
Things in Flow Volume data preserved when containers are created • Compose preserves all volumes used by your services. When docker-compose up runs, if it finds any containers from previous runs, it copies the volumes from the old container to the new container. This process ensures that any data you’ve created in volumes isn’t lost.
 
 (from docker.com) 25
Things in Flow Moving Compose files between environments • Compose supports variables in the Compose file. You can use these variables to customize your composition for different environments, or different users. See Variable substitution for more details. • You can extend a Compose file using the extends field or by creating multiple Compose files. See extends for more details.
 
 (from docker.com) 26
From Local to Cloud
Things in Flow Cloud Deploy: How it works 28 Source: circleci.com
Things in Flow The Workflow 29 bit.ly/R-Docker-workflow R-Docker Workflow for teams and small companies LocalLocalVM(Docker)Cloud(CircleCI/DigitalOcean) v1, Niels Ole Dam, 5/10-2016 Start Client GitHub, Docker Hub and Cloud Server (i.e. DigitalOcean, Amazon, Azure etc.) http://thingsinflow.dk/ Show/Hide Messages Develop R solution + test locally Test passed? No Test/fix on local VM Test passed? No Setup initial deploy to Cloud Deploy to local VM Test/iterate with client Test passed? No Initial solution deployed Build on local VM docker-composeup Build on Cloud server Develop, test, verify and deploy docker-composeup 1. 2. 3. Setup auto deploy on CircleCI Solution updated 4. Switch prod server 5. 6. 7. GitHub Docker CircleCIRstudio Project Git branch bit.ly/R-Docker-workflow = clickabel version of this document Docker Cheat Sheet: - docker run <container> <command> - docker run -v LOCAL_DIR:CONTAINER_DIR<container> <command> - docker exec-it <container_id> <command> - docker images - docker logs<container_id> - docker-composeup -d - docker-composestart - docker rmi [imageID] - docker ps[-a] - docker run ubuntu /bin/echo 'Hello world' Request solution Request change Try out new frameworks on local VM 8. Test php/ Spark? Click on the blue numbers or doc icons for more info on that step/subject.
CircleCI (or Jenkins or TravisCI etc.)
Things in Flow CircleCI: How it works • Connects to GitHub • Listens for any changes on a specific GitHub repo • Looks for a circle.yml script in the root of the GitHub repo • Runs the circle.yml script every time the repo changes • Can be used for automatical running of: • …code tests • …docker image builds for Docker Hub • …deployment(s) to Cloud server(s) 31
Things in Flow Demo 3-4: Setup CircleCI for auto deploy • CircleCI connects to GitHub and Cloud via environment variables (=secure because credentials are not shown in code) • Environment variables could be: • $DOCKER_USER • $DOCKER_PASS • $DEPLOY_TOKEN • $DOCKER_EMAIL • Test that the circle.yml script is working as planned • Now everything is set up for auto deploy on push to GitHub! 32
Things in Flow Known issues with CircleCI • Docker 1.12 is currently NOT supported by CircleCI • Docker 1.10 is currently NOT installed out-of-the-box by CircleCI! (=compose must be installed explicitly) • But should not be a too big issue • …if it is, then use TravisCI (or own Jenkins etc.) instead 33
Things in Flow Demo 5-6: Auto deploy to DigitalOcean server • Make a code change to the RStudio project • Commit and push to Master (GitHub) • Wait 10 min – you’ll also get a mail • A new version of the cloud server has now been build on DigitalOcean with a random IP 34
Things in Flow Demo 7: Switch prod server and setup A record • New cloud server has now been build • Log into DigitalOcean controlpanel • Switch the floating IP address from old to new server • Requires: A subdomain (i.e. http://meetup.thingsinflow.dk) has been setup on your website (A Record) 35
Things in Flow Docker Swarm Mode: How it works • 30 sec demo: • www.johnzaccone.io/3-node-cluster-in-30-seconds/ • Using docker-compose with docker swarm • https://docs.docker.com/compose/swarm/ • Swarm mode overview: • https://docs.docker.com/engine/swarm/ 36
Bonus: Docker for local try outs
Things in Flow Try out new frameworks safely • Test out new frameworks in a local VM without: • …risking the health of your main computer • …messing with the initial configuration • …thinking too much about documenting your steps • Quickly test out new integrations 38
Things in Flow Demo 8: Try out some images • hello-world-nginx • Jenkins • php • Java • MySQL • Spark • Hadoop • … 39
Summing Things Up
Things in Flow Summing up: Steps needed • Connect RStudio to GitHub – see this link at Preferences -> Git/SVN • Make your Shiny App • Add docker-compose.yml and/or Dockerfile • Test Docker setup locally • Connect GitHub to Docker Hub • Connect Docker Hub to CircleCI (or other CI service) • Connect CircelCI to a cloud server (i.e. Amazon, DigitalOcean) • Push to GitHub and watch the magic unfold Tips: • Dockerfile and circle.yml: Use one-liners or shell script • Beware of absolute vs. relative paths in all scripts 41
Things in Flow Stuff not touched today • Partly update of individual containers on cloud server via GitHub and CircleCI (i.e. only on container at a time) • Sharing of volumes between Docker container and local machine • How to add a nice html frontend page (index.html) ;-) 42
Things in Flow Key Take Aways • Docker is production ready for teams and small companies • Docker makes it possible to use the Continuous Integration (CI) paradigme for your R projects • Docker with R is Awesome! :-) 43
Things in Flow Links • Getting started with Docker: • http://www.docker.com/products/overview • https://docs.docker.com/engine/getstarted/ • Manage Data in containers: • https://docs.docker.com/engine/tutorials/dockervolumes/#/creating- and-mounting-a-data-volume-container • Docker Compose: • https://docs.docker.com/compose/overview/ • Automated Docker Deployments with CircleCI: • http://www.mikeheijmans.com/docker/2015/09/07/automated-docker- deployment-with-circle/ • Continuous Integration and Delivery with Docker-CircleCI: • https://circleci.com/docs/docker/ 44
Things in Flow Questions?
Things in Flow Niels Ole Dam – nd@thingsinflow.dk thingsinflow.dk – Mob. +45 2712 9314 Thanks! :-) CopenhagenR - useR Group, Meetup at ITU, Oct. 11 2016 This presentation can be found at thingsinflow.dk/R-Docker-workflow

More Related Content

R meetup 20161011v2

  • 1. Things in Flow | Helping teams and individuals to get things done in flow Things in Flow How to autodeploy your R solution to DigitalOcean using Docker, GitHub and CircleCI CopenhagenR - useR Group Meetup
 IT University of Copenhagen, Oct. 11 2016
 Presenter: Niels Ole Dam, Things in Flow Things in Flow | Helping teams and individuals to get things done in Things in Flow
  • 2. Things in Flow About me • Physics and Communications at Roskilde University • Previously: • Project Manager at Experimentarium science center • Science Journalist at Illustrated Science magazine • Digital Business Developer at Bonnier Publications • Project Manager and Drupal Teamlead at Peytz & Co • Games Producer at Cape Cph • Since 2014: • Independent consultant at Things in Flow • I collect tools and help people get their stuff done digitally and with the right amount of efficient processes so that the work still feels right 2
  • 3. Things in Flow Coursera courses to recommend • Data Science Specialisation by Johns Hopkins University • I did 8 of them a couple of years ago – they where great. • Process Mining by Eindhoven University of Technology • I did this one too – and I’m actually an official Disco partner. • By The Way: • I’m currently looking for a good Process Mining case. • Please contact me if you have that kind of log file data and business itch that needs scratching. 3 Tip: All underlined links are clickable in the pdf-version of this slideshow – click to get more info on the subjects.
  • 4. Things in Flow Why I chose R • Feel the raw data under my fingertips (again) • Visualie raw data • Deliver prototypes to colleagues/clients • Deliver widely available solutions to clients • Bonus: Test new tools without polluting my computer 4
  • 5. Things in Flow Why I chose Docker • Production ready for teams and small companies • Makes it possible to follow the Continuous Integration (CI) paradigme in my own production • Open Source – dedication and low cost • Docker with R is Awesome! :-) 5
  • 6. Hang on! A lot of steps and things to grasp but easy to execute in practice ;-)
  • 7. Things in Flow Ingredients to shop for • GitHub account: • Free if Open Source – otherwise starting at $7/mo • Docker Hub account (optional): • Free if Open Source and only 1 private account, otherwise starting at $7/mo • CircleCI: • Free if Linux and only 1 container – otherwise starting at $39-50/mo • R + RStudio locally: • Free • Cloud Server: • Variable, typically starting at $5/mo • Total: • Starting with Free (Open Source) or $7/m (1 private project) + Cloud server 7
  • 9. Things in Flow GitHub: How it works • Clone code from GitHub • Commit your own changes • Git keeps track on additions and removals automatically • Push your code back to GitHub 9 Source: betterexplained.com
  • 10. Things in Flow Using Git with RStudio • Create a repo at GitHub • Clone the repo to a local RStudio project • Commit to GitHub regularly using RStudio Use branching for trying out alternative solutions: • Open a terminal via the RStudio-link and run: git checkout -b [branchname] • …do your stuff with local commits… • Switch to Master, open a terminal via the RStudio-link and run: git merge --no-ff [branchname] • …the branch is now merged back into master with all it’s Git history… 10
  • 11. Things in Flow Demo 1: GitHub + RStudio • Connect RStudio project with GitHub • Make a RStudio Shiny App project • Make some code changes • Push changes to GitHub repo • Optional: Make a local branch 11
  • 12. Things in Flow New GitHub repo 12 Copy and use for RStudio
  • 13. Things in Flow Create new RStudio project 13
  • 14. Things in Flow RStudio is now connected with GitHub 14
  • 16. Things in Flow Docker: How it works 16 Traditional Virtual Machines: Full OS for each VM Docker: OS code is partly shared
  • 17. Things in Flow The ideal Docker workflow 17 Source: docker.com
  • 18. Things in Flow Another take on the same 18 Source: terranillius.com
  • 19. Things in Flow Using Docker • Install and start Docker for Mac (or Windows) • Verify that Docker is running • Search for Docker images on Docker Hub • … either via http://dockerhub.com • …or via Kitematic (installed as part of Docker for Mac) • Start your first Docker container i.e. hello-world-nginx • You are now running Docker locally 19
  • 20. Things in Flow Demo 2: Deploy Shiny App on local VM • Start Docker for Mac (or Windows) • Verify that Docker is now running • Add Dockerfile to project • Add docker-compose.yml file to project • Open terminal via RStudio-link • Run: docker-compose up -d • Shiny App is now running on local Docker VM • …and can be accessed via http://localhost 20
  • 21. Things in Flow Docker cheatsheet docker run <container> <command> docker run -v LOCAL_DIR:CONTAINER_DIR <container> <command> docker exec -it <container_id> <command> docker images docker logs <container_id> docker-compose up -d --build docker-compose start docker rmi [imageID] docker ps [-a] docker run ubuntu /bin/echo 'Hello world' 21
  • 22. Things in Flow Manage data in containers • If you have some persistent data that you want to share between containers, or want to use from non-persistent containers, it’s best to create a named Data Volume Container, and then to mount the data from it.
 
 (from docker.com) 22
  • 23. Things in Flow Using Compose is basically a three-step process 1. Define your app’s environment with a Dockerfile so it can be reproduced anywhere. 2. Define the services that make up your app in docker- compose.yml so they can be run together in an isolated environment. 3. Lastly, run docker-compose up and Compose will start and run your entire app.
 
 (from docker.com) 23
  • 24. Things in Flow Compose can mange the whole lifecycle • Start, stop and rebuild services (start, stop, up, down) • View the status of running services • Stream the log output of running service (logs) • Run a one-off command on a service (run)
 
 (from docker.com) 24
  • 25. Things in Flow Volume data preserved when containers are created • Compose preserves all volumes used by your services. When docker-compose up runs, if it finds any containers from previous runs, it copies the volumes from the old container to the new container. This process ensures that any data you’ve created in volumes isn’t lost.
 
 (from docker.com) 25
  • 26. Things in Flow Moving Compose files between environments • Compose supports variables in the Compose file. You can use these variables to customize your composition for different environments, or different users. See Variable substitution for more details. • You can extend a Compose file using the extends field or by creating multiple Compose files. See extends for more details.
 
 (from docker.com) 26
  • 27. From Local to Cloud
  • 28. Things in Flow Cloud Deploy: How it works 28 Source: circleci.com
  • 29. Things in Flow The Workflow 29 bit.ly/R-Docker-workflow R-Docker Workflow for teams and small companies LocalLocalVM(Docker)Cloud(CircleCI/DigitalOcean) v1, Niels Ole Dam, 5/10-2016 Start Client GitHub, Docker Hub and Cloud Server (i.e. DigitalOcean, Amazon, Azure etc.) http://thingsinflow.dk/ Show/Hide Messages Develop R solution + test locally Test passed? No Test/fix on local VM Test passed? No Setup initial deploy to Cloud Deploy to local VM Test/iterate with client Test passed? No Initial solution deployed Build on local VM docker-composeup Build on Cloud server Develop, test, verify and deploy docker-composeup 1. 2. 3. Setup auto deploy on CircleCI Solution updated 4. Switch prod server 5. 6. 7. GitHub Docker CircleCIRstudio Project Git branch bit.ly/R-Docker-workflow = clickabel version of this document Docker Cheat Sheet: - docker run <container> <command> - docker run -v LOCAL_DIR:CONTAINER_DIR<container> <command> - docker exec-it <container_id> <command> - docker images - docker logs<container_id> - docker-composeup -d - docker-composestart - docker rmi [imageID] - docker ps[-a] - docker run ubuntu /bin/echo 'Hello world' Request solution Request change Try out new frameworks on local VM 8. Test php/ Spark? Click on the blue numbers or doc icons for more info on that step/subject.
  • 30. CircleCI (or Jenkins or TravisCI etc.)
  • 31. Things in Flow CircleCI: How it works • Connects to GitHub • Listens for any changes on a specific GitHub repo • Looks for a circle.yml script in the root of the GitHub repo • Runs the circle.yml script every time the repo changes • Can be used for automatical running of: • …code tests • …docker image builds for Docker Hub • …deployment(s) to Cloud server(s) 31
  • 32. Things in Flow Demo 3-4: Setup CircleCI for auto deploy • CircleCI connects to GitHub and Cloud via environment variables (=secure because credentials are not shown in code) • Environment variables could be: • $DOCKER_USER • $DOCKER_PASS • $DEPLOY_TOKEN • $DOCKER_EMAIL • Test that the circle.yml script is working as planned • Now everything is set up for auto deploy on push to GitHub! 32
  • 33. Things in Flow Known issues with CircleCI • Docker 1.12 is currently NOT supported by CircleCI • Docker 1.10 is currently NOT installed out-of-the-box by CircleCI! (=compose must be installed explicitly) • But should not be a too big issue • …if it is, then use TravisCI (or own Jenkins etc.) instead 33
  • 34. Things in Flow Demo 5-6: Auto deploy to DigitalOcean server • Make a code change to the RStudio project • Commit and push to Master (GitHub) • Wait 10 min – you’ll also get a mail • A new version of the cloud server has now been build on DigitalOcean with a random IP 34
  • 35. Things in Flow Demo 7: Switch prod server and setup A record • New cloud server has now been build • Log into DigitalOcean controlpanel • Switch the floating IP address from old to new server • Requires: A subdomain (i.e. http://meetup.thingsinflow.dk) has been setup on your website (A Record) 35
  • 36. Things in Flow Docker Swarm Mode: How it works • 30 sec demo: • www.johnzaccone.io/3-node-cluster-in-30-seconds/ • Using docker-compose with docker swarm • https://docs.docker.com/compose/swarm/ • Swarm mode overview: • https://docs.docker.com/engine/swarm/ 36
  • 38. Things in Flow Try out new frameworks safely • Test out new frameworks in a local VM without: • …risking the health of your main computer • …messing with the initial configuration • …thinking too much about documenting your steps • Quickly test out new integrations 38
  • 39. Things in Flow Demo 8: Try out some images • hello-world-nginx • Jenkins • php • Java • MySQL • Spark • Hadoop • … 39
  • 41. Things in Flow Summing up: Steps needed • Connect RStudio to GitHub – see this link at Preferences -> Git/SVN • Make your Shiny App • Add docker-compose.yml and/or Dockerfile • Test Docker setup locally • Connect GitHub to Docker Hub • Connect Docker Hub to CircleCI (or other CI service) • Connect CircelCI to a cloud server (i.e. Amazon, DigitalOcean) • Push to GitHub and watch the magic unfold Tips: • Dockerfile and circle.yml: Use one-liners or shell script • Beware of absolute vs. relative paths in all scripts 41
  • 42. Things in Flow Stuff not touched today • Partly update of individual containers on cloud server via GitHub and CircleCI (i.e. only on container at a time) • Sharing of volumes between Docker container and local machine • How to add a nice html frontend page (index.html) ;-) 42
  • 43. Things in Flow Key Take Aways • Docker is production ready for teams and small companies • Docker makes it possible to use the Continuous Integration (CI) paradigme for your R projects • Docker with R is Awesome! :-) 43
  • 44. Things in Flow Links • Getting started with Docker: • http://www.docker.com/products/overview • https://docs.docker.com/engine/getstarted/ • Manage Data in containers: • https://docs.docker.com/engine/tutorials/dockervolumes/#/creating- and-mounting-a-data-volume-container • Docker Compose: • https://docs.docker.com/compose/overview/ • Automated Docker Deployments with CircleCI: • http://www.mikeheijmans.com/docker/2015/09/07/automated-docker- deployment-with-circle/ • Continuous Integration and Delivery with Docker-CircleCI: • https://circleci.com/docs/docker/ 44
  • 46. Things in Flow Niels Ole Dam – nd@thingsinflow.dk thingsinflow.dk – Mob. +45 2712 9314 Thanks! :-) CopenhagenR - useR Group, Meetup at ITU, Oct. 11 2016 This presentation can be found at thingsinflow.dk/R-Docker-workflow