How to lock a Python in a cage? Managing Python environment inside an R project

Copyright (c) WLOG Solutions
How to lock a Python in a cage?
Managing Python environment inside an R project
PyData meetup
Warsaw, 2018-02-06
Piotr Chaberski, WLOG Solutions

A brief intro...
Imagine that you are developing a project using R and
your big corporate customer, after weeks of processing
requests to establish open-source analytical environment,
finally managed to install R on their production machines.
Now you realized, that it would be nice to use some
Python library in your solution...
How would you tell the client to switch to Python for a
while?

Deployment
https://www.walldevil.com/wallpapers/a88/earth-moon-planet-astronaut.jpg
Scripting
Both R and Python are great for writing scripts, but...

Why R Suite?
• develop your solution maintaining
full control over dependencies
and their versions
• build the solution for a desired
production environment
• close your solution in ready-to-
deploy package that doesn’t
require any installations or
configuration… so doesn’t Python, to be clear

Why Python in R?
Why not only Python? Why not only R?
… but you still need some specific Python functionalities
Your client has R production
environment
You have well managed R
deployment procedure (using R
Suite)
You’d like to create a Shiny
app on top of your solution
You just know/like R better

So how to use Python when you have
R on production?
… just hide it deep down in your R Suite project

Deep learning using Keras/Tensorflow in R
Use case with focus on deployment

Prerequisites
• R
• R Suite CLI
• RStudio (optional but recommended)
• Miniconda
Example created on Windows, but after prerequisites are
installed for specific system, further steps are identical on Linux

Python code adapted from:
https://www.pyimagesearch.com/2016/09/26/a-simple-neural-network-with-python-and-keras
Dataset downloaded from:
https://www.kaggle.com/c/dogs-vs-cats/data

3 ways to use Keras in R

3 ways to use Keras in R
No matter how you use it, remember to use local Python!
use_python(python = dirname(...), required = TRUE)

1. keras R package
• most natural for R users
• wraps all Keras/Tensorflow
features with R syntax
• provides some R Studio
add-ins
• still uses reticulate and runs
Python under the hood
library(keras)
# keras model architecture
model <- keras_model_sequential() %>%
layer_dense(units = 768, input_shape = 3072, kernel_initializer
= "uniform", activation = "relu") %>%
layer_dense(units = 384, kernel_initializer = "uniform",
activation = "relu") %>%
layer_dense(units = 2, activation = "softmax")
# keras model compilation
loginfo("Compiling model...")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01),
loss = "binary_crossentropy",
metrics = "accuracy")
# keras model training and evaluation
loginfo("Model fitting and evaluation...")
model %>% fit(train_data, train_labels,
epochs = 20,
batch_size = 128,
validation_data = list(valid_data, valid_labels))
loginfo("Model training complete.")

2. reticulate interface
• general interface for
Python modules
• automatic conversions
between Python and R
data types (careful when
passing integers, tuples,
lists to Python functions)
• use “$” operator instead
of Python’s “.”
library(reticulate)
Sequential <- import("keras.models")$Sequential
Activation <- import("keras.layers")$Activation
SGD <- import("keras.optimizers")$SGD
Dense <- import("keras.layers")$Dense
model <- Sequential()
model$add(Dense(768L, input_dim = 3072L, init = "uniform",
activation = "relu"))
model$add(Dense(384L, init = "uniform", activation = "relu"))
model$add(Dense(2L))
model$add(Activation("softmax"))
loginfo("Compiling model...")
sgd <- SGD(lr = 0.01)
model$compile(loss = "binary_crossentropy",
optimizer = sgd,
metrics = list("accuracy"))
loginfo("Starting model training...")
model$fit(train_data, train_labels,
epochs = 20L,
batch_size = 128L,
validation_data = list(valid_data, valid_labels))
loginfo("Model training complete.")

3. running raw Python
• handy when you already
have some code in
Python
• hard to control Python
script execution from R
(separate loggers etc.)
• all objects from Python
session available in R
(converted or not)
library(reticulate)
from keras.models import Sequential
from keras.layers import Activation
from keras.optimizers import SGD
from keras.layers import Dense
model = Sequential()
model.add(Dense(768, input_dim=3072, init="uniform",
activation="relu"))
model.add(Dense(384, init="uniform", activation="relu"))
model.add(Dense(2))
model.add(Activation("softmax"))
log.info("Compiling model...")
sgd = SGD(lr=0.01)
model.compile(loss="binary_crossentropy",
optimizer=sgd,
metrics=["accuracy"])
log.info("Starting model training...")
model.fit(train_data, train_labels,
epochs=20,
batch_size=128,
validation_data=[valid_data, valid_labels])
log.info("Model training complete.")
py_script_results <- py_run_file(python_script_fpath, convert =
TRUE)
> py_script_results$model
<keras.models.Sequential>

Deployment package contains...
All R master scripts: All R binary packages:
Internal Python environment: Additional Python scripts:

Package deployment
Prod:
• binary consistent with Dev
• R
Dev:
• binary consistent with Prod
• R
• R Suite
• Miniconda

Soon in R Suite...
• add Python dependencies to R packages DESCRIPTION
• rsuite sysreqs collect; rsuite sysreqs install
• check if conda is installed, then create local Python env
• not only Python dependencies; also e.g. Linux system libraries

Piotr Chaberski
Thank You!
pchaberski@wlogsolutions.com
info@wlogsolutions.com
http://rsuite.io
https://github.com/WLOGSolutions/RSuite

How to lock a Python in a cage? Managing Python environment inside an R project

Related slideshows

More Related Content

How to lock a Python in a cage? Managing Python environment inside an R project