Scala Reflection & Runtime MetaProgramming

Scala runtime Meta-Programming

About Me
Meir Maor
Chief Architect @ SparkBeyond
At SparkBeyond we leverage the collective
human knowledge to solve the world's toughest
problems

This Talk
Showcase Scala reflection, and compiler toolbox
Use a real guiding example to see how these are
used to solve a real problem

SparkBeyond
SparkBeyond generates code to solve machine
learning problems, leveraging world knowledge
Allows crafting deeply embedded pipelines to
explicitly transform your data

Guiding Example
● We would like to build a deeply embedded
framework to transform data.
● Multiple classes define actions
● Can be stacked together
● Embed user generated Scala code
● Auto generate a UI to input parameters for
actions.
● Allow annotating fields to guide UI generation

case class AddColumn(columnName: String,codeExpression: String) extends Action
{...}
case class SpecialParam(a: Int,b : Int => Int)
object SpecialAction {
val myMap = Map("mul2" -> SpecialParam(1,_ * 2),
"add1" -> SpecialParam(2,_ + 1),
"ident" -> SpecialParam(3, identity)
)
}
case class SpecialAction(@PossibleValuesMap(SpecialAction.myMap) foo:
SpecialParam) extends Action

Intro to Scala Meta Programming
Creating and using mirror
import reflect.runtime.universe._
val cm =
runtimeMirror(getClass.getClassLoader)

Know the players
● Type
● Symbol
● Tree
documentation reference: http://docs.scala-lang.org/overviews/reflection/symbols-trees-types.html

Type
val t=typeOf[List[Int]]
t: reflect.runtime.universe.Type =
scala.List[Int]
Compare with =:= or <:< or weak_<:<
Key methods: declarations, baseClasses,
BaseType, typeSymbol

Symbol
● If it has a name it has a Symbol
– Types, Members, parameters
Not a type, even TypeSymbol is less than a Type
val ts=t.typeSymbol
ts: reflect.runtime.universe.Symbol =
class List

Tree
● An Abstract Syntax Tree
val tree = Apply(Select(Ident(TermName("x")),
TermName("$plus")), List(Literal(Constant(2))))
tree: scala.reflect.runtime.universe.Apply = x.$plus(2)
val expr = reify { class Flower { def name = "Rose" } }
expr: scala.reflect.runtime.universe.Expr[Unit] = …
expr.tree

Get param list from primary ctr
scala> val typ = typeOf[AddColumn]
typ: reflect.runtime.universe.Type = AddColumn
val primary=typ.typeSymbol.asClass.primaryConstructor
primary: reflect.runtime.universe.Symbol = constructor AddColumn
//recall a method can have multiple param lists, hence flaten
val paramList = primary.asMethod.paramLists.flatten
paramList: List[reflect.runtime.universe.Symbol] = List(value
columnName, value codeExpression)

Code Generation
● Cglib – byte code generation library
● Compiling externally and loading with a fresh
classloader
● The unsafe way - mutate the existing
classloader
● Scala Compiler toolbox

UrlClassLoader
def getClassInstanceFromJar[T](jarFile: File, className:
String): T = {
val classLoader = new
scala.reflect.internal.util.ScalaClassLoader.URLClassLoader(Seq(ja
rFile.toURI.toURL), this.getClass.getClassLoader)
val mirror =
scala.reflect.runtime.universe.runtimeMirror(classLoader)
val classSymbol = mirror.staticClass(className)
val ctorSymbol = classSymbol.primaryConstructor.asMethod
val ctor =
mirror.reflectClass(classSymbol).reflectConstructor(ctorSymbol)
ctor.apply().asInstanceOf[T]
}

Mutate existing ClassLoader
public static void addURL(URL u) throws IOException {
URLClassLoader sysloader = (URLClassLoader)
ClassLoader.getSystemClassLoader();
Class sysclass = URLClassLoader.class;
try {
Method method = sysclass.getDeclaredMethod("addURL", parameters);
method.setAccessible(true);
method.invoke(sysloader, new Object[]{u});
} catch (Throwable t) {
t.printStackTrace();
throw new IOException("Error, could not add URL to system
classloader");
}//end try catch
}//end method

Compiler ToolBox
● Experimental (even more so than the rest of the reflection api)
import scala.tools.reflect.ToolBox
val tb = cm.mkToolBox()
● Parse, typecheck, eval
def run(code: String) = {
synchronized {
val tree = tb parse s"import scala._nimport Predef._n $code"
tb eval tree
}
}

Using the ToolBox
● Get User code fragment
● Add boilerplate function definition, useful
variables
● Parse and eval
● Apply function on new data

Annotations
● In Java you can easily read annotations reflectively at
runtime
● Annotations can only hold specific types:
– primitive
– String
– Class
– an Enum
– another Annotation
– an array of any of the above

Scala annotations
● Scala annotations are not Java annotations
● You can put anything in a Scala annotation
● But How do you read it?
Documentation: http://docs.scala-lang.org/overviews/reflection/annotations-names-scopes.html

Defining Scala Annotation
Extend StaticAnnotation to make it available at
Runtime
case class ValuesMap(vals: Map[String, Any]) extends
StaticAnnotation

Get annotations from a symbol
● Many things can be annotated
– A type, A Method, A parameter, a val/var
● Recall our example:
case class SpecialAction(@ValuesMap(SpecialAction.myMap) foo:
SpecialParam) extends Action

Get annotations from a symbol
val param: Symbol = longestParmList.head
param: reflect.runtime.universe.Symbol = value foo
val annotations = param.annotations
scala> annotations: List[reflect.runtime.universe.Annotation] =
List(com.sparkbeyond.runtime.util.misc.ValuesMap(SpecialAction.m
yMap)

Recreate Annotation Instance
scala> val tree=annotations.head.tree
tree: reflect.runtime.universe.Tree = new
com.sparkbeyond.runtime.util.misc.ValuesMap(SpecialAction.myMap)
scala> tb.eval(tb.untypecheck(tree))
res15: Any = ValuesMap(Map(mul2 -> SpecialParam(1,<function1>), add1 ->
SpecialParam(2,<function1>), ident -> SpecialParam(3,<function1>)))

Overcoming Erasure - RTTI
● Java Generics for better or worse get erased in
runtime
● We would like to be able to tell the Type of an
instance including Type Parameters
● In most cases this is possible in a meaningful
way.

What is my type?
case class OneIntSeq(a: Int) extends Seq[Int] {...}
case class MyCaseClass[T,U](e: T,o: Seq[U])
val v = MyCaseClass("foo",OneIntSeq(1))
We want a method, given v to produce:
MyCaseClass[String,Int]

val topType=cm.reflect(v).symbol.toTypeConstructor
topType: reflect.runtime.universe.Type = MyCaseClass
//As Before
val primaryParams = topType.typeSymbol.asClass.primaryConstructor...
primaryParams: List[reflect.runtime.universe.Symbol] = List(value elem,
value other)
primaryParams.map(_.typeSignature)
res29: List[reflect.runtime.universe.Type] = List(T, scala.Seq[U])

Inspecting the Type
topType.decls.foreach{case s => println(s.toString + "t" + s.typeSignature) }
value elem => T
value elem T
value other => scala.Seq[U]
value other scala.Seq[U]
constructor MyCaseClass (elem: T, other: scala.Seq[U])MyCaseClass[T,U]
method copy [T, U](elem: T, other: scala.Seq[U])MyCaseClass[T,U]
method copy$default$1 [T, U]=> T @scala.annotation.unchecked.uncheckedVariance
method copy$default$2 [T, U]=> scala.Seq[U] @scala.annotation.unchecked.uncheckedVariance
method productPrefix => java.lang.String
method productArity => scala.Int
method productElement (x$1: scala.Int)scala.Any
method productIterator => Iterator[scala.Any]
method canEqual (x$1: scala.Any)scala.Boolean
method hashCode ()scala.Int
method toString ()java.lang.String
method equals (x$1: scala.Any)scala.Boolean

val instanceMirror = cm.reflect(v)
instanceMirror: reflect.runtime.universe.InstanceMirror = instance
mirror for MyCaseClass(foo,(1))
val elemField=topType.decls.filterNot(x => x.isMethod).head
elemField: reflect.runtime.universe.Symbol = value elem
val elemValue=instanceMirror.reflectField(elemField.asTerm).get
elemValue: Any = foo
val elemTyp=cm.reflect(elemValue).symbol.toTypeConstructor
elemTyp: reflect.runtime.universe.Type = java.lang.String

Similarly for second param other: Seq[U]
val otherTyp= cm.reflect(otherValue).symbol.toTypeConstructor
otherTyp: reflect.runtime.universe.Type = OneIntSeq
val asBase=otherTyp.baseType(otherField.typeSignature.typeSymbol)
asBase: reflect.runtime.universe.Type = Seq[scala.Int]
val uType = asBase.typeArgs.head
uType: reflect.runtime.universe.Type = scala.Int

Combining Things together
val finalType=appliedType(topType,List(elemTyp,uType))
finalType: reflect.runtime.universe.Type =
MyCaseClass[java.lang.String,scala.Int]
Q.E.D

Final Thoughts
Questions?
Like pushing Scala to it’s limits? Thrive on
scalable computing with machine learning on
top?
Join us!

Scala Reflection & Runtime MetaProgramming

More Related Content

Scala Reflection & Runtime MetaProgramming