Scala in Places API

Scala in Nokia Places API

Lukasz Balamut - Places API Team

17 April 2013

Purpose of this talk

Why are we spending time migrating to a new language?
How are we doing that?
What have we learned?

and of course: we are hiring.

Our scala story.

Our migration in lines of code

We’ve discovered more and more things on the way.

The Concordion tests problem
In June 2011 we had problems with Concordion (HTML based acceptance test
framework),

hard to integrate
HTML tests were not readable

Solution:

Multi-line Strings in Scala
Continue using JUnit

val expectedBody = """
{
"id":"250u09wh-83d4c5479b7c4db9836602936dbc8cb8",
"title" : "Marfil (Le)",
"alternativeNames" : [
{
"name" : "Le Marfil",
"language" : "fr"
}
]
}
"""

The test names problem
JUnit + camel case - not that readable for long names

@Test
def preservesSearchResponseItemOrder() { /* ... */ }

JUnit + underscores - violates naming convention, but better to read

@Test
def preserves_search_response_item_order() { /* ... */ }

ScalaTest

test("preserves search response’s item order") { /* ... */ }

It is supported by tools we use and produces nice output when run (as a bonus)

[info] DiscoverAroundAcceptanceTest:
[info] - calls nsp with 15Km radius
[info] - preserves search response’s item order
[info] - when requesting zero results, no request to search is made

Lift-Json

JSON (de)serialisation can be done in really nice way

e.g. getting place id:

{
"place": {
"a_id": "276u33db-6f084959f97c45bc9db26348bafd4563"
}
}

Also there is a generic way of parsing json, that gives you XPath like access:

val id = (json "place" "a_id").extract[String]

Lift-json provides also easy and eﬃcient case class JSON (de)serialisation,
hance case classes

Scala case classes in the model
We model our upstream and downstream datamodels as case classes.
case class Place (
name: String,

placeId: PlaceID
//(...)
)

Each case class is a proper immutable data type - the new Java Bean ;),
A lot of boilerplate code is generated by the compiler e.g.:
accessors,
equals/hashCode,
copy methods,
toString
apply/unapply (for pattern maching),

After decompiling the scala code above we will end up with 191 lines of Java:
package pbapi.model.api;

import scala.*;
import scala.collection.Iterator;
import scala.runtime.BoxesRunTime;
import scala.runtime.ScalaRunTime$;

Scala case classes - Configuration

Easy to update configuration
Easy to compare environments’ configuration

case class HttpClientConfig(
serviceName: String,
serviceBaseUrl: URL,
readTimeout: Int = 0,
connectionTimeout: Int,
proxy: Option[ProxyConfig] = None,
oauthKeys: Option[OAuthConfig] = None,
displayUrl: Option[String] = None
)

"httpConfig":{
"serviceName":"recommendations",
"serviceBaseUrl":"http://nose.svc.ovi.com/rest/v1/recommendations/near
"readTimeout":4000,
"connectionTimeout":500,
"displayUrl":"http://nose.svc.ovi.com/rest/v1/recommendations/nearby/"
}

The Scala Type System

A great way to express assumptions
Have them checked by the compiler through the whole codebase
At every point in the code know what to expect or you will be reminded by
compiler.

The Scala Type System - Options 1

Avoid null references (see The Billion Dollar Mistake )

This will not compile!

case class UpstreamResponse ( unreliableField: Option[String] )

case class MyResponse ( reliableField: String )

def transform(upstream: UpstreamResponse) =
MyResponse(
reliableField = upstream.unreliableField //<-- incompatible type
)


But this will:

case class UpstreamResponse ( unreliableField: Option[String] )

case class MyResponse ( reliableField: String )

def transform(upstream: UpstreamResponse) =
MyResponse(
reliableField = upstream.unreliableField.getOrElse("n/a") // enf
)


how we were learning e.g.: to transform string when it is deﬁned

def transform(str: String): String

reliableField =
if (upstream.unreliableField.isDefined)
transform(upstream.unreliableField.get)
else "n/a"

reliableField =
upstream.unreliableField match {
case Some(f) => transform(f)
case None => "n/a"
}

reliableField =
upstream.unreliableField.map(transform) | "n/a"

The Scala Type System - Standard Types

Immutable Map, List, Sets

val map: Map[String, String] = Map("a" -> "a", "b" -> "b")

val list: List[String] = "a" :: "b" :: Nil

Tuples to express Pairs, Triplets etc.

val pair: Pair[String, Int] = ("a", 1)

val (a, b) = pair // defines a:String and b:Int

The Scala Type System - error handling
Types can help with exceptional cases
val parsed: Validation[NumberFormatException, Int] = someString.parseInt

val optional: Option[Int] = parsed.toOption

val withDefault: Int = optional.getOrElse(0) //if there was parsing prob

Exception to Option
val sub: Option[String] =
allCatch.opt(line.substring(line.indexOf("{"")))

or Either
val parsed: Either[Throwable, Incident] =
allCatch.either(new Incident(parse(jsonString))
and handle it later
parsed match {
case Right(j) => Some(j)
case Left(t) => {
println("Parse error %s".format(t.getMessage))
None

The Scala Type System - functions

we defer translation (Translated) and text rendering (FormattedText) to
serialisation time using function composition so we don’t need to pass user
context through the whole stack e.g.:

case class Place (
//(...)
attribution: Option[Translated[FormattedText]],
//(...)
)

case class Translated[A](translator: TransEnv => A) {
def apply(env: TransEnv): A = translator(env)

def map[B](f: A => B): Translated[B] =
new Translated[B](env => f(apply(env)))
//(...)
}

XML literals

XML can be part of the code and compiler is checking syntax of it:

def toKmlCoordinates(style: String): String =
(<Placemark>
<name>{"bbox " + this}</name>
<styleUrl>{style}</styleUrl>
<Polygon>
<outerBoundaryIs>
<LinearRing>
<coordinates>
{west + "," + north + ",0"}
{east + "," + north + ",0"}
{east + "," + south + ",0"}
{west + "," + south + ",0"}
{west + "," + north + ",0"}
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>).toString

Scalacheck
Automatic testing of assumptions, using set of generated values.
e.g. the code below tests if
BBox.parse(a.toString) == a
is true for any a
val generator: Gen[BBox] =
for {
s <- choose(-90.0, 90.0)
n <- choose(-90.0, 90.0)
if (n > s)
w <- choose(-180.0, 180.0)
e <- choose(-180.0, 180.0)
} yield new BBox(w, s, e, n)

property("parse . toString is identity") {
check {
(a: BBox) =>
BBox.parse(a.toString) == a
}
}
may yield this failing test message after several evaluations.

Scala (Scalding) in our analytics jobs
e.g. popular places job
class PopularPlacesJob(args: Args) extends AccessLogJob(args) {

implicit val mapMonoid = new MapMonoid[String, Long]()
val ord = implicitly[Ordering[(Long, String)]].reverse

readParseFilter()
.flatMap(’entry -> ’ppid) {
entry:AccessLogEntry => entry.request.ppid
}
.mapTo((’entry, ’ppid) -> (’ppid, ’time, ’app)) {
arg: (AccessLogEntry, String) => (arg._2, arg._1.dateTime, a
}
.groupBy((’ppid, ’app)) {
_.min(’time -> ’minTime)
.max(’time -> ’maxTime)
.size(’num)
}
.groupBy((’ppid)) {
_.min(’minTime)
.max(’maxTime)
.sum(’num)

Not so good in Scala

Almost whole team needed to learn the new language - luckily we have
Toralf ;)
Tools are not as mature as those in Java (but they are getting better very
fast)
Longer compilation, compiler has a lot more to do (e.g. type inference)
Multiple inheritance (traits)
Operators
changes in every language release

What we have learned, discovered?

When done right - it’s possible to painlessly migrate code to new
language, developing new feautres in the same time
How to use good typesystem and enjoy it
Take advantage form whole great immutable world of painless
programming
Use functions as ﬁrst class citizens of language

Plans

Scala 2.10
Run Transformations in Futures, Validations etc.
Akka actors, Futures composition
Kill last Java ﬁles?
Stop using Spring

Thank you!

project site:
places.nlp.nokia.com
My contact:
lukasz.balamut@nokia.com
@lbalamut
open position in our team:

Scala in Places API

Related slideshows

More Related Content

Scala in Places API