These are the slides from the Jump Start into Apache Spark and Databricks webinar on February 10th, 2016.
---
Spark is a fast, easy to use, and unified engine that allows you to solve many Data Sciences and Big Data (and many not-so-Big Data) scenarios easily. Spark comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing. We will leverage Databricks to quickly and easily demonstrate, visualize, and debug our code samples; the notebooks will be available for you to download.
1 of 39
Download to read offline
More Related Content
Jump Start into Apache® Spark™ and Databricks
1. Jump Start into
Apache® Spark™ and Databricks
Denny Lee,
Technology Evangelist
denny@databricks.com, @dennylee
2. Technology Evangelist,Databricks
(Working with Spark since v0.5)
Formerly:
• SeniorDirectorof Data Sciences EngineeringatConcur(nowpart of SAP)
• PrincipalProgramManager at Microsoft
Hands-on Data Engineer:
Architect for morethan 15 years, developinginternet-scaleinfrastructure
for both on-premisesandcloud includingBing’sAudienceInsights, Yahoo’s
24TB SSAS cube, and IsotopeIncubation Team (HDInsight).
About Me: Denny Lee
3. Founded by the creators of
Apache Spark in 2013
Share of Spark code
contributed by Databricks
in 2014
75%
3
Data Value
Created Databricks on top of Sparkto make big data simple.
We are Databricks, the company behind Spark.
4. …
Apache Spark Engine
Spark Core
Spark
Streaming
Spark SQL MLlib GraphX
Unified engineacross diverse workloads & environments
Scale out, fault tolerant
Python, Java, Scala, and R
APIs
Standard libraries
6. Large-Scale Usage
Largest cluster
8000 Nodes (Tencent)
Largest single job
1 PB (Alibaba, Databricks)
Top Streaming Intake
1 TB/hour (HHMI
Janelia Farm)
2014 On-Disk SortRecord
Fastest Open Source Engine
for sorting a PB
7. Notable Users
Source: Slide 5 of Spark Community Update
Companies That Presented at Spark Summit 2015 in San Francisco
9. Quick Start with Python
textFile = sc.textFile("/mnt/tardis6/docs/README.md")
textFile.count()
10. Quick Start with Scala
textFile = sc.textFile("/mnt/tardis6/docs/README.md")
textFile.count()
11. RDDs
• RDDs have actions, which return values,and transformations, which return
pointersto new RDDs.
• Transformations are lazy and executed when an action is run
• Transformations: map(), flatMap(), filter(), mapPartitions(), mapPartitionsWithIndex(),
sample(), union(), distinct(), groupByKey(), reduceByKey(), sortByKey(), join(), cogroup(),
pipe(), coalesce(), repartition(), partitionBy(), ...
• Actions: reduce(), collect(), count(), first(), take(), takeSample(), takeOrdered(),
saveAsTextFile(), saveAsSequenceFile(), saveAsObjectFile(), countByKey(), foreach(), ...
• Persist (cache) distributed data in memory or disk
13. History of Spark APIs
RDD
(2011)
DataFrame
(2013)
• Distributecollection
of JVM objects
• Functional Operators
(map, filter, etc.)
• Distributecollection of Rowobjects
• Expression-based operations and UDFs
• Logical plans and optimizer
• Fast/efficient internal representations
DataSet
(2015)
• Internally rows,externally
JVM objects
• “Best ofboth worlds”
type safe + fast
14. Benefit of Logical Plan:
Performance Parity Across Languages
0 2 4 6 8 10
Java/Scala
Python
Java/Scala
Python
R
SQL
Runtime for an example aggregation workload (secs)
DataFrame
RDD
15. 0
50
100
150
200
250
300
350
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9
Spark 1.3.1, 1.4, and 1.5 for 9 queries
1.5 Run A1 1.5 Run A2 1.5 Run B1 1.5 Run B2 1.4 Run A1 1.4 Run A2
NYC Taxi Dataset
16. Dataset API in Spark 1.6
Typed interface over DataFrames / Tungsten
case class Person(name: String, age: Long)
val dataframe = read.json(“people.json”)
val ds: Dataset[Person] = dataframe.as[Person]
ds.filter(p => p.name.startsWith(“M”))
.toDF()
.groupBy($“name”)
.avg(“age”)
21. External Web Service Call via Mapper
# Obtain the unique agents from the accesslog table
ipaddresses = sqlContext.sql("select distinct ip1 from
accesslog where ip1 is not null").rdd
# getCCA2: Obtains two letter country code based on IP address
def getCCA2(ip):
url = 'http://freegeoip.net/csv/' + ip
str = urllib2.urlopen(url).read()
return str.split(",")[1]
# Loop through distinct IP addresses and obtain two-letter country codes
mappedIPs = ipaddresses.map(lambda x: (x[0], getCCA2(x[0])))
22. Join DataFrames and Register Temp Table
# Join countrycodes with mappedIPsDF so we can have IP address and
# three-letter ISO country codes
mappedIP3 = mappedIP2
.join(countryCodesDF, mappedIP2.cca2 == countryCodesDF.cca2, "left_outer")
.select(mappedIP2.ip, mappedIP2.cca2, countryCodesDF.cca3, countryCodesDF.cn)
# Register the mapping table
mappedIP3.registerTempTable("mappedIP3")
23. Add Columns to DataFrames with UDFs
from user_agents import parse
from pyspark.sql.types import StringType
from pyspark.sql.functions import udf
# Create UDFs to extract out Browser Family information
def browserFamily(ua_string) : return xstr(parse(xstr(ua_string)).browser.family)
udfBrowserFamily = udf(browserFamily, StringType())
# Obtain the unique agents from the accesslog table
userAgentTbl = sqlContext.sql("select distinct agent from accesslog")
# Add new columns to the UserAgentInfo DataFrame containing browser information
userAgentInfo = userAgentTbl.withColumn('browserFamily',
udfBrowserFamily(userAgentTbl.agent))
24. Use Python UDFs with Spark SQL
# Define function (converts Apache web log time)
def weblog2Time(weblog_timestr): ...
# Define and Register UDF
udfWeblog2Time = udf(weblog2Time, DateType())
sqlContext.registerFunction("udfWeblog2Time", lambda x: weblog2Time(x))
# Create DataFrame
accessLogsPrime = sqlContext.sql("select hash(a.ip1, a.agent) as UserId,
m.cca3, udfWeblog2Time(a.datetime),...")udfWeblog2Time(a.datetime)
29. References
Spark DataFrames: Simple and FastAnalysison Structured Data [Michael Armbrust]
Apache Spark 1.6 presented by Databricks co-founderPatrick Wendell
Announcing Spark1.6
Introducing Spark Datasets
Spark SQL Data Sources API: Unified Data Access for the Spark Platform
30. Join us at
Spark Summit East
February16-18, 2016 | New York City
34. Spark adoption is
growing rapidly
Spark use is growing
beyond Hadoop
Spark is increasing
access to big data
Spark Survey Report 2015 Highlights
TOP 3 APACHE SPARK TAKEAWAYS
39. NOTABLE USERS THAT PRESENTED AT SPARK SUMMIT 2015 SAN FRANCISCO
Source: Slide 5 of Spark Community Update