Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Jump Start into Apache® Spark™ and Databricks

•
•

These are the slides from the Jump Start into Apache Spark and Databricks webinar on February 10th, 2016. --- Spark is a fast, easy to use, and unified engine that allows you to solve many Data Sciences and Big Data (and many not-so-Big Data) scenarios easily. Spark comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing. We will leverage Databricks to quickly and easily demonstrate, visualize, and debug our code samples; the notebooks will be available for you to download.

1 of 39
Download to read offline
Jump Start into Apache® Spark™ and Databricks Denny Lee, Technology Evangelist denny@databricks.com, @dennylee
Technology Evangelist,Databricks (Working with Spark since v0.5) Formerly: • SeniorDirectorof Data Sciences EngineeringatConcur(nowpart of SAP) • PrincipalProgramManager at Microsoft Hands-on Data Engineer: Architect for morethan 15 years, developinginternet-scaleinfrastructure for both on-premisesandcloud includingBing’sAudienceInsights, Yahoo’s 24TB SSAS cube, and IsotopeIncubation Team (HDInsight). About Me: Denny Lee
Founded by the creators of Apache Spark in 2013 Share of Spark code contributed by Databricks in 2014 75% 3 Data Value Created Databricks on top of Sparkto make big data simple. We are Databricks, the company behind Spark.
… Apache Spark Engine Spark Core Spark Streaming Spark SQL MLlib GraphX Unified engineacross diverse workloads & environments Scale out, fault tolerant Python, Java, Scala, and R APIs Standard libraries
Open Source Ecosystem
Large-Scale Usage Largest cluster 8000 Nodes (Tencent) Largest single job 1 PB (Alibaba, Databricks) Top Streaming Intake 1 TB/hour (HHMI Janelia Farm) 2014 On-Disk SortRecord Fastest Open Source Engine for sorting a PB
Notable Users Source: Slide 5 of Spark Community Update Companies That Presented at Spark Summit 2015 in San Francisco
Quick Start Quick Start Using Python | Quick Start Using Scala
Quick Start with Python textFile = sc.textFile("/mnt/tardis6/docs/README.md") textFile.count()
Quick Start with Scala textFile = sc.textFile("/mnt/tardis6/docs/README.md") textFile.count()
RDDs • RDDs have actions, which return values,and transformations, which return pointersto new RDDs. • Transformations are lazy and executed when an action is run • Transformations: map(), flatMap(), filter(), mapPartitions(), mapPartitionsWithIndex(), sample(), union(), distinct(), groupByKey(), reduceByKey(), sortByKey(), join(), cogroup(), pipe(), coalesce(), repartition(), partitionBy(), ... • Actions: reduce(), collect(), count(), first(), take(), takeSample(), takeOrdered(), saveAsTextFile(), saveAsSequenceFile(), saveAsObjectFile(), countByKey(), foreach(), ... • Persist (cache) distributed data in memory or disk
Spark API Performance
History of Spark APIs RDD (2011) DataFrame (2013) • Distributecollection of JVM objects • Functional Operators (map, filter, etc.) • Distributecollection of Rowobjects • Expression-based operations and UDFs • Logical plans and optimizer • Fast/efficient internal representations DataSet (2015) • Internally rows,externally JVM objects • “Best ofboth worlds” type safe + fast
Benefit of Logical Plan: Performance Parity Across Languages 0 2 4 6 8 10 Java/Scala Python Java/Scala Python R SQL Runtime for an example aggregation workload (secs) DataFrame RDD
0 50 100 150 200 250 300 350 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Spark 1.3.1, 1.4, and 1.5 for 9 queries 1.5 Run A1 1.5 Run A2 1.5 Run B1 1.5 Run B2 1.4 Run A1 1.4 Run A2 NYC Taxi Dataset
Dataset API in Spark 1.6 Typed interface over DataFrames / Tungsten case class Person(name: String, age: Long) val dataframe = read.json(“people.json”) val ds: Dataset[Person] = dataframe.as[Person] ds.filter(p => p.name.startsWith(“M”)) .toDF() .groupBy($“name”) .avg(“age”)
Dataset “Encoder” convertsfromJVM Object into a Dataset Row Checkout[SPARK-9999] JVM Object Dataset Row encoder
Tungsten Execution PythonSQL R Streaming DataFrame (& Dataset) Advanced Analytics
Ad Tech Example AdTech Sample Notebook (Part 1)
Create External Table with RegEx CREATE EXTERNAL TABLE accesslog ( ipaddress STRING, ... ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = '^(S+) (S+) (S+) [([w:/]+s[+-]d{4})] "(S+) (S+) (S+)" (d{3}) (d+) "(.*)" "(.*)" (S+) "(S+), (S+), (S+), (S+)"’ ) LOCATION "/mnt/mdl/accesslogs/"
External Web Service Call via Mapper # Obtain the unique agents from the accesslog table ipaddresses = sqlContext.sql("select distinct ip1 from accesslog where ip1 is not null").rdd # getCCA2: Obtains two letter country code based on IP address def getCCA2(ip): url = 'http://freegeoip.net/csv/' + ip str = urllib2.urlopen(url).read() return str.split(",")[1] # Loop through distinct IP addresses and obtain two-letter country codes mappedIPs = ipaddresses.map(lambda x: (x[0], getCCA2(x[0])))
Join DataFrames and Register Temp Table # Join countrycodes with mappedIPsDF so we can have IP address and # three-letter ISO country codes mappedIP3 = mappedIP2 .join(countryCodesDF, mappedIP2.cca2 == countryCodesDF.cca2, "left_outer") .select(mappedIP2.ip, mappedIP2.cca2, countryCodesDF.cca3, countryCodesDF.cn) # Register the mapping table mappedIP3.registerTempTable("mappedIP3")
Add Columns to DataFrames with UDFs from user_agents import parse from pyspark.sql.types import StringType from pyspark.sql.functions import udf # Create UDFs to extract out Browser Family information def browserFamily(ua_string) : return xstr(parse(xstr(ua_string)).browser.family) udfBrowserFamily = udf(browserFamily, StringType()) # Obtain the unique agents from the accesslog table userAgentTbl = sqlContext.sql("select distinct agent from accesslog") # Add new columns to the UserAgentInfo DataFrame containing browser information userAgentInfo = userAgentTbl.withColumn('browserFamily', udfBrowserFamily(userAgentTbl.agent))
Use Python UDFs with Spark SQL # Define function (converts Apache web log time) def weblog2Time(weblog_timestr): ... # Define and Register UDF udfWeblog2Time = udf(weblog2Time, DateType()) sqlContext.registerFunction("udfWeblog2Time", lambda x: weblog2Time(x)) # Create DataFrame accessLogsPrime = sqlContext.sql("select hash(a.ip1, a.agent) as UserId, m.cca3, udfWeblog2Time(a.datetime),...")udfWeblog2Time(a.datetime)
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
References Spark DataFrames: Simple and FastAnalysison Structured Data [Michael Armbrust] Apache Spark 1.6 presented by Databricks co-founderPatrick Wendell Announcing Spark1.6 Introducing Spark Datasets Spark SQL Data Sources API: Unified Data Access for the Spark Platform
Join us at Spark Summit East February16-18, 2016 | New York City
Thanks!
Appendix
Spark Survey 2015 Highlights
Spark adoption is growing rapidly Spark use is growing beyond Hadoop Spark is increasing access to big data Spark Survey Report 2015 Highlights TOP 3 APACHE SPARK TAKEAWAYS
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
NOTABLE USERS THAT PRESENTED AT SPARK SUMMIT 2015 SAN FRANCISCO Source: Slide 5 of Spark Community Update

More Related Content

Jump Start into Apache® Spark™ and Databricks

  • 1. Jump Start into Apache® Spark™ and Databricks Denny Lee, Technology Evangelist denny@databricks.com, @dennylee
  • 2. Technology Evangelist,Databricks (Working with Spark since v0.5) Formerly: • SeniorDirectorof Data Sciences EngineeringatConcur(nowpart of SAP) • PrincipalProgramManager at Microsoft Hands-on Data Engineer: Architect for morethan 15 years, developinginternet-scaleinfrastructure for both on-premisesandcloud includingBing’sAudienceInsights, Yahoo’s 24TB SSAS cube, and IsotopeIncubation Team (HDInsight). About Me: Denny Lee
  • 3. Founded by the creators of Apache Spark in 2013 Share of Spark code contributed by Databricks in 2014 75% 3 Data Value Created Databricks on top of Sparkto make big data simple. We are Databricks, the company behind Spark.
  • 4. … Apache Spark Engine Spark Core Spark Streaming Spark SQL MLlib GraphX Unified engineacross diverse workloads & environments Scale out, fault tolerant Python, Java, Scala, and R APIs Standard libraries
  • 6. Large-Scale Usage Largest cluster 8000 Nodes (Tencent) Largest single job 1 PB (Alibaba, Databricks) Top Streaming Intake 1 TB/hour (HHMI Janelia Farm) 2014 On-Disk SortRecord Fastest Open Source Engine for sorting a PB
  • 7. Notable Users Source: Slide 5 of Spark Community Update Companies That Presented at Spark Summit 2015 in San Francisco
  • 8. Quick Start Quick Start Using Python | Quick Start Using Scala
  • 9. Quick Start with Python textFile = sc.textFile("/mnt/tardis6/docs/README.md") textFile.count()
  • 10. Quick Start with Scala textFile = sc.textFile("/mnt/tardis6/docs/README.md") textFile.count()
  • 11. RDDs • RDDs have actions, which return values,and transformations, which return pointersto new RDDs. • Transformations are lazy and executed when an action is run • Transformations: map(), flatMap(), filter(), mapPartitions(), mapPartitionsWithIndex(), sample(), union(), distinct(), groupByKey(), reduceByKey(), sortByKey(), join(), cogroup(), pipe(), coalesce(), repartition(), partitionBy(), ... • Actions: reduce(), collect(), count(), first(), take(), takeSample(), takeOrdered(), saveAsTextFile(), saveAsSequenceFile(), saveAsObjectFile(), countByKey(), foreach(), ... • Persist (cache) distributed data in memory or disk
  • 13. History of Spark APIs RDD (2011) DataFrame (2013) • Distributecollection of JVM objects • Functional Operators (map, filter, etc.) • Distributecollection of Rowobjects • Expression-based operations and UDFs • Logical plans and optimizer • Fast/efficient internal representations DataSet (2015) • Internally rows,externally JVM objects • “Best ofboth worlds” type safe + fast
  • 14. Benefit of Logical Plan: Performance Parity Across Languages 0 2 4 6 8 10 Java/Scala Python Java/Scala Python R SQL Runtime for an example aggregation workload (secs) DataFrame RDD
  • 15. 0 50 100 150 200 250 300 350 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Spark 1.3.1, 1.4, and 1.5 for 9 queries 1.5 Run A1 1.5 Run A2 1.5 Run B1 1.5 Run B2 1.4 Run A1 1.4 Run A2 NYC Taxi Dataset
  • 16. Dataset API in Spark 1.6 Typed interface over DataFrames / Tungsten case class Person(name: String, age: Long) val dataframe = read.json(“people.json”) val ds: Dataset[Person] = dataframe.as[Person] ds.filter(p => p.name.startsWith(“M”)) .toDF() .groupBy($“name”) .avg(“age”)
  • 17. Dataset “Encoder” convertsfromJVM Object into a Dataset Row Checkout[SPARK-9999] JVM Object Dataset Row encoder
  • 18. Tungsten Execution PythonSQL R Streaming DataFrame (& Dataset) Advanced Analytics
  • 19. Ad Tech Example AdTech Sample Notebook (Part 1)
  • 20. Create External Table with RegEx CREATE EXTERNAL TABLE accesslog ( ipaddress STRING, ... ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = '^(S+) (S+) (S+) [([w:/]+s[+-]d{4})] "(S+) (S+) (S+)" (d{3}) (d+) "(.*)" "(.*)" (S+) "(S+), (S+), (S+), (S+)"’ ) LOCATION "/mnt/mdl/accesslogs/"
  • 21. External Web Service Call via Mapper # Obtain the unique agents from the accesslog table ipaddresses = sqlContext.sql("select distinct ip1 from accesslog where ip1 is not null").rdd # getCCA2: Obtains two letter country code based on IP address def getCCA2(ip): url = 'http://freegeoip.net/csv/' + ip str = urllib2.urlopen(url).read() return str.split(",")[1] # Loop through distinct IP addresses and obtain two-letter country codes mappedIPs = ipaddresses.map(lambda x: (x[0], getCCA2(x[0])))
  • 22. Join DataFrames and Register Temp Table # Join countrycodes with mappedIPsDF so we can have IP address and # three-letter ISO country codes mappedIP3 = mappedIP2 .join(countryCodesDF, mappedIP2.cca2 == countryCodesDF.cca2, "left_outer") .select(mappedIP2.ip, mappedIP2.cca2, countryCodesDF.cca3, countryCodesDF.cn) # Register the mapping table mappedIP3.registerTempTable("mappedIP3")
  • 23. Add Columns to DataFrames with UDFs from user_agents import parse from pyspark.sql.types import StringType from pyspark.sql.functions import udf # Create UDFs to extract out Browser Family information def browserFamily(ua_string) : return xstr(parse(xstr(ua_string)).browser.family) udfBrowserFamily = udf(browserFamily, StringType()) # Obtain the unique agents from the accesslog table userAgentTbl = sqlContext.sql("select distinct agent from accesslog") # Add new columns to the UserAgentInfo DataFrame containing browser information userAgentInfo = userAgentTbl.withColumn('browserFamily', udfBrowserFamily(userAgentTbl.agent))
  • 24. Use Python UDFs with Spark SQL # Define function (converts Apache web log time) def weblog2Time(weblog_timestr): ... # Define and Register UDF udfWeblog2Time = udf(weblog2Time, DateType()) sqlContext.registerFunction("udfWeblog2Time", lambda x: weblog2Time(x)) # Create DataFrame accessLogsPrime = sqlContext.sql("select hash(a.ip1, a.agent) as UserId, m.cca3, udfWeblog2Time(a.datetime),...")udfWeblog2Time(a.datetime)
  • 29. References Spark DataFrames: Simple and FastAnalysison Structured Data [Michael Armbrust] Apache Spark 1.6 presented by Databricks co-founderPatrick Wendell Announcing Spark1.6 Introducing Spark Datasets Spark SQL Data Sources API: Unified Data Access for the Spark Platform
  • 30. Join us at Spark Summit East February16-18, 2016 | New York City
  • 34. Spark adoption is growing rapidly Spark use is growing beyond Hadoop Spark is increasing access to big data Spark Survey Report 2015 Highlights TOP 3 APACHE SPARK TAKEAWAYS
  • 39. NOTABLE USERS THAT PRESENTED AT SPARK SUMMIT 2015 SAN FRANCISCO Source: Slide 5 of Spark Community Update