Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Fossasia 2018-chetan-khatri

•
•
Chetan Khatri
Chetan Khatri

Apache Spark and Scala DSL can be used to scale processing of TBs of data at production. Spark provides high-level APIs for Scala, Java, Python and R and an optimized engine for distributed execution. The talk discusses Spark core concepts like RDDs and DataFrames/Datasets. It also presents a case study of re-engineering a retail data platform using Spark to enable real-time processing of billions of events and records from a data lake and warehouse in a highly concurrent and elastic manner. Techniques like parallelization of jobs, hyperparameter tuning, physical data splitting and frequent batch processing were used to achieve a 5-10x performance improvement.

1 of 48
Download to read offline
Scaling TB’s of data with Apache Spark and Scala DSL at Production Chetan Khatri FOSSASIA Summit, 2018 @khatri_chetan
Accionlabs Data Engineering | Fast Data WHO AM I ? Lead - Data Science, Technology Evangelist @ Accion labs India Pvt. Ltd. Committer @ Apache Spark, Apache HBase, Elixir Lang. Co-Authored University Curriculum @ University of Kachchh. Software Engineering @: Nazara Games, Eccella Corporation. M.Sc. - Computer Science from University of Kachchh.
Accionlabs Data Engineering | Fast Data Agenda Apache Spark and Scala Resilient Distributed Datasets (RDDs) DataFrames and Datasets Spark Operations Data Platform Components Re-engineering Data processing platform Rethink - Fast Data Architecture Parallelism & Concurrency at Spark
Accionlabs Data Engineering | Fast Data What is Apache Spark ? Apache Spark is a fast and general-purpose cluster computing system / Unified Engine for massive data processing. It provides high level API for Scala, Java, Python and R and optimized engine that supports general execution graphs. Structured Data / SQL - Spark SQL Graph Processing - GraphX Machine Learning - MLlib Streaming - Spark Streaming, Structured Streaming
Accionlabs Data Engineering | Fast Data What is Scala ? Scala is a modern multi-paradigm programming language designed to express common programming patterns in a concise, elegant, and type-safe way. Scala is object-oriented Scala is functional Strongly typed, Type Inference Higher Order Functions Lazy Computation
Accionlabs Data Engineering | Fast Data Data Structures in Apache Spark ? RDD DataFrame DataSet
Accionlabs Data Engineering | Fast Data What are RDDs ?
Accionlabs Data Engineering | Fast Data 1. Distributed Data Abstraction RDD RDD RDD RDD Logical Model Across Distributed Storage on Cluster HDFS, S3
Accionlabs Data Engineering | Fast Data 2. Resilient & Immutable RDD RDD RDD T T RDD -> T -> RDD -> T -> RDD T = Transformation
Accionlabs Data Engineering | Fast Data 3. Compile-time Type Safe / Strongly type inference Integer RDD String or Text RDD Double or Binary RDD
Accionlabs Data Engineering | Fast Data 4. Lazy evaluation RDD RDD RDD T T RDD RDD RDD T A RDD - T - RDD - T - RDD - T - RDD - A - RDD T = Transformation A = Action
Accionlabs Data Engineering | Fast Data Apache Spark Operations Operations Transformation Action
Accionlabs Data Engineering | Fast Data Essential Spark Operations TRANSFORMATIONSACTIONS General Math / Statistical Set Theory / Relational Data Structure / I/O map gilter flatMap mapPartitions mapPartitionsWithIndex groupBy sortBy sample randomSplit union intersection subtract distinct cartesian zip keyBy zipWithIndex zipWithUniqueID zipPartitions coalesce repartition repartitionAndSortWithinPartitions pipe reduce collect aggregate fold first take forEach top treeAggregate treeReduce forEachPartition collectAsMap count takeSample max min sum histogram mean variance stdev sampleVariance countApprox countApproxDistinct takeOrdered saveAsTextFile saveAsSequenceFile saveAsObjectFile saveAsHadoopDataset saveAsHadoopFile saveAsNewAPIHadoopDataset saveAsNewAPIHadoopFile
Accionlabs Data Engineering | Fast Data When to use RDDs ? You care about control of dataset and knows how data looks like, you care about low level API. Don’t care about lot’s of lambda functions than DSL. Don’t care about Schema or Structure of Data. Don’t care about optimization, performance & inefficiencies! Very slow for non-JVM languages like Python, R. Don’t care about Inadvertent inefficiencies.
Accionlabs Data Engineering | Fast Data Inadvertent inefficiencies in RDDs parsedRDD.filter { case (project, sprint, numStories) => project == "finance" }. map { case (_, sprint, numStories) => (sprint, numStories) }. reduceByKey(_ + _). filter { case (sprint, _) => !isSpecialSprint(sprint) }. take(100).foreach { case (project, stories) => println(s"project: $stories") }
Accionlabs Data Engineering | Fast Data Structured in Spark DataFrames Datasets
Accionlabs Data Engineering | Fast Data Why Dataset ? Strongly Typing Ability to use powerful lambda functions. Spark SQL’s optimized execution engine (catalyst, tungsten) Can be constructed from JVM objects & manipulated using Functional transformations (map, filter, flatMap etc) A DataFrame is a Dataset organized into named columns DataFrame is simply a type alias of Dataset[Row]
Accionlabs Data Engineering | Fast Data Structured APIs in Apache Spark SQL DataFrames Datasets Syntax Errors Runtime Compile Time Compile Time Analysis Errors Runtime Runtime Compile Time Analysis errors are caught before a job runs on cluster
Accionlabs Data Engineering | Fast Data Unification of APIs in Apache Spark 2.0 DataFrame Dataset Untyped API Typed API Dataset (2016) DataFrame = Dataset [Row] Alias Dataset [T]
Accionlabs Data Engineering | Fast Data DataFrame API Code // convert RDD -> DF with column names val parsedDF = parsedRDD.toDF("project", "sprint", "numStories") //filter, groupBy, sum, and then agg() parsedDF.filter($"project" === "finance"). groupBy($"sprint"). agg(sum($"numStories").as("count")). limit(100). show(100) project sprint numStories finance 3 20 finance 4 22
Accionlabs Data Engineering | Fast Data DataFrame -> SQL View -> SQL Query parsedDF.createOrReplaceTempView("audits") val results = spark.sql( """SELECT sprint, sum(numStories) AS count FROM audits WHERE project = 'finance' GROUP BY sprint LIMIT 100""") results.show(100) project sprint numStories finance 3 20 finance 4 22
Accionlabs Data Engineering | Fast Data Why Structure APIs ? // DataFrame data.groupBy("dept").avg("age") // SQL select dept, avg(age) from data group by 1 // RDD data.map { case (dept, age) => dept -> (age, 1) } .reduceByKey { case ((a1, c1), (a2, c2)) => (a1 + a2, c1 + c2) } .map { case (dept, (age, c)) => dept -> age / c }
Accionlabs Data Engineering | Fast Data Catalyst in Spark SQL AST DataFrame Datasets Unresolved Logical Plan Logical Plan Optimized Logical Plan Physical Plans CostModel Selected Physical Plan RDD
Accionlabs Data Engineering | Fast Data Dataset API in Spark 2.x val employeesDF = spark.read.json("employees.json") // Convert data to domain objects. case class Employee(name: String, age: Int) val employeesDS: Dataset[Employee] = employeesDF.as[Employee] val filterDS = employeesDS.filter(p => p.age > 3) Type-safe: operate on domain objects with compiled lambda functions.
Accionlabs Data Engineering | Fast Data Example: DataFrame Optimization employees.join(events, employees("id") === events("eid")) .filter(events("date") > "2015-01-01") events file employees table join filter Logical Plan scan (employees) filter Scan (events) join Physical Plan Optimized scan (events) Optimized scan (employees) join Physical Plan With Predicate Pushdown and Column Pruning
Accionlabs Data Engineering | Fast Data DataFrames are Faster than RDDs Source: Databricks
Accionlabs Data Engineering | Fast Data Datasets takes less Memory than RDDs Source: Databricks
Accionlabs Data Engineering | Fast Data Datasets are faster
Accionlabs Data Engineering | Fast Data Case Study
Accionlabs Data Engineering | Fast Data Components of your Data Platform Data Warehouse Fast Queries Transactional Reliability Data Lake Low Cost Massive Scale Streaming Message Bus (e.g. Kinesis, Apache Kafka) Low Latency
Accionlabs Data Engineering | Fast Data How to make them talk? DATA WAREHOUSE Fast Queries Transactional Reliability DATA LAKE Low Cost Massive Scale STREAMING MESSAGE BUS (e.g. Kinesis, Apache Kafka) Low Latency Complex, Slow, ETL Process
Accionlabs Data Engineering | Fast Data Changing the Game with a Re-engineering Data Processing Platform: Retail Business Challenges Weekly Data refresh, Daily Data refresh batch Spark / Hadoop job execution failures with unutilized Spark Cluster. Scalability of massive data: ○ ~4.6 Billion events on every weekly data refresh. ○ Processing historical data: one time bulk load ~30 TB of data / ~17 billion transactional records. Linear / sequential execution mode with broken data pipelines. Joining ~17 billion transactional records with skewed data nature. Data deduplication - outlet, item at retail. Business: explain the who, what, when, where, why and how of retailing.
Accionlabs Data Engineering | Fast Data Solution : Retail Business 5x performance improvements by re-engineering entire data lake to analytical engine pipeline’s. Proposed highly concurrent, elastic, non-blocking, asynchronous architecture to save customer’s ~22 hours runtime (~8 hours from 30 hours) for 4.6 Billion events. 10x performance improvements on historical load by under the hood algorithms optimization on 17 Billion events (Benchmarks ~1 hour execution time only) Master Data Management (MDM) - Deduplication and fuzzy logic matching on retails data(Item, Outlet) improved elastic performance.
Accionlabs Data Engineering | Fast Data How ? Source: https://www.caresearch.com.au
Accionlabs Data Engineering | Fast Data Event Queuing Kafka Batch Transformations Reporting Tool (Dashboard) Processed DW - Ad hoc Analysis / 1st level aggregation Data Ingestion Pipeline RestfulAPI’s Summarized DB - KPI Reporting
Accionlabs Data Engineering | Fast Data Rethink - Fast Data Architectures UNIFIED fast data processing engine that provides: The SCALE of data lake The RELIABILITY & PERFORMANCE of data warehouse The LOW LATENCY of streaming
Accionlabs Data Engineering | Fast Data #1: Sequential job execution to parallel job execution Outlet Item Manufacturer Transactions
Accionlabs Data Engineering | Fast Data #1: Sequential job execution to parallel job execution (...) Outlets Items organization Transactions Files Outlets-by-file Transaction-by-day
Accionlabs Data Engineering | Fast Data #2: Spark / Hive Data processing: Hyper parameters tuning Dynamic resource allocation and Yarn external shuffle service has been enabled at YARN resource manager to allocate resources dynamically to other jobs on yarn. Overallocation of resources for small jobs: Those spark jobs were taking more resources but they were less DISK volume intensive, Hyper-parameters - executors, cores, memory on executor/ driver has been reduced to allow other pending jobs to execute and not to block entire cluster. i.e Job will run little slowly but utilize entire cluster with non-blocking approach for other jobs.
Accionlabs Data Engineering | Fast Data #2: Spark / Hive Data processing: Hyper parameters tuning: Example $SPARK_HOME/bin/spark-submit --class com.talk.chetan.SparkHistoricalJobSample --master yarn --deploy-mode cluster --driver-memory 6g --executors 12 --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=true executor-memory 30g --executor-cores 10 $SPARK_JAR_JOB_PATH For number tasks spark is executing on cluster Number of Parallel tasks = number of executors * cores Ex. number of executors = 8, cores = 8 := 64 Parallel tasks
Accionlabs Data Engineering | Fast Data #2: Spark / Hive Data processing: Hyper parameters tuning (...) Our optimization approaches: 1) Reduce memory / cores & increase executors: This can allow us to better utilize all the resources on the cluster without locking others out. 2) Reduce executors and use the same memory / cores: This will run slower for that run, but will allow others to use the cluster box in parallel.
Accionlabs Data Engineering | Fast Data #3: Physical Data Split up techniques for Analytical Engine (Hive) Physical data split up: Spark map-reduce transformations generates small-small files on larger dataset at some extents which led to increase Disk I/O, Memory I/O, File I/O, Bandwidth I/O etc. Also Downstream Hive Queries, Spark Jobs get impacts on performance and sometime it also fails with disk quota exceeded, container lost etc exceptions. In accordance with business logic and optimized new data model design, physical data split up techniques such as Partitioning, Re-partitioning, Coalesce applied on dimensions of data model with highly optimized, tuned other spark internals techniques to reduce N numbers of files generation. Which made drastic performance change at slice and dice on measures.
Accionlabs Data Engineering | Fast Data #4: Not everything is Streaming at Data Lake ! (~>) Frequent batch mode If you are going beyond few min’s, you should not be doing streaming at all, you should just kick-off batch jobs very frequently. If you run something long enough for example, streaming job runs for 4 months, eventually you can see all possible problems network partition, hardware failures, GC, spike & traffic on appcache etc. All kinds of things ! Kicking off batch job and immediately scale to right size it needs to be. It does it’s work & goes away.
Accionlabs Data Engineering | Fast Data #5: Historical Data Processing: Aggregation of data on ~17 Billion records file_errors Join transaction_by_file outlets_by_file items_by_file Hive External table with parquet partitioned Redshift transactions_error 1 Failed: * Disk quota exceeded * Executor lost failure * container killed by YARN on exceeding memory limits ~27 TB of total historical data Hive transactions_error (Managed table, parquet format with non-partitioned data) Success approach: * Enable YARN external shuffle service * Enable Dynamic Resource Allocation * Tune hyper-parameters to utilize cluster * Apply business transformation here * create temp view * insert to hive managed table 2 Failed
Accionlabs Data Engineering | Fast Data ~1 TB of Output Data Shuffle All executors at entire cluster are utilized
Accionlabs Data Engineering | Fast Data Enabler ? Open Source
Questions ?
Thank you for your Time!

More Related Content

Fossasia 2018-chetan-khatri

  • 1. Scaling TB’s of data with Apache Spark and Scala DSL at Production Chetan Khatri FOSSASIA Summit, 2018 @khatri_chetan
  • 2. Accionlabs Data Engineering | Fast Data WHO AM I ? Lead - Data Science, Technology Evangelist @ Accion labs India Pvt. Ltd. Committer @ Apache Spark, Apache HBase, Elixir Lang. Co-Authored University Curriculum @ University of Kachchh. Software Engineering @: Nazara Games, Eccella Corporation. M.Sc. - Computer Science from University of Kachchh.
  • 3. Accionlabs Data Engineering | Fast Data Agenda Apache Spark and Scala Resilient Distributed Datasets (RDDs) DataFrames and Datasets Spark Operations Data Platform Components Re-engineering Data processing platform Rethink - Fast Data Architecture Parallelism & Concurrency at Spark
  • 4. Accionlabs Data Engineering | Fast Data What is Apache Spark ? Apache Spark is a fast and general-purpose cluster computing system / Unified Engine for massive data processing. It provides high level API for Scala, Java, Python and R and optimized engine that supports general execution graphs. Structured Data / SQL - Spark SQL Graph Processing - GraphX Machine Learning - MLlib Streaming - Spark Streaming, Structured Streaming
  • 5. Accionlabs Data Engineering | Fast Data What is Scala ? Scala is a modern multi-paradigm programming language designed to express common programming patterns in a concise, elegant, and type-safe way. Scala is object-oriented Scala is functional Strongly typed, Type Inference Higher Order Functions Lazy Computation
  • 6. Accionlabs Data Engineering | Fast Data Data Structures in Apache Spark ? RDD DataFrame DataSet
  • 7. Accionlabs Data Engineering | Fast Data What are RDDs ?
  • 8. Accionlabs Data Engineering | Fast Data 1. Distributed Data Abstraction RDD RDD RDD RDD Logical Model Across Distributed Storage on Cluster HDFS, S3
  • 9. Accionlabs Data Engineering | Fast Data 2. Resilient & Immutable RDD RDD RDD T T RDD -> T -> RDD -> T -> RDD T = Transformation
  • 10. Accionlabs Data Engineering | Fast Data 3. Compile-time Type Safe / Strongly type inference Integer RDD String or Text RDD Double or Binary RDD
  • 11. Accionlabs Data Engineering | Fast Data 4. Lazy evaluation RDD RDD RDD T T RDD RDD RDD T A RDD - T - RDD - T - RDD - T - RDD - A - RDD T = Transformation A = Action
  • 12. Accionlabs Data Engineering | Fast Data Apache Spark Operations Operations Transformation Action
  • 13. Accionlabs Data Engineering | Fast Data Essential Spark Operations TRANSFORMATIONSACTIONS General Math / Statistical Set Theory / Relational Data Structure / I/O map gilter flatMap mapPartitions mapPartitionsWithIndex groupBy sortBy sample randomSplit union intersection subtract distinct cartesian zip keyBy zipWithIndex zipWithUniqueID zipPartitions coalesce repartition repartitionAndSortWithinPartitions pipe reduce collect aggregate fold first take forEach top treeAggregate treeReduce forEachPartition collectAsMap count takeSample max min sum histogram mean variance stdev sampleVariance countApprox countApproxDistinct takeOrdered saveAsTextFile saveAsSequenceFile saveAsObjectFile saveAsHadoopDataset saveAsHadoopFile saveAsNewAPIHadoopDataset saveAsNewAPIHadoopFile
  • 14. Accionlabs Data Engineering | Fast Data When to use RDDs ? You care about control of dataset and knows how data looks like, you care about low level API. Don’t care about lot’s of lambda functions than DSL. Don’t care about Schema or Structure of Data. Don’t care about optimization, performance & inefficiencies! Very slow for non-JVM languages like Python, R. Don’t care about Inadvertent inefficiencies.
  • 15. Accionlabs Data Engineering | Fast Data Inadvertent inefficiencies in RDDs parsedRDD.filter { case (project, sprint, numStories) => project == "finance" }. map { case (_, sprint, numStories) => (sprint, numStories) }. reduceByKey(_ + _). filter { case (sprint, _) => !isSpecialSprint(sprint) }. take(100).foreach { case (project, stories) => println(s"project: $stories") }
  • 16. Accionlabs Data Engineering | Fast Data Structured in Spark DataFrames Datasets
  • 17. Accionlabs Data Engineering | Fast Data Why Dataset ? Strongly Typing Ability to use powerful lambda functions. Spark SQL’s optimized execution engine (catalyst, tungsten) Can be constructed from JVM objects & manipulated using Functional transformations (map, filter, flatMap etc) A DataFrame is a Dataset organized into named columns DataFrame is simply a type alias of Dataset[Row]
  • 18. Accionlabs Data Engineering | Fast Data Structured APIs in Apache Spark SQL DataFrames Datasets Syntax Errors Runtime Compile Time Compile Time Analysis Errors Runtime Runtime Compile Time Analysis errors are caught before a job runs on cluster
  • 19. Accionlabs Data Engineering | Fast Data Unification of APIs in Apache Spark 2.0 DataFrame Dataset Untyped API Typed API Dataset (2016) DataFrame = Dataset [Row] Alias Dataset [T]
  • 20. Accionlabs Data Engineering | Fast Data DataFrame API Code // convert RDD -> DF with column names val parsedDF = parsedRDD.toDF("project", "sprint", "numStories") //filter, groupBy, sum, and then agg() parsedDF.filter($"project" === "finance"). groupBy($"sprint"). agg(sum($"numStories").as("count")). limit(100). show(100) project sprint numStories finance 3 20 finance 4 22
  • 21. Accionlabs Data Engineering | Fast Data DataFrame -> SQL View -> SQL Query parsedDF.createOrReplaceTempView("audits") val results = spark.sql( """SELECT sprint, sum(numStories) AS count FROM audits WHERE project = 'finance' GROUP BY sprint LIMIT 100""") results.show(100) project sprint numStories finance 3 20 finance 4 22
  • 22. Accionlabs Data Engineering | Fast Data Why Structure APIs ? // DataFrame data.groupBy("dept").avg("age") // SQL select dept, avg(age) from data group by 1 // RDD data.map { case (dept, age) => dept -> (age, 1) } .reduceByKey { case ((a1, c1), (a2, c2)) => (a1 + a2, c1 + c2) } .map { case (dept, (age, c)) => dept -> age / c }
  • 23. Accionlabs Data Engineering | Fast Data Catalyst in Spark SQL AST DataFrame Datasets Unresolved Logical Plan Logical Plan Optimized Logical Plan Physical Plans CostModel Selected Physical Plan RDD
  • 24. Accionlabs Data Engineering | Fast Data Dataset API in Spark 2.x val employeesDF = spark.read.json("employees.json") // Convert data to domain objects. case class Employee(name: String, age: Int) val employeesDS: Dataset[Employee] = employeesDF.as[Employee] val filterDS = employeesDS.filter(p => p.age > 3) Type-safe: operate on domain objects with compiled lambda functions.
  • 25. Accionlabs Data Engineering | Fast Data Example: DataFrame Optimization employees.join(events, employees("id") === events("eid")) .filter(events("date") > "2015-01-01") events file employees table join filter Logical Plan scan (employees) filter Scan (events) join Physical Plan Optimized scan (events) Optimized scan (employees) join Physical Plan With Predicate Pushdown and Column Pruning
  • 26. Accionlabs Data Engineering | Fast Data DataFrames are Faster than RDDs Source: Databricks
  • 27. Accionlabs Data Engineering | Fast Data Datasets takes less Memory than RDDs Source: Databricks
  • 28. Accionlabs Data Engineering | Fast Data Datasets are faster
  • 29. Accionlabs Data Engineering | Fast Data Case Study
  • 30. Accionlabs Data Engineering | Fast Data Components of your Data Platform Data Warehouse Fast Queries Transactional Reliability Data Lake Low Cost Massive Scale Streaming Message Bus (e.g. Kinesis, Apache Kafka) Low Latency
  • 31. Accionlabs Data Engineering | Fast Data How to make them talk? DATA WAREHOUSE Fast Queries Transactional Reliability DATA LAKE Low Cost Massive Scale STREAMING MESSAGE BUS (e.g. Kinesis, Apache Kafka) Low Latency Complex, Slow, ETL Process
  • 32. Accionlabs Data Engineering | Fast Data Changing the Game with a Re-engineering Data Processing Platform: Retail Business Challenges Weekly Data refresh, Daily Data refresh batch Spark / Hadoop job execution failures with unutilized Spark Cluster. Scalability of massive data: ○ ~4.6 Billion events on every weekly data refresh. ○ Processing historical data: one time bulk load ~30 TB of data / ~17 billion transactional records. Linear / sequential execution mode with broken data pipelines. Joining ~17 billion transactional records with skewed data nature. Data deduplication - outlet, item at retail. Business: explain the who, what, when, where, why and how of retailing.
  • 33. Accionlabs Data Engineering | Fast Data Solution : Retail Business 5x performance improvements by re-engineering entire data lake to analytical engine pipeline’s. Proposed highly concurrent, elastic, non-blocking, asynchronous architecture to save customer’s ~22 hours runtime (~8 hours from 30 hours) for 4.6 Billion events. 10x performance improvements on historical load by under the hood algorithms optimization on 17 Billion events (Benchmarks ~1 hour execution time only) Master Data Management (MDM) - Deduplication and fuzzy logic matching on retails data(Item, Outlet) improved elastic performance.
  • 34. Accionlabs Data Engineering | Fast Data How ? Source: https://www.caresearch.com.au
  • 35. Accionlabs Data Engineering | Fast Data Event Queuing Kafka Batch Transformations Reporting Tool (Dashboard) Processed DW - Ad hoc Analysis / 1st level aggregation Data Ingestion Pipeline RestfulAPI’s Summarized DB - KPI Reporting
  • 36. Accionlabs Data Engineering | Fast Data Rethink - Fast Data Architectures UNIFIED fast data processing engine that provides: The SCALE of data lake The RELIABILITY & PERFORMANCE of data warehouse The LOW LATENCY of streaming
  • 37. Accionlabs Data Engineering | Fast Data #1: Sequential job execution to parallel job execution Outlet Item Manufacturer Transactions
  • 38. Accionlabs Data Engineering | Fast Data #1: Sequential job execution to parallel job execution (...) Outlets Items organization Transactions Files Outlets-by-file Transaction-by-day
  • 39. Accionlabs Data Engineering | Fast Data #2: Spark / Hive Data processing: Hyper parameters tuning Dynamic resource allocation and Yarn external shuffle service has been enabled at YARN resource manager to allocate resources dynamically to other jobs on yarn. Overallocation of resources for small jobs: Those spark jobs were taking more resources but they were less DISK volume intensive, Hyper-parameters - executors, cores, memory on executor/ driver has been reduced to allow other pending jobs to execute and not to block entire cluster. i.e Job will run little slowly but utilize entire cluster with non-blocking approach for other jobs.
  • 40. Accionlabs Data Engineering | Fast Data #2: Spark / Hive Data processing: Hyper parameters tuning: Example $SPARK_HOME/bin/spark-submit --class com.talk.chetan.SparkHistoricalJobSample --master yarn --deploy-mode cluster --driver-memory 6g --executors 12 --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=true executor-memory 30g --executor-cores 10 $SPARK_JAR_JOB_PATH For number tasks spark is executing on cluster Number of Parallel tasks = number of executors * cores Ex. number of executors = 8, cores = 8 := 64 Parallel tasks
  • 41. Accionlabs Data Engineering | Fast Data #2: Spark / Hive Data processing: Hyper parameters tuning (...) Our optimization approaches: 1) Reduce memory / cores & increase executors: This can allow us to better utilize all the resources on the cluster without locking others out. 2) Reduce executors and use the same memory / cores: This will run slower for that run, but will allow others to use the cluster box in parallel.
  • 42. Accionlabs Data Engineering | Fast Data #3: Physical Data Split up techniques for Analytical Engine (Hive) Physical data split up: Spark map-reduce transformations generates small-small files on larger dataset at some extents which led to increase Disk I/O, Memory I/O, File I/O, Bandwidth I/O etc. Also Downstream Hive Queries, Spark Jobs get impacts on performance and sometime it also fails with disk quota exceeded, container lost etc exceptions. In accordance with business logic and optimized new data model design, physical data split up techniques such as Partitioning, Re-partitioning, Coalesce applied on dimensions of data model with highly optimized, tuned other spark internals techniques to reduce N numbers of files generation. Which made drastic performance change at slice and dice on measures.
  • 43. Accionlabs Data Engineering | Fast Data #4: Not everything is Streaming at Data Lake ! (~>) Frequent batch mode If you are going beyond few min’s, you should not be doing streaming at all, you should just kick-off batch jobs very frequently. If you run something long enough for example, streaming job runs for 4 months, eventually you can see all possible problems network partition, hardware failures, GC, spike & traffic on appcache etc. All kinds of things ! Kicking off batch job and immediately scale to right size it needs to be. It does it’s work & goes away.
  • 44. Accionlabs Data Engineering | Fast Data #5: Historical Data Processing: Aggregation of data on ~17 Billion records file_errors Join transaction_by_file outlets_by_file items_by_file Hive External table with parquet partitioned Redshift transactions_error 1 Failed: * Disk quota exceeded * Executor lost failure * container killed by YARN on exceeding memory limits ~27 TB of total historical data Hive transactions_error (Managed table, parquet format with non-partitioned data) Success approach: * Enable YARN external shuffle service * Enable Dynamic Resource Allocation * Tune hyper-parameters to utilize cluster * Apply business transformation here * create temp view * insert to hive managed table 2 Failed
  • 45. Accionlabs Data Engineering | Fast Data ~1 TB of Output Data Shuffle All executors at entire cluster are utilized
  • 46. Accionlabs Data Engineering | Fast Data Enabler ? Open Source
  • 48. Thank you for your Time!