Apache spark

Sentiment analysis on Twitter data has attracted much attention recently. One of the system’s key features, is the immediacy in communication with other users in an easy, user-friendly and fast way. Consequently, people tend to express their feelings freely, which makes Twitter an ideal source for accumulating a vast amount of opinions towards a wide diversity of topics. This amount of information oers huge potential and can be harnessed to receive the sentiment tendency towards these topics. However, since none can invest an innite amount of time to read through these tweets, an automated decision making approach is necessary. Nevertheless, most existing solutions are limited in centralized environments only. Thus, they can only process at most a few thousand tweets. Such a sample, is not representative to dene the sentiment polarity towards a topic due to the massive number of tweets published daily. In this paper, we go one step further and develop a novel method for sentiment le...

Due to the exponential increase in network traffic in the data centers, thousands of servers interconnected with high bandwidth switches are required. Field Programmable Gate Arrays (FPGAs) with Cloud ecosystem offer high performance in efficiency and energy, making them active resources, easy to program and reconfigure. This paper looks at FPGAs as reconfigurable accelerators for the cloud computing presents the main hardware accelerators that have been presented in various widely used cloud computing applications such as MapReduce, Spark, Memcached, Databases.

Recommender systems aim to personalize the shopping experience of a user by suggesting related products, or products that are found to be in the general interests of the user. The information available for users and products is heterogenous, and many systems use one or some of the information. The information available include the user's interactions history with the products and categories, textual information of the products, a hierarchical classification of the products into a taxonomy, user interests based on a questionnaire, the demographics of a user, inferred interests based on product reviews given by a user, interests based on the physical location of a user and so on. Taxonomy discovery for personalized recommendation is work published in 2014 which uses the first three information sources { the user's interaction history, textual information of the products and optionally, an existing taxonomy of the products. In this paper, we describe a parallel implementation o...

1. What is Apache Spark
2. How spark executes your program
3. Spark’s performance optimization
4. Memory Management Overview
5. Determining Memory Consumption
6. Partitions and Concurrency
7. Serialized RDD Storage
8. Garbage Collection Tuning
9. Broadcasting Large Variables
10. Spark’s performance tuning best practices

Data outsourcing allows data owners to keep their data at untrusted clouds that do not ensure the privacy of data and/or computations. One useful framework for fault-tolerant data processing in a distributed fashion is MapReduce, which was developed for trusted private clouds. This paper presents algorithms for data outsourcing based on Shamir's secret-sharing scheme and for executing privacy-preserving SQL queries such as count, selection including range selection, projection, and join while using MapReduce as an underlying programming model. The proposed algorithms prevent the untrusted cloud to know the database or the query while also preventing output size and access-pattern attacks. Interestingly, our algorithms do not need the database owner, which only creates and distributes secret-shares once, to be involved to answer any query, and hence, the database owner also cannot learn the query. We evaluate the efficiency of the algorithms on parameters: (i) the number of communication rounds (between a user and a cloud), (ii) the total amount of bit flow (between a user and a cloud), and (iii) the computational load at the user-side and the cloud-side.

There has been a strong growth in the global demand for energy over the past decade, especially in relation to renewable energy. The accelerating use of energy by household appliances, rapidly rising sales of air conditioners especially in emerging economies and the ever increasing need for heating in households has put a strain on energy demands across the world. As such, it has become more pertinent for energy producers to be able to accurately predict demand to avoid either over or under supply of electricity to households. In this paper, Apache Spark is used to process London smart meter data collected over a period of a year. The processed time-series data is then used to train a LSTM model for energy demand forecasting. When compared to using a single machine, the Spark cluster performed the same data processing task at a much shorter time, supporting the idea that its parallel architecture is more suitable for processing large datasets.

Apache Spark is an engine for large scale data processing, best described as a more flexible version of the venerable MapReduce framework with the capability to perform in-memory processing to speed up computation. BLAST (Basic Local Alignment Search Tool) is a tool used to determine the similarity of DNA and protein sequences that is used extensively in bioinformatics studies. To illustrate the use of Apache Spark in porting bioinformatics tools to the world of cluster computing, I demonstrate the creation of a distributed version of BLAST with minimal coding and simple cluster deployment. I measure performance, result accuracy, and cost efficiency to determine the viability of this approach to BLAST and other bioinformatics programs.

During the recent years, a number of efficient and scalable frequent itemset mining algorithms for big data analytics have been proposed by many researchers. Initially, MapReduce-based frequent itemset mining algorithms on Hadoop cluster were proposed. Although, Hadoop has been developed as a cluster computing system for handling and processing big data, but the performance of Hadoop does not meet the expectation for the iterative algorithms of data mining, due to its high I/O, and writing and then reading intermediate results in the disk. Consequently, Spark has been developed as another cluster computing infrastructure which is much faster than Hadoop due to its in-memory computation. It is highly suitable for iterative algorithms and supports batch, interactive, iterative, and stream processing of data. Many frequent itemset mining algorithms have been re-designed on the Spark, and most of them are Apriori-based. All these Spark-based Apriori algorithms use Hash Tree as the underlying data structure. This paper investigates the efficiency of various data structures for the Spark-based Apriori. Although, the data structure perspective has been investigated previously, but for MapReduce-based Apriori, and it must be re-investigated in the distributed computing environment of Spark. The considered underlying data structures are Hash Tree, Trie, and Hash Table Trie. The experimental results on the benchmark datasets show that the performance of Spark-based Apriori with Trie and Hash Table Trie are almost similar but both perform many times better than Hash Tree in the distributed computing environment of Spark.

( Alternatif Link download Buku Analisis Big Data: http://bit.ly/2x8ta9S )

Alhamdulillahhi robbil alamin, puji syukur kehadirat Allah SWT atas segala rahmat dan karunia-Nya dengan terselesaikannya penulisan buku ini dengan judul “Analisis Big Data”. Buku ini merupakan uraian untuk memudahkan pemahaman konsep, tingkat dasar sampai lanjut dalam sistem cerdas dan penerapannya melalui pemanfaatan teknologi Big Data, dengan mengedepankan keterampilan dalam pembuatan dan hasil implementasi dengan berbagai kombinasi algoritma berbasis sistem cerdas maupun dengan perpaduan berbagai macam tools untuk membangun ekosistem analisis Big Data yang powerfull. Konsep sederhana Analisis Big Data mencakup Volume, Velocity, dan Variety (3V), dan pengembangannya ada yang menyebut (7V) termasuk Volume, Velocity, Variety, Variability, Veracity, Value dan Visualization, atau 10V bahkan lebih dari itu, tetapi sebaiknya tidak membatasi pemahaman Big Data ini hanya dengan sedikit banyaknya istilah dari huruf V yang digunakan :). Kemudian dengan adanya ekosistem tersebut, jika dibutuhkan analisis sederhana maupun yang lebih kompleks, maka harapannya tidak akan ada kendala dari besarnya data yang diolah. Adanya kemajuan teknologi dalam hal penyimpanan, pengolahan, dan analisis Big Data meliputi (a) penurunan secara cepat terhadap biaya penyimpanan data dalam beberapa tahun terakhir; (b) fleksibilitas dan efektivitas biaya pada pusat data dan komputasi awan untuk perhitungan dengan konsep elastisitas dan penyimpanannya; serta (c) pengembangan kerangka kerja baru seperti Hadoop ecosystem (salah satu peluang bisnis yang besar untuk developer untuk saat ini dan ke depannya dalam rangka membangun ekosistem analisis Big Data yang sangat powerfull sekelas Cloudera, Hortonworks, etc), yang memungkinkan pengguna untuk mengambil manfaat dari sistem komputasi terdistribusi, misal untuk menyimpan sejumlah data yang besar melalui pemrosesan parallel, dukungan database NoSQL, dan komputasi berbasis streaming. Sehingga kemajuan teknologi ini telah menciptakan beberapa perbedaan yang sangat signifikan, misal dalam hal kecepatan maupun ketepatan dari hasil yang didapatkan antara analisis tradisional dengan tools yang bukan dengan konsep Big Data versus analisis modern untuk saat ini dan masa depan dengan membangun ekosistem Big Data yang sangat powerfull.
Penulis mengucapkan terimakasih yang sebesar-besarnya kepada kedua orang tua penulis, spesial kepada Istriku tercinta “Alfun Lailatul Musabbikha”, bapak ibu mertua, dik Wiwik, dik Ganda, kak Mat, kak Deri, Bude Gi, Pakde Dul, Bude Gani, Pak Abu, mbak Nik, dik Andre, Mbak As, Mbak Lia, Mas Mualif, Mbak Lis, Pakde Naji, Bulik Yul, Pakde Tasrun, Pakde Zen, Pakde Zud, Paklik Fiq, mas Bowo dan semua sanak saudaraku yang tak bisa kusebutkan seluruhnya, serta beberapa pihak yang telah membantu penyelesaian buku ini:
1. Para penulis artikel Analisis Big Data di forum, web, blog dan buku yang menjadi referensi buku ini untuk memberikan masukan yang sangat berharga sekali untuk perbaikan dan penyelesaian buku ini.
2. Mbak Efi Riyandani, yang telah banyak membantu penulisan buku, dan mahasiswa-mahasiswa terbaik saya semester Ganjil 2016/2017, yaitu: Maryamah, Moh. Fadel Asikin, Daisy Kurniawaty, Selly Kurnia Sari, Nanda Agung Putra, Ardisa Tamara Putri, Dhimas Anjar Prabowo, Listiya Surtiningsih, Raissa Arniantya, Brillian Aristyo Rahadian, Diva Kurnianingtyas, Dyan Putri Mahardika, Tusty Nadia Maghfira.
3. Mahasiswa-mahasiswa terbaik saya semester Ganjil 2017/2018, yaitu: Yessica Inggir F., Kholifaul K., Ni Made Gita D. P., Ema Agasta, Retiana Fadma P. Sinaga, Fachrul Rozy Saputra Rangkuti, Yunita Dwi Alfiyanti, Dyah Ayu Wahyuning Dewi, Annisaa Amalia Safitri, Sarah Aditya Darmawan, Danastri Ramya Mehaninda, Eka Novita Shandra, Fakharuddin Farid Irfani, Rio Cahyo Anggono, Robih Dini, Yulia Kurniawati, Novirra Dwi Asri, Muhammad Vidi Mycharoka, Vania Nuraini Latifah, Olivia Bonita, Eka Miyahil Uyun, Cusen Mosabeth, Evilia Nur Harsanti, Ivarianti Sihaloho.

Semoga kontribusi kalian menjadi ilmu yang barokah dan bermanfaat. Aamiin. :). Tidak ada gading yang tak retak. Maka penulis memohon kritik dan saran untuk perbaikan dan penyempurnaan buku ini. In Syaa Allah pada edisi berikutnya, kami akan memberikan manualisasi Map Reduce, Spark, etc. dari setiap algoritma pada setiap contoh kasusnya. Selamat membaca buku ini dan semoga bermanfaat.

Malang, 19 Juli 2016-24 Mei 2018

Penulis

Nowadays, data processing requirements is growing exponentially, and relational database is not always the best solution for all situations in big data such as increasing growth of data. Thus, NoSQL databases emerged to overcome the limitations of relational database and work with big data. NoSQL databases have four types of models, namely, key-value model, document database, column database, and graph database. Many approaches have been proposed to convert relational database to NoSQL models. However, most of them map relational database to key-value or column or document. Converting relational to graph database is slightly disregarded by the researchers. This paper proposes a comprehensive approach, based on Spark framework, for transformation and migration of relational database to graph database without semantic loss. The approach also supports conversions from Sql commands to cypher commands .It is categorized into two parts. The first part is concerned with “transformation and...

VPN access to a distributed system makes it possible to implement a VPN connection connecting a
remote user to a distributed LAN server by protecting the client's identity and the confidentiality of
the processed data. This distributed system provides the functionality of reducing the time of a heavy
data calculation and processing by linking the nodes that execute the distributed application tasks
for each of them. The work was designed by implementing the OpenVPN in the server's network
host machine for secure VPN connection, Apache Mesos for shared resource management and
Apache Spark for running applications. The mode of interaction between a client and the system is
via a web interface implemented in the server.

The Predictive Retailer is a retail company that utilizes the latest technological developments to connect with its customers to deliver an exceptional personalized experience to each and every one of them. Today, technology such as AI, Machine Learning, Augmented Reality, IoT, Real-time stream processing, social media, and wearables are altering the Customer Experience (CX) landscape and retailers need to jump aboard this fast moving technology or run the risk of being left out in the cold.

The Predictive Retailer reveals how these and other technologies can help shape the customer journey. The book details how the five types of analytics—descriptive, diagnostic, predictive, prescriptive, and edge analytics—affect not only the customer journey, but also just about every operating function of the retailer.

An IoT connected retailer can make its operations smart. Connected devices can help with inventory optimization, supply chain management, labor management, waste management, as well as keep its data centers green and its energy use smart.
Social media is no longer a vanity platform, but rather it is a place to both connect with current customers as well as court new ones. It is also a powerful branding channel that can be utilized to both understand a retailer's position in the market, as well as a place to benchmark its position against its competitors.
Today, technology moves at break-neck speed and it can offer the potential of anticipatory capabilities, but it also comes with a confusing variety of technology and technological terms--Big Data, Cognitive Computing, CX, Data Lakes, Hadoop, Kafka, Personalization, Spark, etc., etc. The Predictive Retailer will help make sense of it all, so that a retail executive can cut through the confusing technological jargon and understand why a Spark-based real-time stream processing data stream might be preferable over a TIBCO Streambase one, or an IBM Streaming Analytics one. This book will help retail executives break through the technological clutter so that they can deliver an unrivaled customer experience to each and every patron that comes through their doors.

The sheer increase in the volume of data over the last decade has triggered research in cluster computing frameworks that enable web enterprises to extract big insights from big data. While Apache Spark defines the state of the art in big data analytics platforms for (i) exploiting data-flow and in-memory computing and (ii) for exhibiting superior scale-out performance on the commodity machines, little effort has been devoted to understanding the performance of in-memory data analytics with Spark on modern scale-up servers. This thesis characterizes the performance of in-memory data analytics with Spark on scale-up servers. Through empirical evaluation of representative benchmark workloads on a dual socket server, we have found that in-memory data analytics with Spark exhibit poor multi-core scalability beyond 12 cores due to thread level load imbalance and work-time inflation (the additional CPU time spent by threads in a multi-threaded computation beyond the CPU time required to perform the same work in a sequential computation). We have also found that workloads
are bound by the latency of frequent data accesses to the memory. By
enlarging input data size, application performance degrades significantly due to the substantial increase in wait time during I/O operations and garbage collection, despite 10% better instruction retirement rate (due to lower L1 cache misses and higher core utilization).

For data accesses, we have found that simultaneous multi-threading is
effective in hiding the data latencies. We have also observed that (i) data
locality on NUMA nodes can improve the performance by 10% on average, (ii) disabling next-line L1-D prefetchers can reduce the execution time by upto 14%. For garbage collection impact, we match memory behavior with the garbage collector to improve the performance of applications between 1.6x to 3x and recommend using multiple small Spark executors that can provide up to 36% reduction in execution time over single large executor. Based on the characteristics of workloads, the thesis envisions near-memory and near storage hardware acceleration to improve the single-node performance of scale-out frameworks like Apache Spark. Using modeling techniques, it
estimates the speed-up of 4x for Apache Spark on scale-up servers augmented with near-data accelerators.

The goal of this paper is to know which is the best Big Data architecture that is substantially adapted to make a large amount of data processed in real-time provided by the Spanish Notaries.
Taking as a starting point the Hadoop ecosystem, where there is a stack of Big Data technologies, which they can solve a daily problem. Those technologies can be extrapolated in many data areas that require real-time processing with proper storage in a cluster.
The results obtained in this project are, on the one hand, the creation of large amounts of synthetic data with characteristics similar to the data of the servers of the Notary, in order to test the chosen architecture, and on the other hand, the creation of a fully functional, small-scale virtualized architecture within a virtual cluster, for real-time data capture, with their respective processing and storage in the virtual cluster.

— Big Data is currently a very burning topic in the fields of Computer Science and Business Intelligence, and with such a scenario at our doorstep, a humungous amount of information waits to be documented properly with emphasis on the market. By market, we mean the current technologies in use, the current prevalent tools, and the companies playing an imperative role in taming the data with such a colossal outreach.

In this presentation I discuss about the emerging field of data science answering the fundamental questions of what, why and how. This paper aims to provide a simple overview of data science field, various learning and career opportunities.

These last years, the new technologies produce each day large quantities of data. Companies are faced with certain problems of collecting, storing, analyzing and exploiting these large volumes of data in order to create the added value. The whole issue, for companies and administrations, is not to pass by valuable information drowned in the mass. It is here where the technology of the "Big Data" intervenes. This technology is based on an analysis of very fine masses of data. It is interesting to note that there are several publishers who offer distributions ready to use for managing a system Big Data namely HortonWorks [1], Cloudera [2], MapR [3], IBM Infosphere BigInsights [4], pivotal HD [5], Microsoft HD Insight [6], etc. The different distributions have an approach and a different positioning in relation to the vision of a platform Hadoop. These solutions are the Apache Projects and therefore available. Yet, the interest of a complete package resides in the compatibility between the components, the simplicity of installation, support, etc. In this article, we shall discuss the world of big data by defining these characteristics and its architecture. Then we shall talk about some distributions Hadoop, and finally, we shall conclude by a comparative study on the top five suppliers of Hadoop distributions of Big Data.

Machine Learning (ML), which is a subset of Artificial intelligence (AI), enhances the ability of a computer to learn, from data, without being explicitly programmed end-to-end. As ML and AI learn they acquire the ability to carry out cognitive functions, such as perceiving, learning, reasoning and automatically digging deeper to identify important insights or leading to new discovery. With the advance in machine learning, in particular its Deep Learning (DL) subset, ML is rapidly spreading across sectors and will continue to do so at an even higher rate with the ever increasing growth of Big Data. Gartner predicts that companies will combine Big Data and Machine Learning to carry out some or most of their service processes by 40% in 2022, up from 5% in 2017.

ML is used to accelerate data-driven discovery in research and development. Recently, it has enabled scientists to discover largely unknown diversity of viruses, amounting to thousands of previously unknown viruses. The book refers to previous as well recent research work, with colleagues, where ML was used to capture subtle variation and to discover rare items, such as rare genes which researchers have so long sought for in vain. Such processes to identify genes or medicine can be daunting, as it may take years and can be expensive and the outcome can be uncertain. ML is used today to shorten the time and even help to identify medicine that can be more effective for people with a particular gene, which will help in turn in personalized medicine.

ML is a critical ingredient for intelligent applications and provides the opportunity to further accelerate discovery processes as well as enhancing decision making processes. These trends promise that every sector will be data-driven and will be using machine learning in the cloud to incorporate artificial intelligence applications and to ultimately supplement existing analytical and decision making tools.

The book introduces ML and its potential along with some ML applications using Spark and R platforms combined. While Spark has the possibility to scale and speed up analytics, it harness R language‘s machine learning capabilities beyond what is available on Spark or any other Big Data system. R and Spark can share codes and different types of data and carry out powerful large scale machine learning capabilities. Machine learning with Spark and R language combined can not only speed up but also light up Big Data Discovery.

—In the Big Data group, Apache Hadoop and Spark are gaining prominence in handling Big Data and analytics. Similarly MapReduce has been seen as one of the key empowering methodologies for taking care of large-scale query processing. These middleware are traditionally written with sockets and do not deliver best performance on datacenters with modern high performance networks. In this paper we investigate the characterizes of two file systems that support in-memory and heterogeneous storage, and discusses the impacts of these two architectures on the performance and fault tolerance of Hadoop MapReduce and Spark applications. We present a complete methodology for evaluating MapReduce and Spark workloads on top of in-memory file systems and provide insights about the interactions of different system components while running these workloads.

1. ABSTRACT Current database engines that provide support for JSON data usually offer a non-standardized and semantically incomplete language. In the other hand, relational databases use follow a common language (SQL), but fail to represent heterogeneous and nesting data. In this context, SQL++ is presented as a complete language that it is a superset of SQL and JSON. In this paper, we present a methodology for transforming a SQL relational engine to an engine that can support the SQL++ query language. We explain the new data encodings, expressions, data types and operators needed to achieve this goal. In order to prove our methodology, we present our work in modifying Spark SQL to transform it to Spark SQL++. Moreover, we provide experiments that show that our changes do not create performance loss for existing queries and for queries that involve semi-structured data can be solved with a performance loss proportional to the number of attributes that are unknown. 2. INTRODUCTION Nowadays, there are numerous databases that claim support for unstructured data such as SQL-on-Hadoop, NewSQL, Spark SQL, MongoDB etc, but they only provide this support with certain limitations and in many cases with query languages that are not expressive enough and specific to their database. In the other side of the spectrum, we find relational databases that don't support nesting and heterogeneous data but provide a more extensive, complete and generic query language called SQL. In this setting, SQL++[7] is presented as a semi-structured query language that is backwards compatible with SQL, in order to be easily understood and adopted by SQL programmers. The semi-structured SQL++ data model is a super-set of JSON and the SQL data model. The SQL++ model expands JSON with bags (as opposed to having JSON arrays only) and enriched values, i.e., atomic values that are not only numbers and strings. Vice versa, one may think of SQL++ as expanding SQL with JSON features: arrays, heterogeneity, and the possibility that any value may be an arbitrary composition of the array, bag and tuple con-structors, hence enabling arbitrary nested structures, such as arrays of arrays. Our contribution with this paper is to present a methodology for transforming an existing SQL relational engine into a SQL++ engine that can handle heterogeneous data, array values and array indexing, unstructured and semi-structured data, and it opens the path for the creation of the new operators (correlate, array scan, etc) proposed for the SQL++ query language. Our methodology includes a new encoding for data that does not have a schema, a type for representing heterogeneity, a new schema option for semi-structured data and a new operator to access unstructured data that allows to abstract other operators from handling it, maintaining the rest of the pipeline unchanged. We test the proposed methodology by modifying the Spark SQL module from Apache Spark [1]. Here, we show the particular steps taken in the modification of Spark SQL into Spark SQL++ following the methodology and how can this be achieved on an incremental fashion. Moreover, we present how the caching functionality of Spark SQL can still be used in Spark SQL++. The experiments section compares the performance of Spark SQL++ with the original version of Spark SQL when we use a full-schema supported by Spark SQL. From this experiment , we show a minimal difference in performance. Moreover , we compare the time execution of the same queries when the data has a complete schema, a partial schema and no schema. We consider the cases where the data is on disk or if it is cached used the in-memory caching of Spark SQL. The results shows us a relation between the schema of the data and the performance loss when this schema is not present. This relation is shown by an incremental performance degradation as more parts of the queried elements correspond to the unstructured part of the data. The presence of schema is also important because it allows to introduce more optimization in the query engine such as columnar storage, indexes, and statistics of the data stored. These optimizations are harder to support and maintain considering that they are not specifically detailed on the schema of the data. The structure of this paper is as follows: First, we present a small background about SQL++ in Section 3, we introduce the assumptions of a relational engine and the methodology proposed in Section 4. Section 5 presents the current state of Spark SQL and the changes made to transform it to Spark SQL++. In Section 6, we show experiments comparing the

Oggigiòrno si sente spesso parlare di Machine Learning ed algoritmi in grado di portare valore aggiunto al business delle aziende. Un settore dove tali algoritmi stanno prendendo piede è quello delle telecomunicazioni ma spesso i provider non riescono ad intravedere del potenziale in tali algoritmi. Per far fronte a questa situazione, grazie alla possibilità avuta di lavorare a supporto del team tecnico-funzionale di Net Reply, è stato realizzato uno use case per il settore Telco utilizzando i dati di telefonia fissa di un Provider telco. Attraverso le fasi di Business Understanding e Data Understanding è stato possibile analizzare prima le richieste del cliente e successivamente, è stato possibile individuare la strategia ed i modelli da utilizzare per raggiungere i suoi obiettivi di business in base ai dati a disposizione. Grazie alle fasi di Modelling ed Evaluation è stato analizzato il modello da un punto di vista funzionale, è stato creato, sono stati stimati i parametri, è stato addestrato ed infine ne sono state valutate le performance. Al fine di aumentare tali performance, è stata applicata una strategia che potesse dimostrare che, all’aumentare dei dati a disposizione, è possible aumentare le performance e diminuire il margine di errore. In ultima fase è stata proposta l’implementazione ad alto livello di questi modelli utilizzando i prodotti in cloud di Azure. Seguendo il framework CRISP-DM è stato possibile portare valore aggiunto a Net Reply, al progetto del cliente e sono state presentate alle due società nuove tecnologie che prestodiventerano, se già non lo sono, "standard tecnologici".

Big Data is one of the concepts of computing that has gained notoriety in several business
organizations. The amount of unstructured data is huge, especially on e-commerce sites. In
this work, a study and implementation of environments for Big Data is carried out. The experiments were carried out in the Oracle Cloud environment and processed with the Apache
Spark framework. This framework allows the distributed processing of large data sets through
clusters of computers. This work aims to evaluate the performance of big data environments
implemented in cloud infrastructure. The data were processed in two environments, a Kubernetes cluster and a cluster of virtual machines with 5 workers in each environment. Each node has
the same hardware configuration with an Intel Skylake processor. The size of the processed file
was 10 GB, 20 GB and 40 GB in .txt format. These files were used to evaluate the performance
of the environments through a script for counting the words inside the text files. Through
the experiments it was possible to evaluate the processing time on each environment. It was
found that the Kubernetes cluster has better performance compared to the Standalone environment. Through the observed analysis, the results of the best performing experiments will help
to analyse, develop and standardize a modern Big Data architecture, through optimizations,
automations and high availability.

Nowadays, data processing requirements is growing exponentially, and relational database is not always the best solution for all situations in big data such as increasing growth of data. Thus, NoSQL databases emerged to overcome the limitations of relational database and work with big data. NoSQL databases have four types of models, namely, key-value model, document database, column database, and graph database. Many approaches have been proposed to convert relational database to NoSQL models. However, most of them map relational database to key-value or column or document. Converting relational to graph database is slightly disregarded by the researchers. This paper proposes a comprehensive approach, based on Spark framework, for transformation and migration of relational database to graph database without semantic loss. The approach also supports conversions from Sql commands to cypher commands .It is categorized into two parts. The first part is concerned with "transformation and migration using Spark", which encompasses three phases: Meta data analyzer, transformation algorithm, and migration algorithm. The second part focuses on "SQL to cypher", which divides into two phases: SQL parser and Translator. The suggested approach has been applied, results and validation for the proposed approach

While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data an-alytics for being a unified framework for both, batch and stream data processing. However, recent studies on micro-architectural characterization of in-memory data analytics are limited to only batch processing workloads. We compare micro-architectural performance of batch processing and stream processing workloads in Apache Spark using hardware performance counters on a dual socket server. In our evaluation experiments, we have found that while batch processing workloads are bounded on the latency of frequent data accesses to DRAM, stream processing workloads are curbed by L1 instruction cache misses. For data accesses we have found that simultaneous multi-threading is effective in hiding the data latencies. We have also observed that (i) data locality on NUMA nodes can improve the performance with up to 12%, (ii) disabling next-line L1-D prefetchers can reduce the execution time by up-to 15% and (iii) multiple small executors can provide up-to 36% speedup over single large executor.

Smart grid is a complete automation system,
where large pool of sensors is embedded in the existing
power grids system for controlling and monitoring it by
utilizing modern information technologies. The data
collected from these sensors are huge and have all the
characteristics to be called as Big Data. The Smart-grid can
be made more intelligent by processing and deriving new
information from these data in real time. This paper
presents Apache spark as a unified cluster computing
platform which is suitable for storing and performing Big
Data analytics on smart grid data for applications like
automatic demand response and real time pricing

While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data analyt-ics. Recent studies propose scale-in clusters with in-storage processing devices to process big data analytics with Spark However the proposal is based solely on the memory band-width characterization of in-memory data analytics and also does not shed light on the specification of host CPU and memory. Through empirical evaluation of in-memory data analytics with Apache Spark on an Ivy Bridge dual socket server, we have found that (i) simultaneous multi-threading is effective up to 6 cores (ii) data locality on NUMA nodes can improve the performance by 10% on average, (iii) disabling next-line L1-D prefetchers can reduce the execution time by up to 14%, (iv) DDR3 operating at 1333 MT/s is sufficient and (v) multiple small executors can provide up to 36% speedup over single large executor.

With the recent boom in big data analytics, many application areas require optimization algorithms that work at massive scale. In this note, we consider a distributed method for solving large-scale optimization problems called alternating minimization algorithm (AMA) and its implementation with Apache Spark.

The combination of the two quick creating logical exploration regions Semantic Web and Web Mining is called Semantic Web Mining. The immense increment in the measure of Semantic Web information turned into an ideal objective for some specialists to apply Data-Mining methods on it. Semantic Web Data is an extra for the WorldWide Web, and the primary goal of this is to make the web information machine-comprehensible. Resource Description Framework (RDF) is one of the advancements used to encode and speak to the semantics information as metadata. It's most likely to host the semantic web data on cloud due to its vast requirement, and also it can be managed well in terms of storage and evaluation. Map-reduce is a programming model that is well known for its scalability, flexibility, parallel processing, and cost-effective solution. Hadoop and Spark are the popular open-source tools for handling (Map-Reduce) and storing (HDFS) a huge-amount of data. Semantic web data can be processed using the SPARQL, which is a primary query language for processing the RDF. In terms of performance, SPARQL has a significant drawback comparing to Map-reduce. For Querying the RDF data, we use Spark and Hadoop components (PIG and HIVE). Where Considering Directed Acyclic Graph (DAG) scheduler as a specific feature for In-memory processing in spark. In this paper, evaluate and analyse performance results using RDF data, which contains 5000 triples by executing the benchmark queries in PIG, HIVE, and SPARK. A scalable and faster framework can be obtained based on practical evaluation and analysis.

The sheer increase in the volume of data over the last decade has triggered research in cluster computing frameworks that enable web enterprises to extract big insights from big data. While Apache Spark defines the state of the art in big data analytics platforms for (i) exploiting data-flow and in-memory computing and (ii) for exhibiting superior scale-out performance on the commodity machines, little effort has been devoted at understanding the performance of in-memory data analytics with Spark on modern scale-up servers. This thesis characterizes the performance of in-memory data analytics with Spark on scale-up servers.

Through empirical evaluation of representative benchmark workloads on a dual socket server, we have found that in-memory data analytics with Spark exhibit poor multi-core scalability beyond 12 cores due to thread level load imbalance and work-time inflation. We have also found that workloads are bound by the latency of frequent data accesses to DRAM. By enlarging input data size, application performance degrades significantly due to substantial increase in wait time during I/O operations and garbage collection, despite 10% better instruction retirement rate (due to lower L1 cache misses and higher core utilization).

For data accesses we have found that simultaneous multi-threading is effective in hiding the data latencies. We have also observed that (i) data locality on NUMA nodes can improve the performance by 10% on average, (ii) disabling next-line L1-D prefetchers can reduce the execution time by up-to 14%. For GC impact, we match memory behaviour with the garbage collector to improve performance of applications between 1.6x to 3x. and recommend to use multiple small executors that can provide up-to 36% speedup over single large executor.

Vehicular traffic has become an important research area because of its specific features and applications as road safety and efficient traffic management. Vehicles should be carry enough of communication systems, onboard computing facilities, storage, and increased geographical monitoring. Hence, several technologies have been deployed to promote Vehicular traffic management. Since this work, Ant Colony Optimization (ACO) algorithm that's based on the apache Spark is in parallel to settle the (TSP) Traveling Salesman Problem. To achieve the parallelization of the phase of solution construction in ant colony optimization, a class of ants was encapsulated to a resilient distributed dataset and the corresponding transformation operators were given. The comparison results between MapReduce based ant colony algorithm and the proposed algorithm under the same experimental environment show that the algorithm proposed notably enhance the optimization speed at least ten times than the Ma...

In the process of knowledge discovery and representation in large datasets using formal concept analysis, complexity plays a major role in identifying the formal concepts and constructing the concept lattice (digraph of the concepts). For identifying the formal concepts and constructing the digraph from the identified concepts in large datasets, various distributed algorithms are available. However, the existing distributed algorithms are not well suited for concept generation, because the generation of concepts is an iterative process. Existing algorithms are implemented using distributed frameworks like MapReduce and Open MP. These frameworks are not appropriate for iterative applications. Hence, there is a need for efficient distributed algorithms for both formal concept generation and concept lattice digraph construction in large formal contexts. In this paper, we present efficient algorithms using Apache Spark. The various performance metrics used in evaluation prove that the proposed algorithms are more efficient for concept generation and lattice graph construction than existing algorithms.

—While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data analytics for being a unified framework for both, batch and stream data processing. However, recent studies on micro-architectural characterization of in-memory data analytics are limited to only batch processing workloads. We compare the micro-architectural performance of batch processing and stream processing workloads in Apache Spark using hardware performance counters on a dual socket server. In our evaluation experiments, we have found that batch processing and stream processing has same micro-architectural behavior in Spark if the difference between two implementations is of micro-batching only. If the input data rates are small, stream processing workloads are front-end bound. However, the front end bound stalls are reduced at larger input data rates and instruction retirement is improved. Moreover, Spark workloads using DataFrames have improved instruction retirement over workloads using RDDs.

Smart grid is a complete automation system, where large pool of sensors is embedded in the existing power grids system for controlling and monitoring it by utilizing modern information technologies. The data collected from these sensors are huge and have all the characteristics to be called as Big Data. The Smart-grid can be made more intelligent by processing and deriving new information from these data in real time. This paper presents Apache spark as a unified cluster computing platform which is suitable for storing and performing Big Data analytics on smart grid data for applications like automatic demand response and real time pricing.

Boosted by Big Data popularity, new languages and frameworks for data analytics are appearing at an increasing pace. Each of them introduces its own concepts and terminology and advocates a (real or alleged) superiority in terms of performances or expressiveness against predecessors. In this hype, for a user approaching Big Data analytics (even an educated computer scientist), it might be difficult to have a clear picture of the programming model underneath these tools and the expressiveness they provide to solve some user-defined problem. To provide some order in the world of Big Data processing, a toolkit of models to identify their common features is introduced, starting from data layout. Data-processing applications are divided into batch vs. stream processing. Batch programs process one or more finite datasets to produce a resulting finite output dataset, whereas stream programs process possibly unbounded sequences of data, called streams, doing so in an incremental manner. Operations over streams may also have to respect a total data ordering—for instance, to represent time ordering.

The Big data is the name used ubiquitously now a day in distributed paradigm on the web. As the name point out it is the collection of sets of very large amounts of data in pet bytes, Exabyte etc. related systems as well as the algorithms used to analyze this enormous data. Hadoop technology as a big data processing technology has proven to be the go to solution for processing enormous data sets. MapReduce is a conspicuous solution for computations, which requirement one-pass to complete, but not exact efficient for use cases that need multi-pass for computations and algorithms. The Job output data between every stage has to be stored in the file system before the next stage can begin. Consequently, this method is slow, disk Input/output operations and due to replication. Additionally, Hadoop ecosystem doesn't have every component to ending a big data use case. Suppose we want to do an iterative job, you would have to stitch together a sequence of MapReduce jobs and execute them in sequence. Every this job has high-latency, and each depends upon the completion of the previous stage. Apache Spark is one of the most widely used open source processing engines for big data, with wealthy language-integrated APIs and an extensive range of libraries. Apache Spark is a usual framework for distributed computing that offers high performance for both batch and interactive processing. In this paper, we aimed to demonstrate a close-up view about Apache Spark and its features and working with Spark using Hadoop. We are in a nutshell discussing about the Resilient Distributed Datasets (RDD), RDD operations, features, and limitation. Spark can be used along with MapReduce in the same Hadoop cluster or can be used lonely as a processing framework. In the last comparative analysis between Spark and Hadoop and MapReduce in this paper.

Classification of imbalanced big data has assembled an extensive consideration by many researchers during the last decade. Standard classification methods poorly diagnosis the minority class samples. Several approaches have been introduced for solving the problem of class imbalance in big data to enhance the generalization in classification. However, most of these approaches neglect the effect of border samples on classification performance; the high impact border samples might expose to misclassification. In this paper, a Spark Based Mining Framework (SBMF) is proposed to address the imbalanced data problem. Two main modules are designed for this purpose. The first is the Border Handling Module (BHM) which under samples the low impact majority border instances and oversamples the minority class instances. The second module is the Selective Border Instances sampling (SBI) Module, which enhances the output of the BHM module. The performance of the SBMF framework is evaluated and compared with other recent systems. A number of experiments were performed using moderate and big datasets with different imbalanced ratio. The results obtained from SBMF framework, when compared to the recent works, show better performance for the different datasets and classifiers.

SQL-on-Hadoop systems have been gaining popularity in recent years. One popular example of SQL-on-Hadoop systems is Apache Hive; the pioneer of SQL-on-Hadoop systems. Hive is located on the top of big data stack as an application layer. Besides the application layer, the Hadoop Ecosystem is composed of 3 different main layers: storage, the resource manager and processing engine. The demand from industry has led to the development of new efficient components for each layer. As the ecosystem evolves over time, Hive employed different execution engines too. Understanding the strengths of components is very important in order to exploit the full performance of the Hadoop Ecosystem. Therefore, recent works in the literature study the importance of each layer separately. To the best of our knowledge, the present work is the first work that focuses on the performance of the combination of both the storage layer and the execution engine. In this work, we compare the Hive's query performance by using three different execution engines: MR, Tez and Spark on the skewed/well-balanced data distribution through the full TPC-H benchmark. Our results show the importance of data distribution on the storage layer for overall job performance of SQL-on-Hadoop systems and empirically showed even distribution improves performance up to 48% compared to skewed distribution. Moreover, the present study provides insightful findings by identifying particular SQL query cases that the certain processing engine deals exceptionally well.

Vehicular traffic has become an important research area because of its specific features and applications as road safety and efficient traffic management. Vehicles are expected to carry relatively more communication systems, onboard computing facilities, storage ,and increased geographical monitoring .Hence, several technologies have been deployed to promote Vehicular traffic management. In this paper, a parallel Ant Colony Optimization (ACO) algorithm based on Spark is paral-lelized to solve Traveling Salesman Problem (TSP). To achieve the parallelization of the phase of solution construction in ant colony optimization, a class of ants was encapsulated to a resilient distributed dataset and the corresponding transformation operators were given. The comparison results between MapReduce based ant colony algorithm and the proposed algorithm under the same experimental environment show that the proposed algorithm significantly improves the optimization speed at least ten times than the MapReduce one.

The statistical analysis of infrastructure metrics comes with several specific challenges, including the fairly large volume of unstructured metrics from a large set of independent data sources. Hadoop and Spark provide an ideal environment in particular for the first steps of skimming rapidly through hundreds of TB of low relevance data to find and extract the much smaller data volume that is relevant for statistical analysis and modelling.
This presentation will describe the new Hadoop service at CERN and the use of several of its components for high throughput data aggregation and ad-hoc pattern searches. We will describe the hardware setup used, the service structure with a small set of decoupled clusters and the first experience with co-hosting different applications and performing software upgrades. We will further detail the common infrastructure used for data extraction and preparation from continuous monitoring and database input sources.

Classification of imbalanced big data has assembled an extensive consideration by many researchers during the last decade. Standard classification methods poorly diagnosis the minority class samples. Several approaches have been introduced for solving the problem of class imbalance in big data to enhance the generalization in classification. However, most of these approaches neglect the effect of border samples on classification performance; the high impact border samples might expose to misclassification. In this paper, a Spark Based Mining Framework (SBMF) is proposed to address the imbalanced data problem. Two main modules are designed for this purpose. The first is the Border Handling Module (BHM) which under samples the low impact majority border instances and oversamples the minority class instances. The second module is the Selective Border Instances sampling (SBI) Module …