Name: Apache Spark
Rating: 8.9 (161 reviews)
Author: Apache

Help Desk

Live Chat

Web and Video Conferencing

Customer Support

K-12 Online Learning

Education

Business Intelligence (BI)

Collaboration

Contract Management

Enterprise Legal Management

Environmental Health and Safety (EHS)

Enterprise

Accounting

Investor Relations

Finance and Accounting

Applicant Tracking

Corporate Learning Management

HR Management

Payroll

Talent Intelligence

Talent Management

Workforce Analytics

Workforce Management

Human Resources

Cloud Storage

Consulting and Advisory Services

Converged Network Adapters (CNAs)

Custom Software Development Services

Database Load Balancing

Edge Computing

Fraud Detection

Integration Platform as a Service (iPaaS)

Managed File Transfer (MFT)

Master Data Management (MDM)

Migration

Network Performance Monitoring

NoSQL Databases

Observability

Privileged Access Management

Release Management

Server Management

Software Composition Analysis (SCA)

Tape Storage

Threat Intelligence

Information Technology

A/B Testing

Ad Serving & Retargeting

Affiliate

All-in-One Marketing

App Store Optimization (ASO)

Contact Management

Content Management

Contest Management

Digital Asset Management

Digital Experience Intelligence

Email Marketing

Gamification

Marketing Analytics

Marketing Automation

Mobile Advertising

Product Intelligence

Social Media Management

Social Media Marketing

Survey & Forms Building

Tag Management

Web Analytics

Marketing

Project Management

Professional Services

Call Recording

Customer Relationship Management (CRM)

Presales

Product Configuration

Quote to Cash

Sales

Cloud Infrastructure Entitlement Management

GDPR Compliance

Security

Construction Workforce Management

Dental

Loan Servicing

Mobility as a Service

Nonprofit Fundraising

Oil and Gas

Vertical-Specific

Find top rated software and services based on in-depth reviews from verified users. 400+ software categories including PaaS, NoSQL, BI, HR, and more.

Apache Hadoop

Apache

Apache Hive

Elasticsearch

Elastic

Google BigQuery

Google

Snowflake

Snowflake Computing

Presto

Open Source

MongoDB

Scala

Hive

Hive Technology

TensorFlow

### Data Processing
Reviewers generally praise Apache Spark for its robust data processing capabilities. Users appreciate the platform's ability to handle large volumes of data efficiently, both in real-time and batch processing scenarios. The consensus highlights Apache Spark's speed and scalability in processing complex datasets, enabling organizations to derive valuable insights and streamline their data processing workflows. While some users have noted minor challenges such as memory consumption issues and the need for manual caching setup, overall, Apache Spark is recognized for its strong performance in data processing tasks. (Source Reviews: [1](https://www.trustradius.com/reviews/apache-spark-2022-08-25-11-35-53), [2](https://www.trustradius.com/reviews/apache-spark-2023-08-15-06-20-45), [3](https://www.trustradius.com/reviews/apache-spark-2022-04-14-14-31-57), [4](https://www.trustradius.com/reviews/apache-spark-2021-04-28-12-40-10), [5](https://www.trustradius.com/reviews/apache-spark-2021-05-16-21-59-51), [6](https://www.trustradius.com/reviews/apache-spark-2021-04-29-12-04-37), [7](https://www.trustradius.com/reviews/apache-spark-2020-09-10-14-25-32), [8](https://www.trustradius.com/reviews/apache-spark-2019-12-12-19-52-46), [9](https://www.trustradius.com/reviews/apache-spark-2020-01-09-04-50-15))

### Analytics and Advanced Analytics
Reviewers have expressed varying opinions regarding Apache Spark's analytics and advanced analytics capabilities. While some users appreciate the ease of processing large datasets and the scalability offered by Spark for analytics tasks, others have highlighted limitations such as memory consumption and lack of support for advanced analytics features. The consensus seems to be that Apache Spark is well-suited for handling large-scale data processing and basic analytics tasks, but may not be the ideal choice for users seeking more advanced analytics functionalities. (Source Reviews: [1](https://www.trustradius.com/reviews/apache-spark-2023-08-15-06-20-45), [2](https://www.trustradius.com/reviews/apache-spark-2021-04-28-12-40-10), [3](https://www.trustradius.com/reviews/apache-spark-2020-09-10-14-25-32), [4](https://www.trustradius.com/reviews/apache-spark-2019-12-12-19-52-46), [5](https://www.trustradius.com/reviews/apache-spark-2020-01-09-04-50-15))

### ETL (Extract, Transform, Load
Opinions on Apache Spark's ETL capabilities vary among users. While some users praise its efficiency in handling large datasets, fault tolerance, and parallel processing for ETL tasks, others find the learning curve steep due to the multitude of actions and transformations involved. Despite the challenges, many users appreciate Spark's wide range of uses for ETL processing, its ease of integration, and multi-language support, making it a robust solution for data science applications. The lightning-fast processing times offered by Apache Spark have been particularly beneficial for users engaged in ETL tasks. (Source Reviews: [1](https://www.trustradius.com/reviews/apache-spark-2020-10-30-13-27-47), [2](https://www.trustradius.com/reviews/apache-spark-2020-09-10-14-25-32), [3](https://www.trustradius.com/reviews/apache-spark-2019-12-12-19-52-46))

### Machine Learning
Reviewers have expressed mixed opinions regarding Apache Spark's machine learning capabilities. While some users appreciate the built-in MLlib library for its ease of use and scalability, others have encountered challenges with memory issues when dealing with large datasets. Additionally, there are concerns about the lack of support for advanced analytics and deep-learning frameworks within the MLlib library. Despite these drawbacks, Apache Spark remains a popular choice for machine learning projects due to its flexibility in handling various data sources and fault-tolerant systems. (Source Reviews: [1](https://www.trustradius.com/reviews/apache-spark-2023-08-15-06-20-45), [2](https://www.trustradius.com/reviews/apache-spark-2022-08-25-11-35-53), [3](https://www.trustradius.com/reviews/apache-spark-2022-04-14-14-31-57), [4](https://www.trustradius.com/reviews/apache-spark-2021-04-28-12-40-10), [5](https://www.trustradius.com/reviews/apache-spark-2020-09-10-14-25-32), [6](https://www.trustradius.com/reviews/apache-spark-2020-01-09-04-50-15), [7](https://www.trustradius.com/reviews/apache-spark-2020-10-30-13-27-47), [8](https://www.trustradius.com/reviews/apache-spark-2019-12-12-19-52-46))

### Big Data Handling
Users consistently highlight Apache Spark's robust capabilities in efficiently managing and processing large datasets, enabling streamlined operations for various data-intensive tasks. While some users express concerns about real-time data processing and automatic optimization, the overall sentiment leans towards acknowledging Apache Spark as a reliable solution for handling big data effectively. (Source Reviews: [1](https://www.trustradius.com/reviews/apache-spark-2022-04-14-14-31-57), [2](https://www.trustradius.com/reviews/apache-spark-2021-04-28-12-40-10), [3](https://www.trustradius.com/reviews/apache-spark-2020-10-30-13-27-47), [4](https://www.trustradius.com/reviews/apache-spark-2020-09-10-14-25-32), [5](https://www.trustradius.com/reviews/apache-spark-2020-01-09-04-50-15))

When using Spark for big data tasks, users commonly recommend familiarizing yourself with the documentation and gaining experience. They emphasize investing time in reading and understanding the documentation to overcome any initial challenges. As users gain experience, they find working with Spark becomes easier and more efficient.

Users also suggest utilizing Spark specifically for true big data problems, where its capabilities and performance shine. They highlight that Spark is well-suited for tackling large-scale data processing tasks.

Additionally, users find value in leveraging the R and Python APIs in Spark. These APIs allow them to work with Spark using familiar programming languages such as R and Python, making it easier to analyze and process data.

Overall, users advise diving into the documentation, utilizing Spark for big data challenges, and leveraging the R and Python APIs to enhance their experience with Spark.

Apache Spark is an incredibly versatile tool that has been widely adopted across various departments for processing very large datasets and generating summary statistics. Users have found it particularly useful for creating simple graphics when working with big data, making it a valuable asset for analytics departments. It is also used extensively in the banking industry to calculate risk-weighted assets on a daily and monthly basis for different positions. The integration of Apache Spark with Scala and Apache Spark clusters enables users to load and process large volumes of data, implementing complex formulas and algorithms. Additionally, Apache Spark is often utilized alongside Kafka and Spark Streams to extract data from Kafka queues into HDFS environments, allowing for streamlined data analysis and processing.

One of the key strengths of Apache Spark lies in its ability to handle large volumes of retail and eCommerce data, providing cost and performance benefits over traditional RDBMS solutions. This makes it a preferred choice for companies in these industries. Furthermore, Apache Spark plays a crucial role in supporting data-driven decision-making by digital data teams. Its capabilities allow these teams to build data products, source data from different systems, process and transform it, and store it in data lakes.

Apache Spark is highly regarded for its ability to perform data cleansing and transformation before inserting it into the final target layer in data warehouses. This makes it a vital tool for ensuring the accuracy and reliability of data. Its faster data processing capabilities compared to Hadoop MapReduce have made Apache Spark a go-to choice for tasks such as machine learning, analytics, batch processing, data ingestion, and report development. Moreover, educational institutions rely on Apache Spark to optimize scheduling by assigning classrooms based on student course enrollment and professor schedules.

Overall, Apache Spark proves itself as an indispensable product that meets the needs of various industries by offering efficient distributed data processing, advanced analytics capabilities, and seamless integration with other technologies. Its versatility allows it to support a wide range of use cases, making it an essential tool for anyone working with big data.

Challenging to Understand and Use: Some users have found Apache Spark to be challenging to understand and use for modeling big data. They struggle with the complexity of the software, leading to a high learning curve.

Lack of User-Friendliness: The software is considered not user-friendly, with a confusing user interface and graphics that are not of high quality. This has resulted in frustration among some users who find it difficult to navigate and work with.

Time-Consuming Processing: Apache Spark can be time-consuming when processing large data sets across multiple nodes. This has been reported by several users who have experienced delays in their data processing tasks, affecting overall efficiency.

Great Computing Engine: Apache Spark is praised by many users for its capabilities in handling complex transformative logic and sophisticated data processing tasks. Several reviewers have mentioned that it is a great computing engine, indicating its effectiveness in solving intricate problems.

Valuable Insights and Analysis: Many reviewers find Apache Spark to be useful for understanding data and performing data analytical work. They appreciate the valuable insights and analysis capabilities provided by the software, suggesting that it helps them gain deeper understanding of their data.

Extensive Set of Libraries and APIs: The extensive set of libraries and APIs offered by Apache Spark has been highly appreciated by users. It provides a wide range of tools and functionalities to solve various day-to-day problems, making it a versatile choice for different data processing needs.

Apache Spark

Hadoop-Related

Apache Spark Tutorial | Spark Tutorial for Beginners | Spark Big Data | Intellipaat

Introduction to Databricks [New demo linked in description]

Apache Spark Architecture | Spark Cluster Architecture Explained | Spark Training | Edureka

Apache Spark Full Course | Apache Spark Tutorial For Beginners | Learn Spark In 7 Hours |Simplilearn

Spark SQL Tutorial | Spark SQL Using Scala | Apache Spark Tutorial For Beginners | Simplilearn

Spark Project | Spark Tutorial | Online Spark Training | Intellipaat

Consumer Services

Likelihood to Recommend

Likelihood to Renew

Usability

Support Rating

The Apache Software Foundation provides support for the Apache community of open-source software projects. The Apache projects are characterized by a collaborative, consensus based development process, an open and pragmatic software license, and a desire to create high quality software that leads the way in its field.

We consider ourselves not simply a group of projects sharing a server, but rather a community of developers and users.

Amazon EMR (Elastic MapReduce)

Amazon S3 (Simple Storage Service)

Apache Spark 2024-12-09 05:23:03

IntelliJ IDEA

Python IDLE

ChatGPT

Assistant Professor

Apache Spark 2023-08-15 06:20:45

Tableau Desktop

Tableau Server

SAP HANA Cloud

Apache Kafka

Apache Airflow

Apache Ignite

Staff Engineer

Apache Spark 2022-08-25 11:35:53

Senior Software Developer (Consultant)

Apache Spark 2022-04-14 14:31:57

IBM InfoSphere Information Server

Informatica PowerCenter

Databricks Data Intelligence Platform

IBM DataStage

Azure Data Factory

Cloudera Distribution Hadoop (CDH)

Senior Data Engineer

Apache Spark 2021-04-29 12:04:37

TrustRadius is a technology and business research firm based in Austin, Texas. The company is known as a review platform for verified B2B technology and software reviews through a proprietary algorithm and human verification.

TrustRadius

Data visualization

Learning curve

Amounts of data

Stores data

Store information

User friendly

Management system

Resource heavy

Need to manually

Well documented

Machine learning

Data sets

Easy to use

Real time

Large data

Multiple languages

Learning algorithms

Fault tolerance

Machine learning algorithms

Data analysis

Home

Hadoop-Related Software

We use Apache Spark on a daily basis as the main computation engine for updating most critical and non-critical data pipelines. We mostly work with batch processing but there are instances for using Spark Streaming as well. The scope is for all analysis pipelines, machine learning datasets and several operational use cases.

Parallel processing
Configurability
Usage with other tools

More ready-to-use solutions for tweaking the Apache Spark configs
Reduce the creation of UDFs for Pyspark by implementing transformations directly

Increased data literacy and adherence to best data engineering practices across the organization
Increased ability for the data analysts to quickly and reliably have access to their data, better supporting data driven decisions
Decreased costs due to better parallelization of resources

dbt, Amazon S3 (Simple Storage Service), Amazon EMR (Elastic MapReduce)

If you are working on large and big scale data with analytics - don't go further without the use of Apache Spark! One of the projects that I was involved in using Apache Spark was a Recommendation Systems based project. My area or domain of research expertise is also Recommendation Systems. The deployment of a RecSys along with the use of Apache Spark - functionalities like scalability, flexibility of using various data sources along with fault-tolerant systems - are very easy. The built-in machine learning library MLlib is a boon to work. We don't require any other libraries.

Fault-tolerant systems: in most cases, no node fails. If it fails - the processing still continues.
Scalable to any extent.
Has built-in machine learning library called - MLlib
Very flexible - data from various data sources can be used. Usage with HDFS is very easy

Its fully not backward compatible.
It is memory-consuming for heavy and large workloads and datasets
Support for advanced analytics is not available - MLlib has minimalistic analytics.
Deployment is a complex task for beginners.

Scalability
We had data across multiple sources. Integration with those data source types was not a problem
Generation of recommendations was achievable easily

We used Apache Spark for one of the research projects. The ROI though cannot be measured here - but the research paper got accepted to a good conference. What else would a project require??!!

Usage of libraries
Usage of HDFS in particular
Basic analysis of data is possible

Understanding internals of the product
changing data sources - was kinda complex
Integration of other ML libraries is not so user friendly

Earlier we were using RDBMS like Oracle for retail and eCommerce data. We faced challenges such as cost, performance, and a huge amount of transactions coming in. After a lot of critical issues we migrated to delta lake. Now, we are using Apache Spark Streaming to deal with all real-time transactions. For batch data as well, we are pretty much handling TBs of data using Apache Spark.

Realtime data processing
Interactive Analysis of data
Trigger Event Detection

Machine Learning
GraphX Lib
True Realtime Streaming

Fast Processing
In-Memory Computing
Provides better insights

No investment as it is open source
Cheap commodity hardwares can save lot of money

Apache Hadoop, SAP HANA Cloud and Apache Ignite

SAP HANA Cloud, Apache Hive, Apache Airflow, Apache Kafka, Tableau Server, Tableau Desktop

We need to calculate risk-weighted assets (RWA) daily and monthly for different positions the bank holds on a T+1 basis. The volume of calculations is large: more than millions of records per day with very complicated formulas and algorithms. In our applications/projects, we used Scala and Apache Spark clusters to load all data we needed for calculation and implemented complicated formulas and algorithms via its DataFrame or DataSet from the Apache Spark platform.

Without adopting the Apache Spark cluster, it would be pretty hard for us to implement such a big system to handle a large volume of data calculations daily. After this system was successfully deployed into PROD, we've been able to provide capital risk control reports to regulation/compliance controllers in different regions in this global financial world.

DataFrame as a distributed collection of data: easy for developers to implement algorithms and formulas.
Calculation in-memory.
Cluster to distribute large data of calculation.

It would be great if Apache Spark could provide a native database to manage all file info of saved parquet.

The speed of processing a large volume of data.
Dataframe with SQL-like operations reduces the learning curve for new developers if they do have very good knowledge of databases and SQL.
Cluster to scale up/down easily.

With the daily risk reports being calculated via Apache Spark, the bank is able to comply with the FHC rule in the US and other regions and control capitals much better with counterparties.

Large volume of data computing.
Speed of computing.

Extend it to other computing in different departments.