Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Learning Apache Spark 2
  • Toc
  • feedback
Learning Apache Spark 2

Learning Apache Spark 2

By : Abbasi
3.8 (6)
close
Learning Apache Spark 2

Learning Apache Spark 2

3.8 (6)
By: Abbasi

Overview of this book

Apache Spark has seen an unprecedented growth in terms of its adoption over the last few years, mainly because of its speed, diversity and real-time data processing capabilities. It has quickly become the preferred choice of tool for many Big Data professionals looking to find quick insights from large chunks of data. This book introduces you to the Apache Spark framework, and familiarizes you with all the latest features and capabilities introduced in Spark 2. Starting with a detailed introduction to Spark’s architecture and the installation procedure, this book covers everything you need to know about the Spark framework in the most practical manner. You will learn how to perform the basic ETL activities using Spark, and work with different components of Spark such as Spark SQL, as well as the Dataset and DataFrame APIs for manipulating your data. Then, you will perform machine learning using Spark MLlib, as well as perform streaming analytics and graph processing using the Spark Streaming and GraphX modules respectively. The book also gives special emphasis on deploying your Spark models, and how they can be operated in a clustered mode. During the course of the book, you will come across implementations of different real-world use-cases and examples, giving you the hands-on knowledge you need to use Apache Spark in the best possible manner.
Table of Contents (12 chapters)
close
close
Preface
Preface
The Past
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
1. Architecture and Installation
chevron up
1. Architecture and Installation
Apache Spark architecture overview
Installing Apache Spark
Writing your first Spark program
Spark architecture
Apache Spark cluster manager types
Running Spark examples
Brain teasers
References
Summary
2
2. Transformations and Actions with Spark RDDs
2. Transformations and Actions with Spark RDDs
What is an RDD?
Operations on RDD
Passing functions to Spark (Scala)
Passing functions to Spark (Java)
Passing functions to Spark (Python)
Transformations
Set operations in Spark
Actions
PairRDDs
Shared variables
References
Summary
3
3. ETL with Spark
3. ETL with Spark
What is ETL?
How is Spark being used?
Commonly Supported File Formats
Commonly supported file systems
Structured Data sources and Databases
References
Summary
4
4. Spark SQL
4. Spark SQL
What is Spark SQL?
What is DataFrame API?
What is DataSet API?
What's new in Spark 2.0?
The Sparksession
Creating a DataFrame
Parquet files
Working with Hive
SparkSQL CLI
References
Summary
5
5. Spark Streaming
5. Spark Streaming
What is Spark Streaming?
Steps involved in a streaming app
Architecture of Spark Streaming
Caching and persistence
Checkpointing
DStream best practices
Fault tolerance
What is Structured Streaming?
References
Summary
6
6. Machine Learning with Spark
6. Machine Learning with Spark
What is machine learning?
Why machine learning?
Types of machine learning
Introduction to Spark MLLib
Why do we need the Pipeline API?
How does it work?
Feature engineering
Classification and regression
Clustering
Collaborative filtering
ML-tuning - model selection and hyperparameter tuning
References
Summary
7
7. GraphX
7. GraphX
Graphs in everyday life
What is a graph?
Why are Graphs elegant?
What is GraphX?
Creating your first Graph (RDD API)
Basic graph operators (RDD API)
Caching and uncaching of graphs
Graph algorithms in GraphX
GraphFrames
Comparison between GraphFrames and GraphX
References
Summary
8
8. Operating in Clustered Mode
8. Operating in Clustered Mode
Clusters, nodes and daemons
Running Spark in standalone mode
Using the Cluster Launch Scripts to Start a Standalone Cluster
Running Spark in YARN
Running Spark in Mesos
References:
Summary
9
9. Building a Recommendation System
9. Building a Recommendation System
What is a recommendation system?
User specific recommendations
Key issues with recommendation systems
Recommendation system in Spark
References
Summary
10
10. Customer Churn Prediction
10. Customer Churn Prediction
Overview of customer churn
Why is predicting customer churn important?
How do we predict customer churn with Spark?
Exploring customer service calls
References
Summary
11
Theres More with Spark
Theres More with Spark
Performance tuning
I/O tuning
Sizing up your executors
The skew problem
Security configuration in Spark
Setting up Jupyter Notebook with Spark
Shared variables
References
Summary

Apache Spark cluster manager types

As discussed previously, Apache Spark currently supports three Cluster managers:

  • Standalone cluster manager
  • ApacheMesos
  • Hadoop YARN

We'll look at setting these up in much more detail in Chapter 8, Operating in Clustered Mode, which talks about the operation in a clustered mode.

Building standalone applications with Apache Spark

Until now we have used Spark for exploratory analysis, using Scala and Python shells. Spark can also be used in standalone applications that can run in Java, Scala, Python, or R. As we saw earlier, Spark shell and PySpark provide you with a SparkContext. However, when you are using an application, you need to initialize your own SparkContext. Once you have a SparkContext reference, the remaining API remains exactly the same as for interactive query analysis. After all, it's the same object, just a different context in which it is running.

The exact method of using Spark with your application differs based on your preference of language. All Spark artifacts are hosted in Maven central. You can add a Maven dependency with the following coordinates:

groupId: org.apache.spark 
artifactId: spark_core_2.10 
version: 1.6.1

You can use Maven to build the project or alternatively use Scale/Eclipse IDE to add a Maven dependency to your project.

Note

Apache Maven is a build automation tool used primarily for Java projects. The word maven means "accumulator of knowledge" in Yiddish. Maven addresses the two core aspects of building software: first, it describes how the software is built and second, it describes its dependencies.

You can configure your IDE's to work with Spark. While many of the Spark developers use SBT or Maven on the command line, the most common IDE being used is IntelliJ IDEA. Community edition is free, and after that you can install JetBrains Scala Plugin. You can find detailed instructions on setting up either IntelliJIDEA or Eclipse to build Spark at http://bit.ly/28RDPFy.

Submitting applications

The spark submit script in Spark's bin directory, being the most commonly used method to submit Spark applications to a Spark cluster, can be used to launch applications on all supported cluster types. You will need to package all dependent projects with your application to enable Spark to distribute that across the cluster. You would need to create an assembly JAR (aka uber/fat JAR) file containing all of your code and relevant dependencies.

A spark application with its dependencies can be launched using the bin/spark-submit script. This script takes care of setting up the classpath and its dependencies, and it supports all the cluster-managers and deploy modes supported by Spark.

Submitting applications

Figure 1.16: Spark submission template

For Python applications:

  • Instead of <application-jar>, simply pass in your .py file.
  • Add Python .zip, .egg, and .py files to the search path with - .py files.

Deployment strategies

  • Client mode: This is commonly used when your application is located near to your cluster. In this mode, the driver application is launched as a part of the spark-submit process, which acts as a client to the cluster. The input and output of the application is passed on to the console. The mode is suitable when your gateway machine is physically collocated with your worker machines, and is used in applications such as Spark shell that involve REPL. This is the default mode.
  • Cluster mode: This is useful when your application is submitted from a machine far from the worker machines (for example, locally on your laptop) to minimize network latency between the drivers and the executors. Currently only YARN supports cluster mode for Python applications. The following table shows that the combination of cluster managers, deployment managers, and usage that are not supported in Spark 2.0.0:

Cluster Manager

Deployment Mode

Application Type

Support

MESOS

Cluster

R

Not Supported

Standalone

Cluster

Python

Not Supported

Standalone

Cluster

R

Not Supported

Local

Cluster

-

Incompatible

-

Cluster

Spark-Shell

Not applicable

-

Cluster

Sql-Shell

Not Applicable

-

Cluster

Thrift Server

Not Applicable

End of Section 6
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete