Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Learning PySpark
  • Table Of Contents Toc
  • Feedback & Rating feedback
Learning PySpark

Learning PySpark

By : Drabas, Lee
3.9 (194)
Buy this Book
close
close
Learning PySpark

Learning PySpark

3.9 (194)
By: Drabas, Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.
Table of Contents (13 chapters)
close
close
close
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
1. Understanding Spark
1. Understanding Spark
What is Apache Spark?
Spark Jobs and APIs
Spark 2.0 architecture
Summary
2
2. Resilient Distributed Datasets
2. Resilient Distributed Datasets
Internal workings of an RDD
Creating RDDs
Global versus local scope
Transformations
Actions
Summary
3
3. DataFrames
3. DataFrames
Python to RDD communications
Catalyst Optimizer refresh
Speeding up PySpark with DataFrames
Creating DataFrames
Simple DataFrame queries
Interoperating with RDDs
Querying with the DataFrame API
Querying with SQL
DataFrame scenario – on-time flight performance
Spark Dataset API
Summary
4
4. Prepare Data for Modeling
4. Prepare Data for Modeling
Checking for duplicates, missing observations, and outliers
Getting familiar with your data
Visualization
Summary
5
5. Introducing MLlib
5. Introducing MLlib
Overview of the package
Loading and transforming the data
Getting to know your data
Creating the final dataset
Predicting infant survival
Summary
6
6. Introducing the ML Package
6. Introducing the ML Package
Overview of the package
Predicting the chances of infant survival with ML
Parameter hyper-tuning
Other features of PySpark ML in action
Summary
7
7. GraphFrames
7. GraphFrames
Introducing GraphFrames
Installing GraphFrames
Preparing your flights dataset
Building the graph
Executing simple queries
Understanding vertex degrees
Determining the top transfer airports
Understanding motifs
Determining airport ranking using PageRank
Determining the most popular non-stop flights
Using Breadth-First Search
Visualizing flights using D3
Summary
8
8. TensorFrames
8. TensorFrames
What is Deep Learning?
What is TensorFlow?
Introducing TensorFrames
TensorFrames – quick start
Summary
9
9. Polyglot Persistence with Blaze
9. Polyglot Persistence with Blaze
Installing Blaze
Polyglot persistence
Abstracting data
Data operations
Summary
10
10. Structured Streaming
chevron up
10. Structured Streaming
What is Spark Streaming?
Why do we need Spark Streaming?
What is the Spark Streaming application data flow?
Simple streaming application using DStreams
A quick primer on global aggregations
Introducing Structured Streaming
Summary
11
11. Packaging Spark Applications
11. Packaging Spark Applications
The spark-submit command
Deploying the app programmatically
Databricks Jobs
Summary
12
Index
Index

Simple streaming application using DStreams

Let's create a simple word count example using Spark Streaming in Python. For this example, we will be working with DStream – the Discretized Stream of small batches that make up the stream of data. The example used for this section of the book can be found in its entirety at: https://github.com/drabastomek/learningPySpark/blob/master/Chapter10/streaming_word_count.py.

This word count example will use the Linux / Unix nc command – it is a simple utility that reads and writes data across network connections. We will use two different bash terminals, one using the nc command to send words to our computer's local port (9999) and one terminal that will run Spark Streaming to receive those words and count them. The initial set of commands for our script are noted here:

1. # Create a local SparkContext and Streaming Contexts
2. from pyspark import SparkContext
3. from pyspark.streaming import StreamingContext
4. 
5. # Create sc...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 5
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY