Sign In Start Free Trial

Learning PySpark

By : Drabas, Lee

3.9 (194)

Learning PySpark

3.9 (194)

By: Drabas, Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Understanding Spark

1. Understanding Spark

What is Apache Spark?

Spark Jobs and APIs

Spark 2.0 architecture

Summary

2. Resilient Distributed Datasets

2. Resilient Distributed Datasets

Internal workings of an RDD

Creating RDDs

Global versus local scope

Transformations

Actions

Summary

3. DataFrames

3. DataFrames

Python to RDD communications

Catalyst Optimizer refresh

Speeding up PySpark with DataFrames

Creating DataFrames

Simple DataFrame queries

Interoperating with RDDs

Querying with the DataFrame API

Querying with SQL

DataFrame scenario – on-time flight performance

Spark Dataset API

Summary

4. Prepare Data for Modeling

4. Prepare Data for Modeling

Checking for duplicates, missing observations, and outliers

Getting familiar with your data

Visualization

Summary

5. Introducing MLlib

5. Introducing MLlib

Overview of the package

Loading and transforming the data

Getting to know your data

Creating the final dataset

Predicting infant survival

Summary

6. Introducing the ML Package

6. Introducing the ML Package

Overview of the package

Predicting the chances of infant survival with ML

Parameter hyper-tuning

Other features of PySpark ML in action

Summary

7. GraphFrames

7. GraphFrames

Introducing GraphFrames

Installing GraphFrames

Preparing your flights dataset

Building the graph

Executing simple queries

Understanding vertex degrees

Determining the top transfer airports

Understanding motifs

Determining airport ranking using PageRank

Determining the most popular non-stop flights

Using Breadth-First Search

Visualizing flights using D3

Summary

8. TensorFrames

8. TensorFrames

What is Deep Learning?

What is TensorFlow?

Introducing TensorFrames

TensorFrames – quick start

Summary

9. Polyglot Persistence with Blaze

9. Polyglot Persistence with Blaze

Installing Blaze

Polyglot persistence

Abstracting data

Data operations

Summary

10. Structured Streaming

10. Structured Streaming

What is Spark Streaming?

Why do we need Spark Streaming?

What is the Spark Streaming application data flow?

Simple streaming application using DStreams

A quick primer on global aggregations

Introducing Structured Streaming

Summary

11. Packaging Spark Applications

11. Packaging Spark Applications

The spark-submit command

Deploying the app programmatically

Databricks Jobs

Summary

Index

Index

Customer Reviews

3.9 (194)

5 star

39.2%

4 star

32%

3 star

13.9%

2 star

7.2%

1 star

7.7%

Building the graph

Now that we've imported our data, let's build our graph. To do this, we're going to build the structure for our vertices and edges. At the time of writing, GraphFrames requires a specific naming convention for vertices and edges:

The column representing the vertices needs to have the name ofid. In our case, the vertices of our flight data are the airports. Therefore, we will need to rename the IATA airport code to id in our airports DataFrame.
The columns representing the edges need to have a source (src) and destination (dst). For our flight data, the edges are the flights, therefore the src and dst are the origin and destination columns from the departureDelays_geo DataFrame.

To simplify the edges for our graph, we will create the tripEdges DataFrame with a subset of the columns available within the departureDelays_Geo DataFrame. As well, we created a tripVertices DataFrame that simply renames the IATA column to id to match the GraphFrame naming convention:

# Note, ensure...

Search

Your notes and bookmarks