Sign In Start Free Trial

Learning PySpark

By : Drabas, Lee

3.9 (194)

Learning PySpark

3.9 (194)

By: Drabas, Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Understanding Spark

1. Understanding Spark

What is Apache Spark?

Spark Jobs and APIs

Spark 2.0 architecture

Summary

2. Resilient Distributed Datasets

2. Resilient Distributed Datasets

Internal workings of an RDD

Creating RDDs

Global versus local scope

Transformations

Actions

Summary

3. DataFrames

3. DataFrames

Python to RDD communications

Catalyst Optimizer refresh

Speeding up PySpark with DataFrames

Creating DataFrames

Simple DataFrame queries

Interoperating with RDDs

Querying with the DataFrame API

Querying with SQL

DataFrame scenario – on-time flight performance

Spark Dataset API

Summary

4. Prepare Data for Modeling

4. Prepare Data for Modeling

Checking for duplicates, missing observations, and outliers

Getting familiar with your data

Visualization

Summary

5. Introducing MLlib

5. Introducing MLlib

Overview of the package

Loading and transforming the data

Getting to know your data

Creating the final dataset

Predicting infant survival

Summary

6. Introducing the ML Package

6. Introducing the ML Package

Overview of the package

Predicting the chances of infant survival with ML

Parameter hyper-tuning

Other features of PySpark ML in action

Summary

7. GraphFrames

7. GraphFrames

Introducing GraphFrames

Installing GraphFrames

Preparing your flights dataset

Building the graph

Executing simple queries

Understanding vertex degrees

Determining the top transfer airports

Understanding motifs

Determining airport ranking using PageRank

Determining the most popular non-stop flights

Using Breadth-First Search

Visualizing flights using D3

Summary

8. TensorFrames

8. TensorFrames

What is Deep Learning?

What is TensorFlow?

Introducing TensorFrames

TensorFrames – quick start

Summary

9. Polyglot Persistence with Blaze

9. Polyglot Persistence with Blaze

Installing Blaze

Polyglot persistence

Abstracting data

Data operations

Summary

10. Structured Streaming

10. Structured Streaming

What is Spark Streaming?

Why do we need Spark Streaming?

What is the Spark Streaming application data flow?

Simple streaming application using DStreams

A quick primer on global aggregations

Introducing Structured Streaming

Summary

11. Packaging Spark Applications

11. Packaging Spark Applications

The spark-submit command

Deploying the app programmatically

Databricks Jobs

Summary

Index

Index

Customer Reviews

3.9 (194)

5 star

39.2%

4 star

32%

3 star

13.9%

2 star

7.2%

1 star

7.7%

What this book covers

Chapter 1, Understanding Spark, provides an introduction into the Spark world with an overview of the technology and the jobs organization concepts.

Chapter 2, Resilient Distributed Datasets, covers RDDs, the fundamental, schema-less data structure available in PySpark.

Chapter 3, DataFrames, provides a detailed overview of a data structure that bridges the gap between Scala and Python in terms of efficiency.

Chapter 4, Prepare Data for Modeling, guides the reader through the process of cleaning up and transforming data in the Spark environment.

Chapter 5, Introducing MLlib, introduces the machine learning library that works on RDDs and reviews the most useful machine learning models.

Chapter 6, Introducing the ML Package, covers the current mainstream machine learning library and provides an overview of all the models currently available.

Chapter 7, GraphFrames, will guide you through the new structure that makes solving problems with graphs easy.

Chapter 8, TensorFrames, introduces the bridge between Spark and the Deep Learning world of TensorFlow.

Chapter 9, Polyglot Persistence with Blaze, describes how Blaze can be paired with Spark for even easier abstraction of data from various sources.

Chapter 10, Structured Streaming, provides an overview of streaming tools available in PySpark.

Chapter 11, Packaging Spark Applications, will guide you through the steps of modularizing your code and submitting it for execution to Spark through command-line interface.

For more information, we have provided two bonus chapters as follows:

Installing Spark: https://www.packtpub.com/sites/default/files/downloads/InstallingSpark.pdf

Free Spark Cloud Offering: https://www.packtpub.com/sites/default/files/downloads/FreeSparkCloudOffering.pdf

Search

Your notes and bookmarks