Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Distributed Computing with Python
  • Table Of Contents Toc
  • Feedback & Rating feedback
Distributed Computing with Python

Distributed Computing with Python

By : Pierfederici
4.3 (3)
Buy this Book
close
close
Distributed Computing with Python

Distributed Computing with Python

4.3 (3)
By: Pierfederici
Buy this Book

Overview of this book

CPU-intensive data processing tasks have become crucial considering the complexity of the various big data applications that are used today. Reducing the CPU utilization per process is very important to improve the overall speed of applications. This book will teach you how to perform parallel execution of computations by distributing them across multiple processors in a single machine, thus improving the overall performance of a big data processing task. We will cover synchronous and asynchronous models, shared memory and file systems, communication between various processes, synchronization, and more.
Table of Contents (10 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Free Chapter
1
1. An Introduction to Parallel and Distributed Computing
Icon
1. An Introduction to Parallel and Distributed Computing
Icon
Parallel computing
Icon
Distributed computing
Icon
Shared memory versus distributed memory
Icon
Amdahl's law
Icon
The mixed paradigm
Icon
Summary
2
2. Asynchronous Programming
Icon
2. Asynchronous Programming
Icon
Coroutines
Icon
An asynchronous example
Icon
Summary
3
3. Parallelism in Python
Icon
3. Parallelism in Python
Icon
Multiple threads
Icon
Multiple processes
Icon
Multiprocess queues
Icon
Closing thoughts
Icon
Summary
4
4. Distributed Applications – with Celery
chevron up
Icon
4. Distributed Applications – with Celery
Icon
Establishing a multimachine environment
Icon
Installing Celery
Icon
Testing the installation
Icon
A tour of Celery
Icon
More complex Celery applications
Icon
Celery in production
Icon
Celery alternatives – Python-RQ
Icon
Celery alternatives – Pyro
Icon
Summary
5
5. Python in the Cloud
Icon
5. Python in the Cloud
Icon
Cloud computing and AWS
Icon
Creating an AWS account
Icon
Creating an EC2 instance
Icon
Storing data in Amazon S3
Icon
Amazon elastic beanstalk
Icon
Creating a private cloud
Icon
Summary
6
6. Python on an HPC Cluster
Icon
6. Python on an HPC Cluster
Icon
Your typical HPC cluster
Icon
Job schedulers
Icon
Running a Python job using HTCondor
Icon
Running a Python job using PBS
Icon
Debugging
Icon
Summary
7
7. Testing and Debugging Distributed Applications
Icon
7. Testing and Debugging Distributed Applications
Icon
The big picture
Icon
Common problems – clocks and time
Icon
Common problems – software environments
Icon
Common problems – permissions and environments
Icon
Common problems – the availability of hardware resources
Icon
Challenges – the development environment
Icon
A useful strategy – logging everything
Icon
A useful strategy – simulating components
Icon
Summary
8
8. The Road Ahead
Icon
8. The Road Ahead
Icon
The first two chapters
Icon
The tools
Icon
The cloud and the HPC world
Icon
Debugging and monitoring
Icon
Where to go next
9
Index
Icon
Index

Chapter 4. Distributed Applications – with Celery

This chapter is a follow up on some of the topics that we discussed so far. In particular, it explores asynchronous programming and distributed computing in detail with some example applications. It concentrates on Celery, a sophisticated Python framework that is used to build distributed applications. Toward the end, the chapter explores some alternative packages to Celery: Pyro and Python-RQ.

You should be familiar, at this point, with the basic ideas behind parallelism, distributed computing, and asynchronous programming. If not, it might be worthwhile to skim the previous chapters to get a refresher.

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 1
Next Section

Create a Note

Modal Close icon
You need to login to use this feature.
notes
bookmark search playlist font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Delete Note

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Edit Note

Modal Close icon
Write a note (max 255 characters)
Cancel
Update Note

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY