Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Distributed Computing with Python
  • Table Of Contents Toc
  • Feedback & Rating feedback
Distributed Computing with Python

Distributed Computing with Python

By : Pierfederici
4.3 (3)
Buy this Book
close
close
Distributed Computing with Python

Distributed Computing with Python

4.3 (3)
By: Pierfederici
Buy this Book

Overview of this book

CPU-intensive data processing tasks have become crucial considering the complexity of the various big data applications that are used today. Reducing the CPU utilization per process is very important to improve the overall speed of applications. This book will teach you how to perform parallel execution of computations by distributing them across multiple processors in a single machine, thus improving the overall performance of a big data processing task. We will cover synchronous and asynchronous models, shared memory and file systems, communication between various processes, synchronization, and more.
Table of Contents (10 chapters)
close
close
Preface
In Progress | 0 / 7 sections completed | 0%
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Free Chapter
1
1. An Introduction to Parallel and Distributed Computing
In Progress | 0 / 7 sections completed | 0%
Icon
1. An Introduction to Parallel and Distributed Computing
Icon
Parallel computing
Icon
Distributed computing
Icon
Shared memory versus distributed memory
Icon
Amdahl's law
Icon
The mixed paradigm
Icon
Summary
2
2. Asynchronous Programming
In Progress | 0 / 4 sections completed | 0%
Icon
2. Asynchronous Programming
Icon
Coroutines
Icon
An asynchronous example
Icon
Summary
3
3. Parallelism in Python
In Progress | 0 / 6 sections completed | 0%
chevron up
Icon
3. Parallelism in Python
Icon
Multiple threads
Icon
Multiple processes
Icon
Multiprocess queues
Icon
Closing thoughts
Icon
Summary
4
4. Distributed Applications – with Celery
In Progress | 0 / 10 sections completed | 0%
Icon
4. Distributed Applications – with Celery
Icon
Establishing a multimachine environment
Icon
Installing Celery
Icon
Testing the installation
Icon
A tour of Celery
Icon
More complex Celery applications
Icon
Celery in production
Icon
Celery alternatives – Python-RQ
Icon
Celery alternatives – Pyro
Icon
Summary
5
5. Python in the Cloud
In Progress | 0 / 8 sections completed | 0%
Icon
5. Python in the Cloud
Icon
Cloud computing and AWS
Icon
Creating an AWS account
Icon
Creating an EC2 instance
Icon
Storing data in Amazon S3
Icon
Amazon elastic beanstalk
Icon
Creating a private cloud
Icon
Summary
6
6. Python on an HPC Cluster
In Progress | 0 / 7 sections completed | 0%
Icon
6. Python on an HPC Cluster
Icon
Your typical HPC cluster
Icon
Job schedulers
Icon
Running a Python job using HTCondor
Icon
Running a Python job using PBS
Icon
Debugging
Icon
Summary
7
7. Testing and Debugging Distributed Applications
In Progress | 0 / 10 sections completed | 0%
Icon
7. Testing and Debugging Distributed Applications
Icon
The big picture
Icon
Common problems – clocks and time
Icon
Common problems – software environments
Icon
Common problems – permissions and environments
Icon
Common problems – the availability of hardware resources
Icon
Challenges – the development environment
Icon
A useful strategy – logging everything
Icon
A useful strategy – simulating components
Icon
Summary
8
8. The Road Ahead
In Progress | 0 / 6 sections completed | 0%
Icon
8. The Road Ahead
Icon
The first two chapters
Icon
The tools
Icon
The cloud and the HPC world
Icon
Debugging and monitoring
Icon
Where to go next
9
Index
In Progress | 0 / 1 sections completed | 0%
Icon
Index

Multiprocess queues

When using multiple processes, the issue that comes up is how to exchange data between the workers. The multiprocessing module offers a mechanism to do that in the form of queues and pipes. Hence, we are going to look at multiprocess queues.

The multiprocessing.Queue class is modeled after the queue.Queue class with the additional twist that items stored in the multiprocessing queue need to be pickable. To illustrate how to use these queues, create a new Python script (queues.py) with the following code:

import multiprocessing as mp


def fib(n):
    if n <= 2:
        return 1
    elif n == 0:
        return 0
    elif n < 0:
        raise Exception('fib(n) is undefined for n < 0')
    return fib(n - 1) + fib(n - 2)


def worker(inq, outq):
    while True:
        data = inq.get()
        if data is None:
            return
        fn, arg = data
        outq.put(fn(arg))


if __name__ == '__main__':
    import argparse

    parser = argparse...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 4
Next Section

Create a Note

Modal Close icon
You need to login to use this feature.
notes
bookmark search playlist font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Delete Note

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY