Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Learning Data Mining with Python
  • Toc
  • feedback
Learning Data Mining with Python

Learning Data Mining with Python

By : Robert Layton
close
Learning Data Mining with Python

Learning Data Mining with Python

By: Robert Layton

Overview of this book

This book teaches you to design and develop data mining applications using a variety of datasets, starting with basic classification and affinity analysis. This book covers a large number of libraries available in Python, including the Jupyter Notebook, pandas, scikit-learn, and NLTK. You will gain hands on experience with complex data types including text, images, and graphs. You will also discover object detection using Deep Neural Networks, which is one of the big, difficult areas of machine learning right now. With restructured examples and code samples updated for the latest edition of Python, each chapter of this book introduces you to new algorithms and techniques. By the end of the book, you will have great insights into using Python for data mining and understanding of the algorithms as well as implementations.
Table of Contents (14 chapters)
close
close
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
Getting Started with Data Mining
Getting Started with Data Mining
Introducing data mining
Using Python and the Jupyter Notebook
A simple affinity analysis example
Product recommendations
A simple classification example
What is classification?
Summary
2
Classifying with scikit-learn Estimators
Classifying with scikit-learn Estimators
scikit-learn estimators
Preprocessing
Pipelines
Summary
3
Predicting Sports Winners with Decision Trees
Predicting Sports Winners with Decision Trees
Loading the dataset
Decision trees
Sports outcome prediction
Random forests
Summary
4
Recommending Movies Using Affinity Analysis
Recommending Movies Using Affinity Analysis
Affinity analysis
Dealing with the movie recommendation problem
Understanding the Apriori algorithm and its implementation
Summary
5
Features and scikit-learn Transformers
Features and scikit-learn Transformers
Feature extraction
Feature selection
Feature creation
Principal Component Analysis
Creating your own transformer
Unit testing
Putting it all together
Summary
6
Social Media Insight using Naive Bayes
Social Media Insight using Naive Bayes
Disambiguation
Downloading data from a social network
Text transformers
Naive Bayes
Applying of Naive Bayes
Getting useful features from models
Summary
7
Follow Recommendations Using Graph Mining
Follow Recommendations Using Graph Mining
Loading the dataset
Getting follower information from Twitter
Creating a graph
Finding subgraphs
Summary
8
Beating CAPTCHAs with Neural Networks
Beating CAPTCHAs with Neural Networks
Artificial neural networks
Creating the dataset
Training and classifying
Predicting words
Summary
9
Authorship Attribution
Authorship Attribution
Attributing documents to authors
Getting the data
Using function words
Support Vector Machines
Character n-grams
The Enron dataset
Putting it all together
Evaluation
Summary
10
Clustering News Articles
Clustering News Articles
Trending topic discovery
Extracting text from arbitrary websites
Grouping news articles
The k-means algorithm
Clustering ensembles
Online learning
Summary
11
Object Detection in Images using Deep Neural Networks
Object Detection in Images using Deep Neural Networks
Object classification
Application scenario
Deep neural networks
An Introduction to TensorFlow
Using Keras
GPU optimization
Application
Summary
12
Working with Big Data
chevron up
Working with Big Data
Big data
MapReduce
Applying MapReduce
Naive Bayes prediction
Extracting the blog posts
Training Naive Bayes
Putting it all together
Training on Amazon's EMR infrastructure
Summary
13
Next Steps...
Next Steps...
Getting Started with Data Mining
Classifying with scikit-learn Estimators
Predicting Sports Winners with Decision Trees
Recommending Movies Using Affinity Analysis
Extracting Features with Transformers
Social Media Insight Using Naive Bayes
Discovering Accounts to Follow Using Graph Mining
Beating CAPTCHAs with Neural Networks
Authorship Attribution
Clustering News Articles
Classifying Objects in Images Using Deep Learning
Working with Big Data
More resources

Training Naive Bayes

Now that we have extracted the blog posts, we can train our Naive Bayes model on them. The intuition is that we record the probability of a word being written by a particular gender, and record these values in our model. To classify a new sample, we would multiply the probabilities and find the most likely gender.

The aim of this code is to output a file that lists each word in the corpus, along with the frequencies of that word for each gender. The output file will look something like this:

"'ailleurs" {"female": 0.003205128205128205}
"'air" {"female": 0.003205128205128205}
"'an" {"male": 0.0030581039755351682, "female": 0.004273504273504274}
"'angoisse" {"female": 0.003205128205128205}
"'apprendra" {"male": 0.0013047113868622459, "female": 0.0014172668603481887}
"'attendent" {"female": 0.00641025641025641}
"'autistic" {"male": 0.002150537634408602}
"'auto" {"female": 0.003205128205128205}
"'avais" {"female": 0.00641025641025641}
"'avait" {"female": 0.004273504273504274...
End of Section 1
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete