Python: Deeper Insights into Machine Learning

Table of Contents

Python: Deeper Insights into Machine Learning

Python: Deeper Insights into Machine Learning

Credits

Preface

What this learning path covers

What you need for this learning path

Who this learning path is for

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Module 1

1. Giving Computers the Ability to Learn from Data

Building intelligent machines to transform data into knowledge

The three different types of machine learning

Making predictions about the future with supervised learning

Classification for predicting class labels

Regression for predicting continuous outcomes

Solving interactive problems with reinforcement learning

Discovering hidden structures with unsupervised learning

Finding subgroups with clustering

Dimensionality reduction for data compression

An introduction to the basic terminology and notations

A roadmap for building machine learning systems

Preprocessing – getting data into shape

Training and selecting a predictive model

Evaluating models and predicting unseen data instances

Using Python for machine learning

Installing Python packages

Summary

2. Training Machine Learning Algorithms for Classification

Artificial neurons – a brief glimpse into the early history of machine learning

Implementing a perceptron learning algorithm in Python

Training a perceptron model on the Iris dataset

Adaptive linear neurons and the convergence of learning

Minimizing cost functions with gradient descent

Implementing an Adaptive Linear Neuron in Python

Large scale machine learning and stochastic gradient descent

Summary

3. A Tour of Machine Learning Classifiers Using Scikit-learn

Choosing a classification algorithm

First steps with scikit-learn

Training a perceptron via scikit-learn

Modeling class probabilities via logistic regression

Logistic regression intuition and conditional probabilities

Learning the weights of the logistic cost function

Training a logistic regression model with scikit-learn

Tackling overfitting via regularization

Maximum margin classification with support vector machines

Maximum margin intuition

Dealing with the nonlinearly separable case using slack variables

Alternative implementations in scikit-learn

Solving nonlinear problems using a kernel SVM

Using the kernel trick to find separating hyperplanes in higher dimensional space

Decision tree learning

Maximizing information gain – getting the most bang for the buck

Building a decision tree

Combining weak to strong learners via random forests

K-nearest neighbors – a lazy learning algorithm

Summary

4. Building Good Training Sets – Data Preprocessing

Dealing with missing data

Eliminating samples or features with missing values

Imputing missing values

Understanding the scikit-learn estimator API

Handling categorical data

Mapping ordinal features

Encoding class labels

Performing one-hot encoding on nominal features

Partitioning a dataset in training and test sets

Bringing features onto the same scale

Selecting meaningful features

Sparse solutions with L1 regularization

Sequential feature selection algorithms

Assessing feature importance with random forests

Summary

5. Compressing Data via Dimensionality Reduction

Unsupervised dimensionality reduction via principal component analysis

Total and explained variance

Feature transformation

Principal component analysis in scikit-learn

Supervised data compression via linear discriminant analysis

Computing the scatter matrices

Selecting linear discriminants for the new feature subspace

Projecting samples onto the new feature space

LDA via scikit-learn

Using kernel principal component analysis for nonlinear mappings

Kernel functions and the kernel trick

Implementing a kernel principal component analysis in Python

Example 1 – separating half-moon shapes

Example 2 – separating concentric circles

Projecting new data points

Kernel principal component analysis in scikit-learn

Summary

6. Learning Best Practices for Model Evaluation and Hyperparameter Tuning

Streamlining workflows with pipelines

Loading the Breast Cancer Wisconsin dataset

Combining transformers and estimators in a pipeline

Using k-fold cross-validation to assess model performance

The holdout method

K-fold cross-validation

Debugging algorithms with learning and validation curves

Diagnosing bias and variance problems with learning curves

Addressing overfitting and underfitting with validation curves

Fine-tuning machine learning models via grid search

Tuning hyperparameters via grid search

Algorithm selection with nested cross-validation

Looking at different performance evaluation metrics

Reading a confusion matrix

Optimizing the precision and recall of a classification model

Plotting a receiver operating characteristic

The scoring metrics for multiclass classification

Summary

7. Combining Different Models for Ensemble Learning

Learning with ensembles

Implementing a simple majority vote classifier

Combining different algorithms for classification with majority vote

Evaluating and tuning the ensemble classifier

Bagging – building an ensemble of classifiers from bootstrap samples

Leveraging weak learners via adaptive boosting

Summary

8. Applying Machine Learning to Sentiment Analysis

Obtaining the IMDb movie review dataset

Introducing the bag-of-words model

Transforming words into feature vectors

Assessing word relevancy via term frequency-inverse document frequency

Cleaning text data

Processing documents into tokens

Training a logistic regression model for document classification

Working with bigger data – online algorithms and out-of-core learning

Summary

9. Embedding a Machine Learning Model into a Web Application

Serializing fitted scikit-learn estimators

Setting up a SQLite database for data storage

Developing a web application with Flask

Our first Flask web application

Form validation and rendering

Turning the movie classifier into a web application

Deploying the web application to a public server

Updating the movie review classifier

Summary

10. Predicting Continuous Target Variables with Regression Analysis

Introducing a simple linear regression model

Exploring the Housing Dataset

Visualizing the important characteristics of a dataset

Implementing an ordinary least squares linear regression model

Solving regression for regression parameters with gradient descent

Estimating the coefficient of a regression model via scikit-learn

Fitting a robust regression model using RANSAC

Evaluating the performance of linear regression models

Using regularized methods for regression

Turning a linear regression model into a curve – polynomial regression

Modeling nonlinear relationships in the Housing Dataset

Dealing with nonlinear relationships using random forests

Decision tree regression

Random forest regression

Summary

11. Working with Unlabeled Data – Clustering Analysis

Grouping objects by similarity using k-means

K-means++

Hard versus soft clustering

Using the elbow method to find the optimal number of clusters

Quantifying the quality of clustering via silhouette plots

Organizing clusters as a hierarchical tree

Performing hierarchical clustering on a distance matrix

Attaching dendrograms to a heat map

Applying agglomerative clustering via scikit-learn

Locating regions of high density via DBSCAN

Summary

12. Training Artificial Neural Networks for Image Recognition

Modeling complex functions with artificial neural networks

Single-layer neural network recap

Introducing the multi-layer neural network architecture

Activating a neural network via forward propagation

Classifying handwritten digits

Obtaining the MNIST dataset

Implementing a multi-layer perceptron

Training an artificial neural network

Computing the logistic cost function

Training neural networks via backpropagation

Developing your intuition for backpropagation

Debugging neural networks with gradient checking

Convergence in neural networks

Other neural network architectures

Convolutional Neural Networks

Recurrent Neural Networks

A few last words about neural network implementation

Summary

13. Parallelizing Neural Network Training with Theano

Building, compiling, and running expressions with Theano

What is Theano?

First steps with Theano

Configuring Theano

Working with array structures

Wrapping things up – a linear regression example

Choosing activation functions for feedforward neural networks

Logistic function recap

Estimating probabilities in multi-class classification via the softmax function

Broadening the output spectrum by using a hyperbolic tangent

Training neural networks efficiently using Keras

Summary

2. Module 2

1. Thinking in Machine Learning

The human interface

Design principles

Types of questions

Are you asking the right question?

Tasks

Classification

Regression

Clustering

Dimensionality reduction

Errors

Optimization

Linear programming

Models

Geometric models

Probabilistic models

Logical models

Features

Unified modeling language

Class diagrams

Object diagrams

Activity diagrams

State diagrams

Summary

2. Tools and Techniques

Python for machine learning

IPython console

Installing the SciPy stack

NumPY

Constructing and transforming arrays

Mathematical operations

Matplotlib

Pandas

SciPy

Scikit-learn

Summary

3. Turning Data into Information

What is data?

Big data

Challenges of big data

Data volume

Data velocity

Data variety

Data models

Data distributions

Data from databases

Data from the Web

Data from natural language

Data from images

Data from application programming interfaces

Signals

Data from sound

Cleaning data

Visualizing data

Summary

4. Models – Learning from Information

Logical models

Generality ordering

Version space

Coverage space

PAC learning and computational complexity

Tree models

Purity

Rule models

The ordered list approach

Set-based rule models

Summary

5. Linear Models

Introducing least squares

Gradient descent

The normal equation

Logistic regression

The Cost function for logistic regression

Multiclass classification

Regularization

Summary

6. Neural Networks

Getting started with neural networks

Logistic units

Cost function

Minimizing the cost function

Implementing a neural network

Gradient checking

Other neural net architectures

Summary

7. Features – How Algorithms See the World

Feature types

Quantitative features

Ordinal features

Categorical features

Operations and statistics

Structured features

Transforming features

Discretization

Normalization

Calibration

Principle component analysis

Summary

8. Learning with Ensembles

Ensemble types

Bagging

Random forests

Extra trees

Boosting

Adaboost

Gradient boosting

Ensemble strategies

Other methods

Summary

9. Design Strategies and Case Studies

Evaluating model performance

Model selection

Gridsearch

Learning curves

Real-world case studies

Building a recommender system

Content-based filtering

Collaborative filtering

Reviewing the case study

Insect detection in greenhouses

Reviewing the case study

Machine learning at a glance

Summary

3. Module 3

1. Unsupervised Machine Learning

Principal component analysis

PCA – a primer

Employing PCA

Introducing k-means clustering

Clustering – a primer

Kick-starting clustering analysis

Tuning your clustering configurations

Self-organizing maps

SOM – a primer

Employing SOM

Further reading

Summary

2. Deep Belief Networks

Neural networks – a primer

The composition of a neural network

Network topologies

Restricted Boltzmann Machine

Introducing the RBM

Topology

Training

Applications of the RBM

Further applications of the RBM

Deep belief networks

Training a DBN

Applying the DBN

Validating the DBN

Further reading

Summary

3. Stacked Denoising Autoencoders

Autoencoders

Introducing the autoencoder

Topology

Training

Denoising autoencoders

Applying a dA

Stacked Denoising Autoencoders

Applying the SdA

Assessing SdA performance

Further reading

Summary

4. Convolutional Neural Networks

Introducing the CNN

Understanding the convnet topology

Understanding convolution layers

Understanding pooling layers

Training a convnet

Putting it all together

Applying a CNN

Further Reading

Summary

5. Semi-Supervised Learning

Introduction

Understanding semi-supervised learning

Semi-supervised algorithms in action

Self-training

Implementing self-training

Finessing your self-training implementation

Improving the selection process

Contrastive Pessimistic Likelihood Estimation

Further reading

Summary

6. Text Feature Engineering

Introduction

Text feature engineering

Cleaning text data

Text cleaning with BeautifulSoup

Managing punctuation and tokenizing

Tagging and categorising words

Tagging with NLTK

Sequential tagging

Backoff tagging

Creating features from text data

Stemming

Bagging and random forests

Testing our prepared data

Further reading

Summary

7. Feature Engineering Part II

Introduction

Creating a feature set

Engineering features for ML applications

Using rescaling techniques to improve the learnability of features

Creating effective derived variables

Reinterpreting non-numeric features

Using feature selection techniques

Performing feature selection

Correlation

LASSO

Recursive Feature Elimination

Genetic models

Feature engineering in practice

Acquiring data via RESTful APIs

Testing the performance of our model

Twitter

Translink Twitter

Consumer comments

The Bing Traffic API

Deriving and selecting variables using feature engineering techniques

The weather API

Further reading

Summary

8. Ensemble Methods

Introducing ensembles

Understanding averaging ensembles

Using bagging algorithms

Using random forests

Applying boosting methods

Using XGBoost

Using stacking ensembles

Applying ensembles in practice

Using models in dynamic applications

Understanding model robustness

Identifying modeling risk factors

Strategies to managing model robustness

Further reading

Summary

9. Additional Python Machine Learning Tools

Alternative development tools

Introduction to Lasagne

Getting to know Lasagne

Introduction to TensorFlow

Getting to know TensorFlow

Using TensorFlow to iteratively improve our models

Knowing when to use these libraries

Further reading

Summary

10. Chapter Code Requirements

A. Biblography

Index

Python: Deeper Insights into Machine Learning

Python: Deeper Insights into Machine Learning

Leverage benefits of machine learning techniques using Python

A course in three modules

BIRMINGHAM - MUMBAI

Python: Deeper Insights into Machine Learning

Copyright © 2016 Packt Publishing

All rights reserved. No part of this course may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this course to ensure the accuracy of the information presented. However, the information contained in this course is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this course.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this course by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Published on: August 2016

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78712-857-6

www.packtpub.com

Credits

Authors

Sebastian Raschka

David Julian

John Hearty

Reviewers

Richard Dutton

Dave Julian

Vahid Mirjalili

Hamidreza Sattari

Dmytro Taranovsky

Dr. Vahid Mirjalili

Jared Huffman

Ashwin Pajankar

Content Development Editor

Amrita Noronha

Production Coordinator

Arvindkumar Gupta

Preface

Machine learning and predictive analytics are becoming one of the key strategies for unlocking growth in a challenging contemporary marketplace .It is one of the fastest growing trends in modern computing and everyone wants to get into the field of machine learning. In order to obtain sufficient recognition in this field, one must be able to understand and design a machine learning system that serves the needs of a project. The idea is to prepare a Learning Path that will help you to tackle the real-world complexities of modern machine learning with innovative and cutting-edge techniques. Also, it will give you a solid foundation in the machine learning design process, and enable you to build customized machine learning models to solve unique problems

What this learning path covers

Module 1, Python Machine Learning, discusses the essential machine algorithms for classification and provides practical examples using scikit-learn. It teaches you to prepare variables of different types and also speaks about polynomial regression and tree-based approaches. This module focuses on open source Python library that allows us to utilize multiple cores of modern GPUs.

Module 2, Designing Machine Learning Systems with Python, acquaints you with large library of packages for machine learning tasks. It introduces broad topics such as big data, data properties, data sources, and data processing .You will further explore models that form the foundation of many advanced nonlinear techniques. This module will help you in understanding model selection and parameter tuning techniques that could help in various case studies.

Module 3, Advanced Machine Learning with Python, helps you to build your skill with deep architectures by using stacked denoising autoencoders. This module is a blend of semi-supervised learning techniques, RBM and DBN algorithms .Further this focuses on tools and techniques which will help in making consistent working process.

What you need for this learning path

Module 1, Python Machine Learning will require an installation of Python 3.4.3 or newer on Mac OS X, Linux or Microsoft Windows. Use of Python essential libraries like SciPy, NumPy, scikit-Learn, matplotlib, and pandas. is essential.

Before you start, Please refer:

The direct link to the Iris dataset would be: https://raw.githubusercontent.com/rasbt/python-machine-learning-book/master/code/datasets/iris/iris.data

We've added some additional notes to the code notebooks mentioning the offline datasets in case there are server errors. https://www.dropbox.com/sh/tq2qdh0oqfgsktq/AADIt7esnbiWLOQODn5q_7Dta?dl=0

Module 2, Designing Machine Learning Systems with Python, will need an inclination to learn machine learning and the Python V3 software, which you can download from https://www.python.org/downloads/.

Module 3, Advanced Machine Learning with Python, leverages openly available data and code, including open source Python libraries and frameworks.

Who this learning path is for

This title is for Data scientist and researchers who are already into the field of Data Science and want to see Machine learning in action and explore its real-world application. Prior knowledge of Python programming and mathematics is must with basic knowledge of machine learning concepts.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this course—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail <feedback@packtpub.com>, and mention the course's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt course, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this course from your account at http://www.packtpub.com. If you purchased this course elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the course in the Search box.

Select the course for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this course from.

Click on Code Download.

You can also download the code files by clicking on the Code Files button on the course's webpage at the Packt Publishing website. This page can be accessed by entering the course's name in the Search box. Please note that you need to be logged in to your Packt account.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the course is also hosted on GitHub at https://github.com/PacktPublishing/Python-Deeper-Insights-into-Machine-Learning.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our courses—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this course. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your course, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.

To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the course in the search field. The required information will appear under the Errata section.

Piracy

Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <copyright@packtpub.com> with a link to the suspected pirated material.

We appreciate your help in protecting our authors and our ability to bring you valuable content.

Questions

If you have a problem with any aspect of this course, you can contact us at <questions@packtpub.com>, and we will do our best to address the problem.

Part 1. Module 1

Python Machine Learning

Leverage benefits of machine learning techniques using Python

Chapter 1. Giving Computers the Ability to Learn from Data

In my opinion, machine learning, the application and science of algorithms that makes sense of data, is the most exciting field of all the computer sciences! We are living in an age where data comes in abundance; using the self-learning algorithms from the field of machine learning, we can turn this data into knowledge. Thanks to the many powerful open source libraries that have been developed in recent years, there has probably never been a better time to break into the machine learning field and learn how to utilize powerful algorithms to spot patterns in data and make predictions about future events.

In this chapter, we will learn about the main concepts and different types of machine learning. Together with a basic introduction to the relevant terminology, we will lay the groundwork for successfully using machine learning techniques for practical problem solving.

In this chapter, we will cover the following topics:

The general concepts of machine learning

The three types of learning and basic terminology

The building blocks for successfully designing machine learning systems

Installing and setting up Python for data analysis and machine learning

Building intelligent machines to transform data into knowledge

In this age of modern technology, there is one resource that we have in abundance: a large amount of structured and unstructured data. In the second half of the twentieth century, machine learning evolved as a subfield of artificial intelligence that involved the development of self-learning algorithms to gain knowledge from that data in order to make predictions. Instead of requiring humans to manually derive rules and build models from analyzing large amounts of data, machine learning offers a more efficient alternative for capturing the knowledge in data to gradually improve the performance of predictive models, and make data-driven decisions. Not only is machine learning becoming increasingly important in computer science research but it also plays an ever greater role in our everyday life. Thanks to machine learning, we enjoy robust e-mail spam filters, convenient text and voice recognition software, reliable Web search engines, challenging chess players, and, hopefully soon, safe and efficient self-driving cars.

The three different types of machine learning

In this section, we will take a look at the three types of machine learning: supervised learning, unsupervised learning, and reinforcement learning. We will learn about the fundamental differences between the three different learning types and, using conceptual examples, we will develop an intuition for the practical problem domains where these can be applied:

Making predictions about the future with supervised learning

The main goal in supervised learning is to learn a model from labeled training data that allows us to make predictions about unseen or future data. Here, the term supervised refers to a set of samples where the desired output signals (labels) are already known.

Considering the example of e-mail spam filtering, we can train a model using a supervised machine learning algorithm on a corpus of labeled e-mail, e-mail that are correctly marked as spam or not-spam, to predict whether a new e-mail belongs to either of the two categories. A supervised learning task with discrete class labels, such as in the previous e-mail spam-filtering example, is also called a classification task. Another subcategory of supervised learning is regression, where the outcome signal is a continuous value:

Classification for predicting class labels

Classification is a subcategory of supervised learning where the goal is to predict the categorical class labels of new instances based on past observations. Those class labels are discrete, unordered values that can be understood as the group memberships of the instances. The previously mentioned example of e-mail-spam detection represents a typical example of a binary classification task, where the machine learning algorithm learns a set of rules in order to distinguish between two possible classes: spam and non-spam e-mail.

However, the set of class labels does not have to be of a binary nature. The predictive model learned by a supervised learning algorithm can assign any class label that was presented in the training dataset to a new, unlabeled instance. A typical example of a multi-class classification task is handwritten character recognition. Here, we could collect a training dataset that consists of multiple handwritten examples of each letter in the alphabet. Now, if a user provides a new handwritten character via an input device, our predictive model will be able to predict the correct letter in the alphabet with certain accuracy. However, our machine learning system would be unable to correctly recognize any of the digits zero to nine, for example, if they were not part of our training dataset.

The following figure illustrates the concept of a binary classification task given 30 training samples: 15 training samples are labeled as negative class (circles) and 15 training samples are labeled as positive class (plus signs). In this scenario, our dataset is two-dimensional, which means that each sample has two values associated with it: and . Now, we can use a supervised machine learning algorithm to learn a rule—the decision boundary represented as a black dashed line—that can separate those two classes and classify new data into each of those two categories given its and values: