Pre-orders are Now Open for Deep Learning on Windows

Pre-orders for my new book, Deep Learning on Windows, are Now Open at Amazon.com!

Deep Learning on Windows is my latest book, and it is the longest and the most comprehensive book I have written to date. The book is meant for both beginners and intermediates to deep learning. It covers topics from setting up your tools on Windows and getting started, to complex but fun topics in deep learning and computer vision.

The Cover of 'Deep Learning on Windows'

The Windows OS accounts for over 70% of the desktop PC usage. Windows provides many conveniences, with a wide variety of available productivity tools, causing it to gather a large userbase. This means that there is a large percentage of you - AI enthusiasts and developers - out there that primarily work on the Windows OS, and would prefer to develop deep learning models on Windows itself.

Major Update on My New Book: Deep Learning on Windows

If you recall, I originally intended to incrementally release my new book - Deep Learning on Windows – and have it completed around the June/July timeframe. Well, the month of July came and went, but no book got released. I also was not posting articles in the blog frequently for the past couple of months.

This was due to a major change in the plan for the book.

Deep Learning on Windows was picked up by a publisher!

The book will now be published under Apress, and it became a much larger offering than what I initially envisioned.

The new cover for the book is shown below:

New Cover for Deep Learning on Windows

On late-May, while I was working on the first drafts of the book on LeanPub, I was contacted by Apress on whether I would be interested in publishing the book under Apress banner. 

Easy Text-to-Speech in Windows 10 Using PyWin32

Some time back, we've talked about how to build a speech recognition system in Python. Now let's look in to the other end of it: how to make a Python program that talks. More specifically, let's looks at building a text-to-speech system.

There are several libraries out there that would let you build a text-to-speech model: gTTS, tts_watson, Pyttsx, etc. But today, we'll be talking about using PyWin32 on Windows 10.

Windows 10 has a built-in speech engine, and you can access it through the PyWin32 library. As it uses the built-in system, it's quite efficient than other TTS methods on Windows, and does not require any external tools to playback the audio.

The PyWin32 library gets installed automatically if you're using Anaconda Python. If it's not installed, you can install it using either `conda install pywin32` or `pip install pywin32`.

Text-to-speech with PyWin32

Book Update: First 3 Chapters are Now Released!

I'm excited to let you know that the first 3 Chapters of my new book - Deep Learning on Windows - is now released at LeanPub!

Deep Learning on Windows

This is currently the only book focusing specifically on setting up and developing Deep Learning models on Microsoft Windows. As I mentioned in my earlier post, most of the queries and questions I have received over the past year or so were related to building deep learning models on the Windows OS, and how-to setup and troubleshoot the tools on it. There were very little material out there addressing Windows specifically.

So, this book is my answer to that.

A total of 12 chapters are planned for the book, covering topics from setting up your tools on Windows, building your first models, to some advanced topics like transfer learning, deploying your models, computer vision, generative adversarial networks, and reinforcement learning.

The book will be released in an incremental manner. The first 3 chapters are now released.
Buying the book now guarantees that you will get all the remaining content, and all future updates and revisions for free as they get released.

The price of the book will increase over time as new chapters gets added, so purchasing early gives you the best value.

Get Your Copy Now!

My New Book is in the works: Deep Learning on Windows

Hope you all are staying safe!

With the lockdown and working from home from my job I've had a bit of free time these past few days. I've spent some of that time on going through the comments of the blog, and the questions and queries I have received from you readers through email and social media channels.

What I was trying to do was to identify any major areas you are struggling in, and try to come up with solutions to address those areas.

What I found was that, most of the queries were related to building deep learning models on the Windows OS, and how-to setup and troubleshoot the tools on it.

So, as a means of addressing this area, I've decided to start writing a new book, which will be named "Deep Learning on Windows".

"Working" Cover Page of the New Book (cover would change in the final version)

Materials and support for setting up deep learning frameworks and building models on Windows is a little bit scarce out there. This is mainly because native support for Windows in many of the popular tools were not there until recently.

This is why I decided to write a book specifically on getting deep learning development started on Windows.

How to Install TensorFlow 2.0 on Anaconda (with CUDA support)

TensorFlow 2.0 has been released for a few months now. This latest version comes with many new features and improvements, such as eager execution, multi-GPU support, tighter Keras integration, and new deployment options such as TensorFlow Serving.

So, it's time we all switched to TensorFlow 2.0.

The Anaconda-native TensorFlow 2.0 packages are now available in the main conda repository. So, let's see how we can install TensorFlow 2.0 on Anaconda Python. This method will work on both Windows and Linux. And, if you have a CUDA capable NVIDIA GPU, you can enable GPU support as well.

Step 1.Creating a New Conda Environment

We'll start by creating a new conda environment. (I'll name it 'tensorflow2'. You can choose another name if you like):

conda create --name tensorflow2 python=3.7 anaconda

Create a new conda environment

How to Build and Install the Latest Version of Dlib on Anaconda on Windows

Dlib is a toolkit for C++ and Python containing machine learning algorithms and tools for creating complex software to solve real world problems. Dlib provides algorithms for machine learning/deep learning, multi-class classification and clustering models, support vector machines, regression models, a large set of numerical algorithms for areas such as matrix manipulations and linear algebra, graphical model inference algorithms, and utility algorithms for computer vision and image processing. And due to C++ implementations backing most of these implementations, they’re optimized to the point that can be used in some real-time applications as well.

If you’re interested in facial recognition models or facial emotion processing, then Dlib is a library you should definitely try out.

Dlib v19.19 in action on conda on Windows

But with all the great features in Dlib, installing it has always been a little bit troublesome because of some specific dependency requirements it needs which had a habit of almost always conflicting with your other libraries. With the latest versions however, installing Dlib has become somewhat simple.

If you’re using Anaconda Python for your python experiments, like me, you’ll find that there is no native Dlib package in the native conda package list. In one of my earlier tutorials I showed how to install the Dlib conda package from the conda-forge channel in to your conda environment. The conda-force package works perfectly fine, and it’s still one of the quickest ways to install Dlib.

But if you really want the latest official package of Dlib installed (v19.19 as the latest at the time of this writing) then using the pip package is the way to go. In order to install the Dlib pip package you’ll first have to setup some dependencies.

Failed Attempts at Building TensorFlow GPU from Source

For the last 3 weeks, I've been trying to build TensorFlow from source. I wanted to get TensorFlow GPU version working on Windows with CUDA 9.2 and cuDNN 7.1. Since the pre-built wheels only work with CUDA 9.0, the only way we can get it working with 9.2 is to build it ourselves from source.

The Windows build of TensorFlow is done using CMake. The official instructions are here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/cmake

Unfortunately, as I found out after multiple attempts, the build process is not as simple as it sounds.

Every attempt I have made to build it failed so far.

But, I decided to post the steps I took - which didn't work - so that you all may be able use it as a reference if you decided to try it out yourselves. Again, note that these steps did not work.

First, I started with gathering all the dependencies to build on Windows 10:

• Visual Studio 2015 Community Edition With Update 3 (14.0.25431.01) with C++
• Anaconda Python 3.6.5
• Git for Windows 2.18.0
• Swigwin 3.0.12
• CUDA Toolkit 9.2
• cuDNN 7.1
• CMake 3.11.3

Fixing the Matplotlib PyPlot import errors

About a week back, I was reinstalling Keras, TensorFlow and all the other libraries after a reformat of my PC. When I started verifying the library installations, I came across a strange error. When I tried to run a simple deep learning model, Python runtime crashed. As soon as I execute the script I was getting the "python.exe has stopped working" error message (I'm using Windows 10).

Matplotlib is an important part of my deep learning workflows

A little bit of debugging narrowed down the error to the following line in my script.

import matplotlib.pyplot as plt

(I was using matplotlib to graph the training history of the model. See "How to Graph Model Training History in Keras")

The error did not occur if I simply import matplotlib. It only occurred when specifically importing the pyplot module.

import matplotlib
# no errors

import matplotlib.pyplot as plt
# crash!!!

This is a known issue due to some library conflicts in the installation, which should hopefully be fixed in a future release. Until then, if you're getting this error, you can fix it by following the steps below.

Setting up TensorFlow with CUDA on Windows

I did the post about How to setup TensorFlow on Windows about a month back. I only covered setting up the CPU version of TensorFlow there, and promised that I'll do the guide for the GPU version soon. But I haven't had the change to come round to write the guide until now.

I guess better late than never. So, here's how to setup the GPU version of TensorFlow on Windows.

So, what do you need to get TensorFlow working on GPU?
You need to setup the following prerequisites, in that order.

1. Microsoft Visual Studio 2015 (The free community edition of VS 2015 will work)
2. CUDA 8.0
3. cuDNN 5.1 for CUDA 8.0

Start by installing Visual Studio 2015. You can get the free community edition from here. When you install, make sure to select ‘Custom Installation’, and select ‘Visual C++’ in the programming language selection (by default, C++ is not selected). Once installed, check whether you have C++ capability by checking the ‘New Project’ options.

Visual Studio 2015 installed with C++

Installing OpenCV 3 on Anaconda Python 3.5 on Windows

If you checked my last post, you know that TensorFlow now officially works on Windows. This was really exciting to me, since I was using a Linux machine for my TensorFlow experiments, I had to switch between the Linux and Windows for my other tasks. Now, I should be able to perform some of those experiments on Windows as well. (I would still switch to Linux for more processor intensive deep learning models). So, I decided to set everything up on Windows again.

I've been using OpenCV 3 on Anaconda Python 3.5 on Linux. (Check out my guide on Installing OpenCV from source on Anaconda Python on Ubuntu 16.10 if you are trying to install on Linux). But on Windows, I've previously only tried OpenCV on Python 2.7. Since the OpenCV Downloads page has pre-built binaries for OpenCV 3 (version 3.2 being latest), I thought the installation would be straightforward.

I downloaded and ran the OpenCV 3.2 installer (which just extracts OpenCV to a directory of your choosing), set the OPENCV_DIR environment variable, add the bin dir in OpenCV to the PATH, and finally copy the cv2.pyd file to the site-packages directory in my Anaconda environment. I fired up the Anaconda environment, and tried loading cv2 in Python. But I got the following error,

 (tensorflow) C:\>python  
 Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 5 2016, 11:41:13) [MSC v  
 .1900 64 bit (AMD64)] on win32  
 Type "help", "copyright", "credits" or "license" for more information.  
 >>> import cv2  
 Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
 ImportError: DLL load failed: The specified module could not be found.  

Setting up TensorFlow on Windows

TensorFlow, since it has been released just over an year ago, has gained a huge popularity with its capabilities. But the platforms it worked on was limited - there were only pre-packaged Python packages for Linux initially, while it gained a Mac OS compatible version along the way. Windows users were left wondering when (or if) there will be a Windows version of TensorFlow. There has been few unofficial builds for Windows, and some have attempted to build from source for Windows with various degrees of success. But, there was nothing official.

Now, I'm happy to learn that TensorFlow now officially supports Windows. The release notes suggests that Windows compatibility was present from TensorFlow v0.12.0 RC0, and it has been tested on Windows 7, Windows 10, and Windows Server 2016. It only supports 64-Bit Python 3.5. Both the standard Python distribution (from Python.org) and Anaconda Python are supported.

This was great news for me, since I was switching between Linux and Windows for my machine learning experiments. So, I gave a try to setup TensorFlow on Windows.

I've been using Anaconda Python for most of my experiments, so I created an Anaconda environment - named 'tensorflow' - with all the basic packages I need.

 conda create --name tensorflow numpy scipy scikit-learn pillow h5py mingw libpython  

Then I activated the environment I just created,

 activate tensorflow  

Now for the big step, installing TensorFlow from pip.

Switching between TensorFlow and Theano on Keras

Keras speeds up the task of building Neural Networks by providing high-level simplified functions to create and manipulate neural models. It itself does not provide the lower level neural and deep learning functions, but it’s rather meant to be run on an engine – which Keras refers to as a “backend” - which would provide such low-level functions.

Currently, Keras supports two such backends – TensorFlow and Theano.

The current version of Keras (v1.1.0 at the time of this writing) uses TensorFlow by default.

Most models written on top of Keras can be switched to a different backend without changes – at least it’s what’s said in the documentation. I’m yet to test this.

Which backend Kesas will use is defined in the Keras config file, which is located in the .keras directory in your home directory:
e.g.: on linux it would be ~/.keras/keras.json and on windows you can get to it on %USERPROFILE%\.keras\keras.json

For the default of using the TensorFlow backend, use the following config,

 {  
   "image_dim_ordering": "tf",  
   "epsilon": 1e-07,  
   "floatx": "float32",  
   "backend": "tensorflow"  
 }  

Notice the "backend" is set to "tensorflow" and "image_dim_ordering" is set to "tf".

To use the Theano backend, use the following,

 {  
   "image_dim_ordering": "th",   
   "epsilon": 1e-07,   
   "floatx": "float32",   
   "backend": "theano"  
 }  

Apart from the obvious "backend": "theano", note that "image_dim_ordering" is set to "th".

See my new post to see what the image_dim_ordering parameter in Keras does, and why is it important to set it properly.

Update: If you use Jupyter notebooks, and need to switch between TensorFlow and Theano backends quite often, fellow blogger desertnaut has a solution to dynamically switch the backend. Check out his solution at: Dynamically switch Keras backend in Jupyter notebooks

Related posts:
What is the image_dim_ordering parameter in Keras, and why is it important

Related links:
https://keras.io/backend/

Build Deeper: Deep Learning Beginners' Guide is the ultimate guide for anyone taking their first step into Deep Learning.

Get your copy now!

Installing Dlib on Anaconda Python on Windows

Update - 16/Dec/2019: There is now an easier way to install Dlib from the official pip package. Check out the new tutorial How to Build and Install the Latest Version of Dlib on Anaconda on Windows

Dlib is a Machine Learning library, primarily written in C++, but has a Python package also. It has many useful and optimized algorithms useful for machine learning, linear algebra, data structures, image processing, and much more available out-of-the-box.

"Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real world problems. It is used in both industry and academia in a wide range of domains including robotics, embedded devices, mobile phones, and large high performance computing environments. Dlib's open source licensing allows you to use it in any application, free of charge." - dlib.net

One of the most popular features in Dlib is Facial Landmark Detection. Dlib installation ships with a pre-trained shape predictor model named shape_predictor_68_face_landmarks.dat, which as the name suggests, is trained to detect 68 facial keypoints including eyes, eyebrows, mouth, nose, face outline, etc.

Dlib's Facial Landmark Detection in action

You can view the sample code for face landmark detection here at the Dlib website, and download the pre-trained model from http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2

(Make sure to unzip the .bz2 file once you download it)

Of course, Dlib is capable of much more than face landmark detection. I'm hoping to dig into some cool features of Dlib in later posts.

But first, we need to install it.

Working Theano configs

Here are the Theano configurations that I have tested and worked.
These were tested on Windows 10 64-Bit, and Windows 7 64-Bit.
(I will update when I test on other OS's and setups)

With GPU support, on CUDA and cuDNN

In order to allow Theano to use the GPU, you need to be on a machine with a supported Nvidia GPU, and have the CUDA toolkit and cuDNN setup. I will cover how to setup CUDA on a different post.

 [global]  
 floatX = float32  
 device = gpu  
   
 [nvcc]  
 flags=-LC:\Users\Thimira\Anaconda3  
 compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin  
   
 [dnn]  
 enabled = True  
   
 [lib]  
 cnmem=0.75  
   
 [blas]   
 ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas  

device = gpu tells Theano to use the GPU instead of the CPU.
flags=-LC:\Users\Thimira\Anaconda3 point this to your Python installation (I'm using Anaconda Python)
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin point this to the bin dir of your Visual Studio installation (Note: CUDA only worked with Visual Studio 2013 for me)
[dnn] enabled = True this enables cuDNN
cnmem=0.75 set the memory limit Theano can use of the GPU. Here it's set to 75% of the GPU memory
ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas point to your OpenBLAS installation. Refer to my earlier post Getting Theano working with OpenBLAS on Windows

With only CPU support

Since not everyone have a compatible Nvidia GPU to have CUDA.

 [global]  
 floatX = float32  
 device = cpu  
   
 [blas]  
 ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas  

device = cpu tells Theano to use the CPU.
ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas point to your OpenBLAS installation. Refer to my earlier post Getting Theano working with OpenBLAS on Windows

Build Deeper: Deep Learning Beginners' Guide is the ultimate guide for anyone taking their first step into Deep Learning.

Get your copy now!

Getting Theano working with OpenBLAS on Windows

I wanted to try out Machine Learning with Python, so my first choice was Keras with Theano.

Got Theano installed from Git (to get the latest development version):

 pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git  

Then, I needed to setup Theano with OpenBLAS (otherwise, training Keras models was painfully slow).
Since I was on Windows, I had to look around for instructions on how to setup OpenBLAS properly.

Luckily, OpenBLAS provides binaries for Windows - both 32Bit and 64Bit - although, they may not be for the latest version of OpenBLAS.

Head over to https://sourceforge.net/projects/openblas/files/ and see which release has the binaries already built for Windows. We need both OpenBLAS and MinGW binaries.

At the time of this writing the latest version of OpenBLAS was v0.2.19, which unfortunately doesn't have the Windows binaries released.

But, going back a few releases, we find that the release v0.2.15 includes the binaries - OpenBLAS-v0.2.15-Win64-int32.zip and mingw64_dll.zip.

Download both of the Zip files, and first extract the OpenBLAS Zip to a globally accessible location on your hard disk. (I would suggest a location such as C:\Dev_Tools\openblas\).
Then, extract the mingw Zip, and copy it's contents to the bin directory of your extracted OpenBLAS directory. e.g. If you extracted OpenBLAS to C:\Dev_Tools\openblas\, then copy the contents (3 DLL files) of mingw to C:\Dev_Tools\openblas\bin\.
i.e.: The extracted openblas\bin will have the libopenblas.dll in it. When you extract mingw, it will have 3 more DLLs - libgcc_s_seh-1.dll, libgfortran-3.dll, libquadmath-0.dll. Copy those to openblas\bin also.

Then, add the openblas\bin directory to your system path.

Finally, edit (or create) your .theanorc file with the following settings: (assuming you extracted OpenBLAS to C:\Dev_Tools\openblas\)
Note: If you don't already have a .theanorc file, create a file named .theanorc in the home directory of your user account, e.g. C:\Users\<your user>\.theanorc

 [global]  
 floatX = float32  
 device = cpu  
   
 [blas]  
 ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas  

Now, run your Keras/Theano program and see whether Theano picks up OpenBLAS.


Related Links:
http://www.openblas.net/

Build Deeper: Deep Learning Beginners' Guide is the ultimate guide for anyone taking their first step into Deep Learning.

Get your copy now!