You would think that they hate each other. But in fact, they work together to give you great features!

Why the hell would I want to mix Tensorflow+Windows?

I am a huge Linux fan, I have been using Linux distros on my laptop since 2008-2009. On Linux, everything installs well and everything is good – Tensorflow even had a build guide for windows in version <1.0 which just said switch to Linux.
I use a Thinkpad X1 extreme as my personal laptop and I like to mix between casual browsing + movies + data science + machine learning. Despite the hate for Windows, let me take 10 seconds here to explain why I want(ed) Tensorflow on Windows. On Linux, you have the open source GPU drivers and then you have the proprietary NVIDIA drivers. NVIDIA drivers are great, but since they don’t have 100% kernel level integration, they can’t do switching, and I need that. When I want to do machine learning, I want to be able watch a movie or so, without having to end my experiments or restart my workspace. On Linux, that’s the biggest drawback. No distro, driver or spaghetti script allowed me run CUDA without having to restart my desktop environment and re-route to the NVIDIA GPU from the Intel one. Optimus is a big win!

This is pretty much an instruction guide to get Tensorflow 2.0alpha working on Windows with VS2019 and CUDA 10.1. Head down below for the pre-installation guide, build guide and a usage guide.

You tried pip install tensorflow-gpu==2.0.0-alpha0 and it failed!

Same here. I installed it under a conda environment, and it miserably failed. The install went through and the moment you do import tensorflow, everything goes bonkers.

Initially as a casual thought, I got on Windows to try this experiment. I spawned up a miniconda environment, and ran pip install tensorflow and I ran into DLL hell on import.

This seemed to occur in a couple of files. You have probably seen this, or I forgot to update PATH before doing this install.
On a different note, Miniconda is great, because of their MKL support, being soooo windows friendly, and not annoying like their big brother Anaconda.

Instead of quitting here, the most natural response after this seemed to be to build tensorflow for source, rather than using a pre-packaged version. A customly built tensorflow also has the advantage of being perfect for the set of libraries I have, and the hardware I have. For example, I don’t have to compile for every possible architecture of compute capability for my graphic card, but I can do it for my specific version alone.

Let’s get Ready

Things are not so direct with Tensorflow 2.0. There is a reason it is still in alpha, and not even in Beta. Cuda 10.1 isn’t officially supported by tensorflow, and neither is Visual Studio 2017. There are a couple of tricks here and there, which we got to jump through before we have a full fledged build environment.

Assumptions

  1. You have Windows 10
  2. You wanted the latest and greatest and went ahead to install Cuda v10.1. You probably have also managed to install CudNN 7.0
  3. Again, you live on the edge, so you went ahead and downloaded Visual Studio 2019. I had installed the Python Development tools as well while installing this.
  4. You have installed Python one way or the other. Either you have installed it using the Anaconda distribution, or you have installed it using the official installer of Python from python.org . I’ve used Miniconda to create an environment based on Python 3.6. Anaconda is great because of its built in support for MKL etc. Also, a lot of modules are packaged, so you do not have to waste time setting up an environment.
  5. You seem to want to break an arm and get Tensorflow 2.0 alpha on this unsupported environment. Ofcourse, the reasons for which you want it include Eager execution and the possibility of a Keras frontend.
  6. You seem to have certain build tools – such as Git (SublimeMerge is a great visual Git client). You also seem to know how to work with the command line to a certain extent. I use the barebones git for windows.
  7. You probably tried the official build guide and gave up
    https://www.tensorflow.org/install/source_windows

The Basic stuff

The following is pretty much a copy paste of Tensorflow’s official install guide. I’ve made some tweaks here and there. This would probably get outdated soon once 2.0 goes stable.

  • Download the Tensorflow 2.0 source : This involves a git clone
    http://github.com/tensorflow/tensorflow/
    . You could also download the entire source archive from https://www.tensorflow.org/
    • You might have to switch to the 2.0 alpha release branch if you used Git to download it. This is possible with git checkout r2.0
    • Do a git pull after a checkout, just to see whether there are some newer commits deviating from the master branch.
    • Alternatively, you could check out the fixed branch called tf2_winfixes from my fork of tensorflow at
      https://github.com/atemysemicolon/tensorflow/
  • Download Bazel v20. The latest version at the time of writing is Bazel 24.1 and that semi supports Visual Studio 2019. It also gets confused with Cuda 10.1, but that is probably because of a tensorflow script. However, no support is better than semi support.
    • Copy this to somewhere in Windows where your PATH variable points to , or add the path of the folder in which bazel exists to PATH.
  • Cheat 1 – Cheat Tensorflow/Bazel into thinking you really have Visual Studio 2017. This seemed to be the most seemless way to get an older version of Bazel to work with our newer versions of Visual Studio.
    • Navigate to the folder in which most of Visual Studio lies in. On my PC, this is : C:\Program Files (x86)\Microsoft Visual Studio
    • Make a hard link! mklink /D 2017 2019
    • This just makes a fake folder called 2017 which in reality points to 2019. This is better than a shortcut, because a shortcut is identified with an extension, and hardlinks are stored on the filesystem.
  • Cheat 2 – Modify relevant Bazel scripts and Cuda Scripts to prevent build errors in the following step
    • For your ease, I’ve made a fork of tensorflow, with my fixes. Have a look at
      https://github.com/atemysemicolon/tensorflow/commit/54faf06d5c10e8875b9ce36b941004aceb44c8b9 . Ideally, just copy the changes from the two files I have modified, or download my version of the files and patch it to your tensorflow folder.
    • You could download the entire repository if you want, and switch to the branch tf2_winfixes instead of cloning the official tensorflow branch. I would NOT suggest that as you would miss future changes of Tensorflow 2.0 (I will not attempt to keep the repo updated

You have the all the building blocks. All you have to do is create an environment and just press build.

Build!

So now that you have your code and tools ready, lets set up a build environment and start the build. This is pretty much the same as Tensorflow’s official guide.

1. Make a Python Virtualenv

Python Virtual environments are great! They prevent you from polluting the other python libraries already installed in your system, and to an extent, they offer portability. I use conda for my virtual environments, but you have a bunch of options – You could you virtualenv barebones, or the virtualenvwrapper or Pyenv, or even pipenv.

Open up an Anaconda Prompt (or a prompt where your python commands work)

conda create --name tf_build_env python=3.6.5

For a change, we will use a version of Python that is officially supported, therefore do not forget to introduce python=3.6.5

2. Install Tensorflow’s dependencies

I’m using Miniconda, but I wouldn’t take the risk to use conda. So this is where we mix regular pip with a conda environment. Let’s get your tensorflow dependencies in order. The keras stuff actually allow to use tensorflow’s keras as a frontend (new as of 2.0 alpha)

conda activate tf_build_env # Don't forget to activate your environment!
pip install six numpy wheel 
pip install keras_applications==1.0.6 --no-deps 
pip install keras_preprocessing==1.0.5 --no-deps

In addition, Install MSYS2 and add that to your PATH. Download it from here. If you install it to the default location, append C:\msys64\usr\bin to your path. Don’t forget to install the unzip and the patch packages for MSYS, I was stuck here for a while without reading that detail!

pacman -S unzip patch

If you have problems with your path, that is not going to work. Instead, navigate to the bin/ folder of your MSYS installation, start an MSYSterminal, and execute the command over there.

3. Configure, Build and Install.

To configure, execute the following, from within the tensorflow source : python ./configure.py
This is going to ask you multiple questions.

So many questions! But that is still better than compilation flags! Put things here that fit your installation. Put your version of CudNN, and if you would like to introduce other custom compiler flags, you can do so here. Also, make sure you put the right Compute Capability that depends on your graphic card. You can find that here :
https://developer.nvidia.com/cuda-gpus

It’s quite interactive, you don’t have to be a pro Linux or a windows user to understand that. We are almost there. To build, do the following :-

bazel build --config=opt --config=cuda --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package --define=no_tensorflow_py_deps=true --copt=-nvcc_options=disable-warnings

That would take an hour or so. In my case, I left it on the entire night. Just follow it for the first few minutes to check if something is broken. If you’re using a laptop, plug it in, this is going to be CPU intensive.

Once you have a build, it’s time to put all those different things that you built into a python package. The following does that and stores the built package in C:\tmp\tensorflow_pkg :-

bazel-bin\tensorflow\tools\pip_package\build_pip_package C:/tmp/tensorflow_pkg

All you have to do now is to install it with pip. Pip supports wheels, and the package you built is a wheel so all you need to do is the following (remember to replace the version number with the version of your build, or just install whatever tensorflow package was built) :-

pip install C:/tmp/tensorflow_pkg/tensorflow-version-cp36-cp36m-win_amd64.whl

That’s it, you’re good to go. You go tensorflow installed.

Baby Steps – Your first code

If you’re not sure what your first script could be, just use the script I used to train and test MNIST. It is a copy-paste of the code on the tensorflow website.

As usual, if you have any issues with the installation, comment and I’ll try to help you out. And, feel free to comment, as well, if you want to thank me xd,.