unit08_cnn/lab08_fine_tune_partial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " # Lab 8:  Transfer Learning with a Pre-Trained Deep Neural Network\n",
    "\n",
    "As we discussed earlier, state-of-the-art neural networks involve millions of parameters that are prohibitively difficult to train from scratch.  In this lab, we will illustrate a powerful technique called *fine-tuning* where we start with a large pre-trained network and then re-train only the final layers to adapt to a new task.  The method is also called *transfer learning* and can produce excellent results on very small datasets with very little computational time.  \n",
    "\n",
    "This lab is based partially on this\n",
    "[excellent blog](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html).  In performing the lab, you will learn to:\n",
    "* Build a custom image dataset\n",
    "* Fine tune the final layers of an existing deep neural network for a new classification task.\n",
    "* Load images with a `DataGenerator`.\n",
    "\n",
    "You may run the lab on a CPU machine (like your laptop) or a GPU.  See the [notes](../GCP/gpu_setup.md) on setting up a GPU instance on Google Cloud Platform.  The GPU training is much faster (< 1 minute).  But, even the CPU machine training time will be less than 20 minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Dataset\n",
    "\n",
    "In this example, we will try to develop a classifier that can discriminate between two classes:  `cars` and `bicycles`.  One could imagine this type of classifier would be useful in vehicle vision systems.   The first task is to build a dataset.  \n",
    "\n",
    "TODO:  Create training and test datasets with:\n",
    "* 1000 training images of cars\n",
    "* 1000 training images of bicylces\n",
    "* 300 test images of cars\n",
    "* 300 test images of bicylces\n",
    "* The images don't need to be the same size.  But, you can reduce the resolution if you need to save disk space.\n",
    "\n",
    "The images should be organized in the following directory structure:\n",
    "\n",
    "    ./train\n",
    "        /car\n",
    "           car_0000.jpg\n",
    "           car_0001.jpg\n",
    "           ...\n",
    "           car_0999.jpg\n",
    "        /bicycle\n",
    "           bicycle_0000.jpg\n",
    "           bicycle_0001.jpg\n",
    "           ...\n",
    "           bicycle_0999.jpg\n",
    "    ./test\n",
    "        /car\n",
    "           car_0000.jpg\n",
    "           car_0001.jpg\n",
    "           ...\n",
    "           car_0299.jpg\n",
    "        /bicycle\n",
    "           bicycle_0000.jpg\n",
    "           bicycle_0001.jpg\n",
    "           ...\n",
    "           bicycle_0299.jpg\n",
    "           \n",
    "The naming of the files within the directories does not matter.  The `ImageDataGenerator` class below will find the filenames.  Just make sure there are the correct number of files in each directory.\n",
    "           \n",
    "A nice automated way of building such a dataset if through the [FlickrAPI](demo08_2_flickr_images.ipynb).           \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a Pre-Trained Deep Network\n",
    "\n",
    "We follow the [VGG16 demo](./vgg16.ipynb) to load a pre-trained deep VGG16 network.  We first load the appropriate Keras packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import applications\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras import optimizers\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dropout, Flatten, Dense\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also load some standard packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clear the Keras session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the dimensions of the input image.  The sizes below would work on a CPU machine.  But, if you have a GPU image, you can use a larger image size, like 150 x 150."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Set to larger values if you are using a GPU.\n",
    "nrow = 64\n",
    "ncol = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we follow the [VGG16 demo](./vgg16.ipynb) and load the deep VGG16 network.  Alternatively, you can use any other pre-trained model in keras.  When using the `applications.VGG16` method you will need to:\n",
    "* Set `include_top=False` to not include the top layer\n",
    "* Set the `image_shape` based on the above dimensions.  Remember, `image_shape` should be height x width x 3 since the images are color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Load the VGG16 network\n",
    "# input_shape = ...\n",
    "# base_model = applications.VGG16(weights='imagenet', ...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create now new model, we create a Sequential model.  Then, loop over the layers in `base_model.layers` and add each layer to the new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a new model\n",
    "model = Sequential()\n",
    "\n",
    "# TODO:  Loop over base_model.layers and add each layer to model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, loop through the layers in `model`, and freeze each layer by setting `layer.trainable = False`.  This way, you will not have to *re-train* any of the existing layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, add the following layers to `model`:\n",
    "* A `Flatten()` layer which reshapes the outputs to a single channel.\n",
    "* A fully-connected layer with 256 output units and `relu` activation\n",
    "* A final fully-connected layer.  Since this is a binary classification, there should be one output and `sigmoid` activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO\n",
    "# model.add(...)\n",
    "# model.add(...)\n",
    "# ...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the model summary.  This will display the number of trainable parameters vs. the non-trainable parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Generators to Load Data\n",
    "\n",
    "Up to now, the training data has been represented in a large matrix.  This is not possible for image data when the datasets are very large.  For these applications, the `keras` package provides a `ImageDataGenerator` class that can fetch images on the fly from a directory of images.  Using multi-threading, training can be performed on one mini-batch while the image reader can read files for the next mini-batch. The code below creates an `ImageDataGenerator` for the training data.  In addition to the reading the files, the `ImageDataGenerator` creates random deformations of the image to expand the total dataset size.  When the training data is limited, using data augmentation is very important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2000 images belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "train_data_dir = './train'\n",
    "batch_size = 32\n",
    "train_datagen = ImageDataGenerator(rescale=1./255,\n",
    "                                   shear_range=0.2,\n",
    "                                   zoom_range=0.2,\n",
    "                                   horizontal_flip=True)\n",
    "train_generator = train_datagen.flow_from_directory(\n",
    "                        train_data_dir,\n",
    "                        target_size=(nrow,ncol),\n",
    "                        batch_size=batch_size,\n",
    "                        class_mode='binary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, create a similar `test_generator` for the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO\n",
    "# test_generator = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function displays images that will be useful below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Display the image\n",
    "def disp_image(im):\n",
    "    if (len(im.shape) == 2):\n",
    "        # Gray scale image\n",
    "        plt.imshow(im, cmap='gray')    \n",
    "    else:\n",
    "        # Color image.  \n",
    "        im1 = (im-np.min(im))/(np.max(im)-np.min(im))*255\n",
    "        im1 = im1.astype(np.uint8)\n",
    "        plt.imshow(im1)    \n",
    "        \n",
    "    # Remove axis ticks\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how the `train_generator` works, use the `train_generator.next()` method to get a minibatch of data `X,y`.  Display the first 8 images in this mini-batch and label the image with the class label.  You should see that bicycles have `y=0` and cars have `y=1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model\n",
    "\n",
    "Compile the model.  Select the correct `loss` function, `optimizer` and `metrics`.  Remember that we are performing binary classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO.\n",
    "# model.compile(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using an `ImageDataGenerator`, we have to set two parameters manually:\n",
    "* `steps_per_epoch =  training data size // batch_size`\n",
    "* `validation_steps =  test data size // batch_size`\n",
    "\n",
    "We can obtain the training and test data size from `train_generator.n` and `test_generator.n`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we run the fit.  If you are using a CPU on a regular laptop, each epoch will take about 3-4 minutes, so you should be able to finish 5 epochs or so within 20 minutes.  On a reasonable GPU, even with the larger images, it will take about 10 seconds per epoch.\n",
    "* If you use `(nrow,ncol) = (64,64)` images, you should get around 90% accuracy after 5 epochs.\n",
    "* If you use `(nrow,ncol) = (150,150)` images, you should get around 96% accuracy after 5 epochs.  But, this will need a GPU.\n",
    "\n",
    "You will get full credit for either version.  With more epochs, you may get slightly higher, but you will have to play with the damping.\n",
    "\n",
    "Remember to record the history of the fit, so that you can plot the training and validation accuracy curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nepochs = 5  # Number of epochs\n",
    "\n",
    "# Call the fit_generator function\n",
    "hist = model.fit_generator(\n",
    "    train_generator,\n",
    "    steps_per_epoch=steps_per_epoch,\n",
    "    epochs=epochs,\n",
    "    validation_data=test_generator,\n",
    "    validation_steps=validation_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Plot the training accuracy and validation accuracy curves on the same figure.\n",
    "\n",
    "# TO DO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate your model on some test images\n",
    "\n",
    "Apply your trained model to some specific test data and evaluate the accuracy. Also, display the test image and classified label to see whether the prediction makes sense. Note that the result depends on the test images you use. You may not get very accurate prediction for a particular minibatch. You can run this several times to see the effect on different minibatches.\n",
    "\n",
    "*  Generate a mini-batch `Xtest,ytest` from the `test_generator.next()` method\n",
    "*  Predict the labels using the `model.predict( )` method and compute predicted labels `yhat`.\n",
    "*  Display the images with their ground truth labels and predicted labels \n",
    "*  Look at the images and their predicted labels, to see why the prediction is wrong in some cases.\n",
    "*  If you did not get any prediction error in one minibatch, run it multiple times.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TO DO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train the model with Batch Normalization and Dropout\n",
    "Repeat the previous process with batch normalization and dropout for training. \n",
    "\n",
    "Make sure that you clear the session and name your model with a different name, so that it will not continue to train from the previous model, and you can plot the performance of this model separately.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TO DO\n",
    "\n",
    "# Set up the model and fit\n",
    "# TO DO\n",
    "\n",
    "# Plot the training performance\n",
    "# TO DO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question:  Observe the difference in the training and validation accuracy curves with and without using batch normalization and dropout. Are the results as expected? Explain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}