unit07_neural/demo07_2_mnist_neural.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Classification via a Neural Network\n",
    "\n",
    "In the the [MNIST SVM example](../svm/mnist_svm.ipynb), we introduced the classic MNIST digit classification problem and trained a simple SVM classifier for the model.  In this demo, we will try a simple neural network.  The network we will create will not perform quite as well -- we will obtain an accuracy of only around 97%, while the SVM classifier obtains an accuracy of over 98%.  However, once we understand these simple neural networks, we will be able to build more sophisticated networks that can obtain much better classification rate.  Also, in doing this demo, you will learn several important features of the `keras` package in addition to the concepts shown in the [simple neural network example](./synthetic.ipynb):\n",
    "\n",
    "* How to construct multi-class classifiers using categorical cross entropy.\n",
    "* How to optimize using mini-batches.\n",
    "* How to save and load the model after training.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Keras package and the MNIST data\n",
    "\n",
    "We first load the `keras` package as in the [simple neural network example](./synthetic.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also load some other common packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we get the MNIST data as in the [MNIST SVM example](../svm/mnist_svm.ipynb).  We rescale the input `X` from values from -1 to 1 as this works better for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "mnist = fetch_mldata(\"MNIST original\")\n",
    "\n",
    "X = 2*(mnist.data/255-0.5)   # Change X from -1 to 1\n",
    "y = mnist.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also split the data into traing and test.  A standard split uses 50,000 examples for training and 10,000 for test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "ntr = 50000\n",
    "nts = 10000\n",
    "\n",
    "Xtr, Xts, ytr, yts = train_test_split(X,y,train_size=ntr, test_size=nts,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also use the `plt_digit` function from the [MNIST SVM example](../svm/mnist_svm.ipynb) to display digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAABeCAYAAAAUjW5fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACrtJREFUeJzt3Wdszd8fB/B3//jZRIwHaoUnxAq1KnZiC2ILiRWrQlCx\nKkbQmLVXIxWrfWDEiBKbRqRoiFkjVlFSM6Uozf0/kPO556v39t5bvd+ee/t+PfHJuev8Tvo7+XzP\nDHE4HCAiosL3v8KuABER/cEOmYjIEOyQiYgMwQ6ZiMgQ7JCJiAzBDpmIyBDskImIDMEOmYjIEOyQ\niYgMwQ6ZiMgQxX15c0hICPdZe+BwOELy8zm2rVfeOxyOqr5+iG3rlXy1LcD29ZJX7csMmQLJi8Ku\nQBBj2/qXV+3LDpmIyBDskImIDMEOmYjIEOyQiYgMwQ6ZiMgQ7JCJiAzBDpmIyBA+bQwh8qRYsWIA\ngE2bNknZxIkTAQCXLl2Ssi5duthbMaIAwAyZiMgQ7JCJiAwRlEMWFStWBACUK1fO5evh4eEAgL59\n+0rZ2bNnAQAHDx6UsqysLH9VMagUL+78M9qwYQMAYNKkSVJ29epVABymIPKEGTIRkSHYIRMRGSLE\n4fD+5DxTjtlTQxINGzaUsrlz50rcsmVLAEC1atW8/s6QkD+nZj558kTKXr16JfHixYsBAJcvX87z\ne4rK8Zv6MMX69esljoiIAAA8e/ZMytq3bw8AePPmzb/+bIrD4Wjh64dMadsZM2YAABo0aCBlqamp\nEm/duhUAkJOTI2W/fv2yqXb5a1ugcNo3JiYGgPPvDQAePXoEwNmOALB9+3Z7K+aeV+3LDJmIyBDG\nZ8ilSpUCAEyYMEHKIiMjAQA1atRw+RmV7br7b3v69CkAIDo6OtfvbN682eVn7t+/D8CZfQPAjx8/\ncr0v2DNktc5448aNUjZ58mSJVUbXv39/KTt58mRB/XzAZMiqnfSJ5d69ewMA9u3bl+dnMzIyJG7S\npInE7969K8gq/s3IDLls2bISnzhxQuIOHTq4/Yz+hDF79myJd+zYAaDQJuuZIRMRBRJ2yEREhjB+\nyGL16tUAgJkzZ3r9GTUZl5SUJGX6Vt47d+4AsD66nDt3DgDQqVOnPL97yJAhEh86dCjX68E+ZBEW\nFgYAuH79usvXU1JSAFiHdgpQwAxZ9OnTBwBw7Ngxnz+rP3I3btxYYn0C0A+MHLKoXr26xPoku/L9\n+3eJ1V6Cnj17Spk++ZycnAzAOvyp+gIbcMiCiCiQGLlTT9/RNW3aNLfvU1ktAEyZMkXi9PR0AMDX\nr1/z/J0zZ85I3LlzZwDuJwKV/fv3S+wqQw5G+jKt8+fP53pdLTcCnJlhUaRns3v37nX7Pj0D1jM8\nJSoqSmI/Z8XGmzp1ap6vt27dWuK7d+/mev3ixYsSq4nAK1euSFnHjh0lvnnzZn6rWWCYIRMRGYId\nMhGRIYwcsjh69KjE//33HwAgMzNTymbNmgUAiI2N9fo71TpjwLmeUQ1TAM61yx8/fpSy+fPnS6x+\nSx/mCGaNGjWSeMuWLRKXL18egHVHY7t27SR+//69DbUz07Zt2yRWu0l1CQkJAIDExEQp04fAKLfh\nw4e7LFd9xIMHD/L8/OPHjyVWQxb62vDBgwdLzCELIiIS7JCJiAxh5JCFTq16+PLli5R5O1RRpkwZ\nifUVEWqtsb6iQm2n7tWrl5TpqweUrl27evXbgU6t/wachwMBzqEd9fgNFO1hitDQUIn1w66U+Ph4\niefNmwcASEtL83/FgoS+jlh348YNANYVK4o+3DZw4MBcr3/69EliX4Y97cAMmYjIEMZnyPkxbtw4\nANZJuTp16kis1idPnz5dynbt2mVP5QynJpm6devm8vWHDx8CAFatWmVbnUymDroCrBN5nz9/BuDM\nigFmxvmhH3c7bNgwiUeMGAEAqFy5spSpJ9qVK1dKmZqE1j1//txlbAJmyEREhmCHTERkCCOHLPRH\nu/r16wOw3v6hDrbR13C2bdtW4kGDBgGwTtqp84wB5zZrT7d/FEXdu3cH4Jy8+5s6nzc7O9u2Opms\nWbNmLsvVsJinYYpKlSpJPHLkSADWtbXqwJyiSu05AKxDFqpfUP/6Ii4u7t8r5ifMkImIDGFkhtyq\nVSuJ1YEhNWvWlLLmzZtb/v2bmng6fvy4lC1YsEBiZnd/qCVFhw8fljJ9kkRR7Qmw7QpCixbOUxj1\n+99UuX74TVHPkPVbUnbv3i3xqFGjfP6unz9/AnB/dKwJmCETERmCHTIRkSGMGbLQd9UNGDBA4tq1\nawPwfE6x/giT11m05DR06FAA1jOMVTvr18/PmTPH3ooFEL1t9uzZI7Fa937v3r1cn6lbt67EJUuW\nzPW6Pmyk/39RSJdzFip9J96YMWMkVocL9evXT8rUoWELFy6UMn3X3unTpwEA165d809lCwAzZCIi\nQ7BDJiIyRKEPWajtpupxArDOQqtHaE9DFkV9Nrqg6WdS6zFZ6Y+/+t/w2LFjAVivv/KWvqKoatWq\nEr948SI/VQxKR44csfyri4iIsLs6BYYZMhGRIQolQ1Y76QBg7dq1AIAaNWpImX7Uplp7mJGRIWVL\nly7N9Z361d5LliwpuMoGMXVAiyuvX7+2sSbBwdWFvCpTBpwTdC9fvpSydevWSbxixQoAwK1bt6SM\nWbHvPD1Nm4wZMhGRIdghExEZwtYhC3VAkL5eU63D1E/x12+oUAet6Gs31dpP/bJC/XVyT9+Wrm5O\n0X379g0AsHz5cruqFJTU8IV+8WmVKlUAAB8+fJAy/dArdbuFu4OdKPgxQyYiMgQ7ZCIiQ9g6ZNGj\nRw8A1u2i6tqV8PBwKVPX3+jUJaSAc+bZ1aWSlLdFixZJXKpUqVyv79y5E0DRvri0IOlnG7uin8nd\ntGlTAMDt27f9WicyFzNkIiJD2JohqzXF+jrBU6dOAXCdFetq1aolsdrJxMkP7+iH1agbP9xxdSkk\n+Y++/p5tT8yQiYgMwQ6ZiMgQtg5ZuDooSJ+sU/TDhdTaTX1dbIUKFXJ9j37VC1mNHj1a4tDQ0Dzf\nm56e7ufaUN++fSXWDxJSkpOT7axO0KlXr57L8kA4JIsZMhGRIQr9+E11oMr48eOlTL/aW13E6erA\nkKioKIljYmL8VcWAp9/A4klCQoIfa0IAULp0aYnV3zfg3K26YcMG2+sUTNQtQ4C131C3i8TFxdle\nJ28xQyYiMgQ7ZCIiQ9g6ZLFmzRoAQGRkpJSp3WLudt39/v0bAJCUlCRl6nCiAwcOSFl2dnbBVjaI\nXLhwQWJ9R+THjx8BWM+XTk1Nta9iZHmkVrtW09LSCqs6AS06OhqA+/OQY2Nj7axOvjBDJiIyBDtk\nIiJDhPhy3UlISMg/3Y1SrFgxAM5zkQHn1Uv6ecbLli2TODMzEwDw9u3bf/lp2zgcjnzt5/7Xti0i\nUhwORwvPb7MyrW31lRXx8fESq+MB2rRpY3udkM+2BcxpX3W2tL5KS9+foNYnZ2Vl2VuxP7xqX2bI\nRESGsHVSLycnB4B1NxgvJKWiRk1UA9YdqGriOiwsTMpSUlLsq1iAS0xMBACUKFFCyvSn7ULKjH3C\nDJmIyBDskImIDGHrpF5RwEk9vwqKST1DBfyknuE4qUdEFEjYIRMRGYIdMhGRIdghExEZwtd1yO8B\nvPBHRYJEbc9vcYtt61l+25dt6xn/dv3Lq/b1aZUFERH5D4csiIgMwQ6ZiMgQ7JCJiAzBDpmIyBDs\nkImIDMEOmYjIEOyQiYgMwQ6ZiMgQ7JCJiAzxf/GVb8kz3cihAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x3900075fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plt_digit(x):\n",
    "    nrow = 28\n",
    "    ncol = 28\n",
    "    xsq = x.reshape((nrow,ncol))\n",
    "    plt.imshow(xsq,  cmap='Greys_r')\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    \n",
    "# Select random digits\n",
    "nplt = 4\n",
    "nsamp = X.shape[0]\n",
    "Iperm = np.random.permutation(nsamp)\n",
    "\n",
    "# Plot the images using the subplot command\n",
    "for i in range(nplt):\n",
    "    ind = Iperm[i]\n",
    "    plt.subplot(1,nplt,i+1)\n",
    "    plt_digit(X[ind,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Simple Neural Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create the neural network, we first import the appropriate sub-packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Model, Sequential\n",
    "from keras.layers import Dense, Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we clear the session.  As in the [simple neural network example](./synthetic.ipynb), this step is not necessary, but it is good practice as it clears any model layers that you have built before.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import keras.backend as K\n",
    "K.clear_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a very simple network.  The features are:\n",
    "*  We have one hidden layer with `nh=100` units.  \n",
    "*  One output layer with `nout=10` units, one for each of the 10 possible classes\n",
    "*  The output activation is `softmax`, which is used for multi-class targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nin = X.shape[1]  # dimension of input data\n",
    "nh = 100     # number of hidden units\n",
    "nout = int(np.max(y)+1)    # number of outputs = 10 since there are 10 classes\n",
    "model = Sequential()\n",
    "model.add(Dense(nh, input_shape=(nin,), activation='sigmoid', name='hidden'))\n",
    "model.add(Dense(nout, activation='softmax', name='output'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We print the model summary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "hidden (Dense)               (None, 100)               78500     \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 79,510\n",
      "Trainable params: 79,510\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Network\n",
    "\n",
    "As before, to train the network, we have to select an optimizer and a loss function.  Since this is a multi-class classification problem, we select the `sparse_categorial_crossentropy` loss.  We use the `adam` optimizer as before.  You may want to play with the learning rate `lr`.   We also set the `metrics` that we wish to track during the optimization.  In this case, we select `accuracy` on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras import optimizers\n",
    "\n",
    "opt = optimizers.Adam(lr=0.001) # beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n",
    "model.compile(optimizer=opt,\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras provides a simple method `fit` to run the optimization.  You simply specify the number of epochs and the batch size, both discussed in class.  In addition, we specify the `validation_data` so that it can print the accuracy on the test data set as it performs the optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/30\n",
      "50000/50000 [==============================] - 2s 44us/step - loss: 0.5378 - acc: 0.8654 - val_loss: 0.3119 - val_acc: 0.9105\n",
      "Epoch 2/30\n",
      "50000/50000 [==============================] - 2s 41us/step - loss: 0.2681 - acc: 0.9241 - val_loss: 0.2364 - val_acc: 0.9327\n",
      "Epoch 3/30\n",
      "50000/50000 [==============================] - 2s 43us/step - loss: 0.2135 - acc: 0.9394 - val_loss: 0.2015 - val_acc: 0.9404\n",
      "Epoch 4/30\n",
      "50000/50000 [==============================] - 2s 44us/step - loss: 0.1784 - acc: 0.9499 - val_loss: 0.1763 - val_acc: 0.9483\n",
      "Epoch 5/30\n",
      "50000/50000 [==============================] - 2s 49us/step - loss: 0.1541 - acc: 0.9563 - val_loss: 0.1620 - val_acc: 0.9535\n",
      "Epoch 6/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.1345 - acc: 0.9619 - val_loss: 0.1531 - val_acc: 0.9556\n",
      "Epoch 7/30\n",
      "50000/50000 [==============================] - 2s 41us/step - loss: 0.1199 - acc: 0.9656 - val_loss: 0.1371 - val_acc: 0.9592\n",
      "Epoch 8/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.1058 - acc: 0.9701 - val_loss: 0.1290 - val_acc: 0.9619\n",
      "Epoch 9/30\n",
      "50000/50000 [==============================] - 2s 41us/step - loss: 0.0959 - acc: 0.9735 - val_loss: 0.1285 - val_acc: 0.9612\n",
      "Epoch 10/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0868 - acc: 0.9759 - val_loss: 0.1229 - val_acc: 0.9645\n",
      "Epoch 11/30\n",
      "50000/50000 [==============================] - 2s 40us/step - loss: 0.0794 - acc: 0.9784 - val_loss: 0.1130 - val_acc: 0.9643\n",
      "Epoch 12/30\n",
      "50000/50000 [==============================] - 2s 49us/step - loss: 0.0716 - acc: 0.9807 - val_loss: 0.1106 - val_acc: 0.9670\n",
      "Epoch 13/30\n",
      "50000/50000 [==============================] - 2s 45us/step - loss: 0.0663 - acc: 0.9816 - val_loss: 0.1079 - val_acc: 0.9662\n",
      "Epoch 14/30\n",
      "50000/50000 [==============================] - 2s 43us/step - loss: 0.0620 - acc: 0.9834 - val_loss: 0.1042 - val_acc: 0.9682\n",
      "Epoch 15/30\n",
      "50000/50000 [==============================] - 2s 41us/step - loss: 0.0549 - acc: 0.9856 - val_loss: 0.1014 - val_acc: 0.9689\n",
      "Epoch 16/30\n",
      "50000/50000 [==============================] - 2s 41us/step - loss: 0.0515 - acc: 0.9864 - val_loss: 0.0960 - val_acc: 0.9692\n",
      "Epoch 17/30\n",
      "50000/50000 [==============================] - 2s 41us/step - loss: 0.0469 - acc: 0.9877 - val_loss: 0.0999 - val_acc: 0.9693\n",
      "Epoch 18/30\n",
      "50000/50000 [==============================] - 2s 43us/step - loss: 0.0428 - acc: 0.9896 - val_loss: 0.0969 - val_acc: 0.9698\n",
      "Epoch 19/30\n",
      "50000/50000 [==============================] - 2s 47us/step - loss: 0.0396 - acc: 0.9902 - val_loss: 0.0937 - val_acc: 0.9723\n",
      "Epoch 20/30\n",
      "50000/50000 [==============================] - 2s 46us/step - loss: 0.0365 - acc: 0.9912 - val_loss: 0.0946 - val_acc: 0.9708\n",
      "Epoch 21/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0344 - acc: 0.9915 - val_loss: 0.0941 - val_acc: 0.9709\n",
      "Epoch 22/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0313 - acc: 0.9929 - val_loss: 0.0925 - val_acc: 0.9722\n",
      "Epoch 23/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0291 - acc: 0.9938 - val_loss: 0.0912 - val_acc: 0.9720\n",
      "Epoch 24/30\n",
      "50000/50000 [==============================] - 2s 43us/step - loss: 0.0268 - acc: 0.9943 - val_loss: 0.0942 - val_acc: 0.9711\n",
      "Epoch 25/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0252 - acc: 0.9948 - val_loss: 0.0953 - val_acc: 0.9720\n",
      "Epoch 26/30\n",
      "50000/50000 [==============================] - 2s 44us/step - loss: 0.0234 - acc: 0.9954 - val_loss: 0.0942 - val_acc: 0.9700\n",
      "Epoch 27/30\n",
      "50000/50000 [==============================] - 2s 50us/step - loss: 0.0222 - acc: 0.9954 - val_loss: 0.1062 - val_acc: 0.9678\n",
      "Epoch 28/30\n",
      "50000/50000 [==============================] - 2s 43us/step - loss: 0.0210 - acc: 0.9955 - val_loss: 0.0942 - val_acc: 0.9706\n",
      "Epoch 29/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0176 - acc: 0.9970 - val_loss: 0.1089 - val_acc: 0.9683\n",
      "Epoch 30/30\n",
      "50000/50000 [==============================] - 2s 42us/step - loss: 0.0196 - acc: 0.9961 - val_loss: 0.0956 - val_acc: 0.9713\n"
     ]
    }
   ],
   "source": [
    "hist = model.fit(Xtr, ytr, epochs=30, batch_size=100, validation_data=(Xts,yts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the 10 epochs, you should obtain a test accuracy of around 96.5%.  If we run it for another a few epochs, we can get slightly higher results.  We can just run the `model.fit` command again, and it will start where it left off. You should get a little more than 97% accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training accuracy and validation accuracy as a function of epoches.\n",
    "We see that the training accuracy keeps growing to 1, while the validation accuracy saturates to a value around 0.97"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x3919773d68>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4lNXZ+PHvnX0jCwEDJEhQWQKBsAkoSEFEUeu+71KQ\n2rq19te3am212/taa622tbVqVVwBtdYNQREi0Aqy72HfkrCF7Psy9++PZxJCEsgQmEwyuT/XNdfM\ns865M8ncec45zzmiqhhjjDHNCfB1AYwxxrQPljCMMcZ4xBKGMcYYj1jCMMYY4xFLGMYYYzxiCcMY\nY4xHLGEYY4zxiCUMY4wxHrGEYYwxxiNBvi7A6dSlSxdNTk5u0bElJSVERkae3gL5kL/FA/4Xk7/F\nA/4Xk7/FA41jWrlyZY6qdvXkWL9KGMnJyaxYsaJFx6anpzN+/PjTWyAf8rd4wP9i8rd4wP9i8rd4\noHFMIrLH02O9ViUlIq+KyCER2XCc7SIifxaR7SKyTkSG1ds2WUS2uLc94q0yGmOM8Zw32zBeByaf\nYPulQB/3YzrwdwARCQRecG8fANwiIgO8WE5jjDEe8FrCUNVFQO4JdrkKeEMdS4FYEekOjAS2q+pO\nVa0EZrr3NcYY40O+7CWVCOyrt5zpXne89cYYY3yo3Td6i8h0nCotEhISSE9Pb9F5iouLW3xsW+Rv\n8YD/xeRv8YD/xeRv8cCpxeTLhJEF9Ky3nOReF3yc9U1S1ZeAlwBGjBihLe3R4G+9IfwtHvC/mPwt\nHvC/mPwtHji1mHxZJfUxcKe7t9RooEBV9wPLgT4i0ltEQoCb3fsaY4zxIa9dYYjIu8B4oIuIZAJP\n4Fw9oKovAnOAy4DtQCkwxb2tWkTuB+YBgcCrqrrRW+U0xpi2JKe4gvWZBWw+UEinsGCSYsNJigsn\nMS6ciBDftiJ47d1V9ZZmtitw33G2zcFJKMYY47fySytZl1nA+qwC1mXmsz6zgOyC8uPu3zkyhMTa\nBFKXSCJIigsnpXu018vb7hu9jTHmdKusdpFbUklWsYstB4pQFJcLFEUVVMGliuJ+VqXGBTUupcal\nVLtcuFSprnGWa9S9vkbJKa5gXVYB6zML2JtbWveeyfERDE/uzPeSYhiUGENKj2jKKmvIzCslM6+M\nzLwysvKd560Hi1iQcYiKahcA8ZEhrPzFJK//XCxhGGP8mqpSWllDflkVBaVV5JdVkltSyZHiSo4U\nV5BT4jwfKa7kSEklOcUVFJVXHz3BkkWnvUxJceEMTorhlpFnMjgphtQeMcREBDfaLzosmIToMIb3\najquIyWVZOaVUVReddrL2BRLGMaYdqH2iz+vtJL80ipySyrrXtc+F5Q5j/zSyrrXBWVVVNVok+cM\nEKeaJz4ylPioEFITY4iPDKFLVAjxUaHs27GV1NSBCCAiiECACAIEBIDgrBMRAkUIDBCCAoUAEYIC\nnOXAAOd1gPs5OiyYuMiQU/55iAhdokLpEhV6yufylCUMY4zP1biUg4Xl7qoXpwpmX24pWfllHCk+\nmhgqa1zHPUdUaBCxEcHEhAcTGxFMt5gwYsJDjq4Ld55jwoOJj3ISRFxECIEBctxzppfuZPyg7t4I\nuV2yhGGMaRWqyoHCctZlFrDtYBH7csvIzHeSQ3Z+WaOrgIToUBJjw+kVH8GQnrHERgYTFxFCXEQw\nsREhx7yOjQgmONCm9/E2SxjGGK84XFTB+qx8pxdQZgHrsgo4XFRRt71rp1B3XX4slw3qTk93b5+k\nuHB6xIYTFhzow9KbpljCMMacFJdLKa2qoaSi2v2ooaSymsKyKubuqOSdvStYn1XAfnf30ACBc86I\nYlyfrgxOimFQUgwp3aIJD7GE0N5YwjDGAFBV4yI7v4zdR0rZc6SE3TnO8/6CckorqymuqKG0sprS\nypoTnuesrsWM6t2ZQUmxDE6KYUD3aCJD7avGH9inaEwHoKoUVVST6+46eqS4gn15ZU5icCeIzLwy\nalxH2xEiQgLpFR9Jj5gwosKCiAgJIjIkkMjQICJD3c8hQe5nZzkrYzWXTRrvu0CNV1nCMMYPlFXW\nsGzXETZmF7rvJ6iou9cgt8R5NNXDqFNoEMldIhmUGMMVg3vQKz6C5C6R9IqPoGtUKCLH70HUlLwd\nJ7e/aV8sYRjTDrlcyqb9hSzelsPibYdZsTuvLiFEhgTSOSqEzpGhdIsJY2CPaDpHhRAf6ayLd79O\niosgLiL4pJOC6bgsYRjTTuwvKGPxthyWbMthyfYccksqAejfrRN3nd+LC/p0ZXivOGsvMF5jv1nG\ntEHlVTVsPVjE5v2FbMwu5L87jrD9UDEAXaJCGd+3K2P7dGHsOV04IzrMx6U1HYUlDGN8SFU5VFTB\npv2FbN5fyOb9TpLYebiY2vbniJBAhveK46YRPbmgbxf6JXSyaiTjE5YwjPGyiuoaDhSUk51fzv4C\n567m7IJy1mwv4+HF8+uqlgASY8NJ6d6JS1O7kdI9mpTu0fTqHEHACYavMKa1WMIw5jTJzi/jq80H\n2ZlTwv78crILysjOLyenuKLRvp0jQ4gJgkkpCaR070RK92j6d48mJrzxiKXGtBWWMIw5BTsPFzN3\n4wHmbTjA2swCwKlC6hEbTveYMFK6RTuvY8NIdK/rHhNOeEige27lwT6OwBjPWcIw5iSoOt1Z5204\nwNyNB9h60GmITusZy88m9+eSgQn07hJpbQzGL1nCMKYZNS5lzb585m7Yz9yNB9iXW0aAwMjenXny\nigFcPLAbPWLDfV1MY7zOEoYxDbhcSsaBIr7ZeYSlO4+wbOcRCsurCQ4Uxp7ThfsnnMNFKQnEt+LE\nNca0BZYwTIfncilbDxXxzQ53gtiVS36pM+Vlr/gILhvUnfPP6cL4fl2JDrNGadNxeTVhiMhk4Hkg\nEHhFVZ9qsD0OeBU4GygHvqeqG9zbfgxMAxRYD0xR1XJvltf4v7LKGvbllbL3SCm7j5Swck8ey3bl\n1nVt7dk5nIsHJDD6rHhGnxVvVU3G1OO1hCEigcALwCQgE1guIh+r6qZ6uz0GrFHVa0Skv3v/iSKS\nCDwIDFDVMhGZDdwMvO6t8hr/oKrsLyhnb24pe3NL2ed+OMtljbq4JsaGM6HfGZx3djyjz+pMUlyE\nj0puTNvnzSuMkcB2Vd0JICIzgauA+gljAPAUgKpmiEiyiCTUK1u4iFQBEUC2F8tq2jGXS1m9L4/P\n1zs9lzLzyuq2BQj0iA2nZ1wEE/ufwZnxEfTsHEHPuHDO7Bxh7RDGnARvJoxEYF+95UxgVIN91gLX\nAotFZCTQC0hS1ZUi8gywFygDvlDVL7xYVtPOVNe4+HZXLp9vOMC8jQc4VFRR1yg9fdxZ9O4SyZmd\nI+gRG25zPRtzmoiqNr9XS04scj0wWVWnuZfvAEap6v319onGaeMYitNO0R+4B9gDfADcBOQD7wHv\nq+pbTbzPdGA6QEJCwvCZM2e2qLzFxcVERUW16Ni2yN/iAcgvLGZvRRgrDtaw+mA1RVUQEgCDugYy\nPCGIIV0DiQhuP/c/+ONn5G8x+Vs80DimCRMmrFTVEZ4c680rjCygZ73lJPe6OqpaCEwBEOdOp13A\nTuASYJeqHnZv+xdwPtAoYajqS8BLACNGjNDx48e3qLDOXbctO7Yt8pd4qmtc/GfHET5ancXn60so\nq64gKjSICwf04NLUbnynX1ciQtpnZz9/+Yzq87eY/C0eOLWYvPmXthzoIyK9cRLFzcCt9XcQkVig\nVFUrcXpELVLVQhHZC4wWkQicKqmJwAovltW0IarKxuxCPlydxcdrszlcVEF0WBDDE4KYctEQxpzT\nhbDgQF8X05gOx2sJQ1WrReR+YB5Ot9pXVXWjiNzr3v4ikALMEBEFNgJT3duWicj7wCqgGliN+yrC\n+K/MvFI+WpPNv1dnse1QMcGBwoX9z+CaoYmM73cGS/+zmPEpCc2fyBjjFV69llfVOcCcButerPf6\nG6DvcY59AnjCm+UzvldQVsXn6/fz4eoslu3KBeDc5Dh+d00qlw/qTmxEiI9LaIyp1T4rf027Vl5V\nw4KMQ3yyNpuvMg5RWe3irC6R/GRSX64aksiZ8XYvhDFtkSUM0yqqalws2ZbDx2uz+WLjAUoqa+gS\nFcqtI8/kmqGJDE6KsRFejWnjLGEYr6lxKd/uyuXjtdl8vmE/+aVVRIcF8d3BPbhySA9GnxVPoM0k\nZ0y7YQnDnHYbsgr416osPlufzcHCCiJCApk0IIErBvdgXN+uhATZjXTGtEeWMMxps2znEf6yYDtL\ntucQEhjA+H5duSKtBxNTzmi390oYY46yv2JzSlSVJdtz+MtX2/l2dy5dokJ57LL+3HTumTY/dVtV\nfBi2zYOIeEgcAVFdW++9qyshZwsU7oegEAgMdZ6Dwhq8DoGgUGddgF2RthWWMEyLqCoLtxziz19t\nZ82+fLpFh/HEFQO4ZeSZdlNdW1RZAhlzYP1s2P4VaM3RbbFnOokjaYTz3D0NgsNO/T1Lc+HgBjiw\nHg64nw9ngKvq5M4TlQCxvSCuV+Pn6CQIbPA1pgqlRyBvD+Tvdj/vOfpcmA1d+0OfSdDnYkgcDgHt\n/He2ohhCvT+EiSUMc1JcLuWLTQf4y4LtbMwuJCkunN9dk8r1w5MIDWrnf3T+pqYadqXDutmw+VOo\nKnG+YMc8CAOvhcpiyFwBWStg37ew8V/OcQFBkJB6NIF06QsouKrBVeN+dr/Wess1VSTv+hKyX3SS\nQ2Hm0bJEJUC3QdDnIufcsb2cxFFd4TxqKpyrj+ryY19Xl0NhlvNlv28ZbPjXsclOAiEm0TlfSCTk\n73X2rSo59mcREe/s0z0N+lwC2ath8R9h0R8gPA7OvtBJHmdPbN0rrlPhqoFtX8CyF50rtvuWgZd7\nGlrCMB6pcSmfrsvmhYXb2XqwmN5dIvnD9YO5emiijQbblqg6X4brZsOGD6DkEITGwKDrYfCNcOb5\nx1bx9Dr/6OuiA0cTSOYKWDsTlr9yUm/fiwDo2hd6neckhm6DnEfUGacnvppqJxE1vGrI2wNleRCX\nDL2/0+Bq5EwI7dT4XKW5sHMhbJsP2790fl4I9BjiJI9zJh2bnE6Fq8a5slv5upMA+10KA66CM1JO\n/lxl+bDmbfj2JcjbDdGJcO5UqKl0qvG8yBKGadbm/YU88sE61mYW0OeMKJ6/eQjfHdzDusS2JlcN\nlBc4X4qluc5z3aPecvYaOLLNaQPoewkMutH58vOkiqlTN0j5rvOofc/DW5wv5IAgp9pGAo++brQu\niMXrdjFu4iXe+zkEBjlJIS751M8V0RlSr3MeLhccWOskj21fOFceX/+e84OjofBK6P9dOGvCyVfV\nFe6H1W/BqhlQsA8iu0LnsyD9KUj/P+jSz0kcA66ChIEnvkI4vAWW/cNJ5FUlcOZ5cNGTTtkCW6e9\n0BKGOa7yqhr+umA7L369g5jwYJ67aQhXpvUgwBLFsaoroPig8x968UEoL4SqUqfd4JjnUucPvbIU\nqkoZlncYMsIbV/PUX9Ya57/qymKc2YqbIhAW41StxPWC8+93voDC404troBASBjgPDzkCmyn85wF\nBECPoc7jOz91kvKOBeQtnkHCpo+dL/3gSKdKrf8VTvtHeGzT53LVwI6FsPI12PK58xmeNR4u/g30\nu9xp2C86CBmfwMZ/w+JnYNHTEH/O0eTRbbCTPFyuo9VOOxc6nQAG3QCjpjvVa63MEoZp0rKdR3j0\nX+vZmVPCdcOSePzyFOIiO+i4TgVZkLXSSQhF+93JYb/zR1+03/kP/0SCI5xHSITzpRPiLFcFx0BM\nQqP/0p1HwNHXEghh0U4CCO/sPEe4n8PjnGTR3htt25qIzjDoejYf6ULC2PNh9yKnHWjLHNj0EQQE\nQ+8LnP/u+10G0d2d34/Vb8LKN6BgL0R0cZL3sLsg/uxjz98pAc6d5jyKD0PGp7Dp37DkOadtJa63\n066yYwHk7YJOPeDCX8DwuyGyi09+JGAJwzRQWF7FU59n8M6yvSTFhfPm1JFc0KedNAKeLjVVsHep\n85/d9vlwqN6swhLoVN1EJTjVImeOdpY7dYOobs4XQVjM0cQQFH7cbqHr/XCuBb8UFALnXOQ8Ln/W\naePZ/InzJf/Zw87jjAGQs9W5Kuw9Dib9ykkmQR78kxXVFUZMcR4lR9zJ4yOnGitxBEz8JaRc0WrV\nTidiCcPUmbfxAL/8aAOHiyqYNrY3D1/ct33dcOdywZ4lsH8tdOruNAbGJDqvm/tjK8hyksO2L2Dn\n11BZ5PwX2es8mPQbSB4LMUnOf412X0DHFRAAPUc6j0m/droJb/4Udn0No3/oXAE0vJo4GZHxMPwu\n5+FytbnftXb0bWC85VBROU9+vJE56w/Qv1snXrpjBGk9j1M/2xYVZMGad2DNW06vkYYkwPnvPybJ\nSSDRiRDT0+m5s3+N09B5aKOzb3QSDLrO6SFz1nea7l1jDDhtDGekOI/v/PT0n7+NJQuwhNGh1biU\n2Sv28X9zNlNe7eKnl/Rj+riz2kc32epK2Po5rHoTdnwF6oLkC2DCz52+9CWHne6XBZlOQinMcnqp\n7F/n3MBWU+GcJyDI6W0y6ddOkjgjxet92Y1pryxhdECqyoKMQ/x+bgZbDxYzMrkz/3fdIM7u2g4m\nuz+U4TQsrp0JpTlOY+DYh2HobU53xVqR8XBG/6bPUXsncGG20w4RFt0qRTemvbOE0cGs3JPLU59n\nsHx3HsnxEbxw6zAuG9St9eeiqCiCFa/CrkVOW0FQqHvsIPdYQse8dhoOh656F9K3OFcF/S6FoXfC\nORNPvoeQiNPTxIe9TYxpjyxhdBDbDxXx9NwtfLHpIF2iQvnN1ancfG7P1q9+Kslx+pR/+5JzI9oZ\nA5wv/LphISqPHS7CVV13aFBEElz8Wxh8c/sZvsEYP2IJw88dKCjnuflbmb1iHxEhQfxkUl++N7Y3\nkaGt/NHn74Nv/gorZ0B1mdPl8IKHnYHfTsRV404elSxfuprx509onfIaYxqxhOGnSqqU38/N4NUl\nu3Cpctf5ydw/4Rzio7w71kwjh7fCf56DdbOc5UE3wtgfQdd+nh0fEOjcz0CENUYb42NeTRgiMhl4\nHggEXlHVpxpsjwNeBc4GyoHvqeoG97ZY4BUgFWdMhO+p6jfeLK+/+PfqLH6+qJTS6h1cPSSRhyf1\npWfniJadTNXperrqTWdQu8iu7hvVujs3qXXqfvSmtciuR4eazloFS551+qgHhTl3tJ53P8T2PH2B\nGmNaldcShogEAi8Ak4BMYLmIfKyq9W6b5TFgjapeIyL93ftPdG97HpirqteLSAjQwm+8jqO6xsX/\nzsng1f/som9cAH+643wG9ohp2clKc2H9e06iOLje+dJPOheKsp3EUXKYRmMbSYCTNMJinLtew2Jg\n3P+DUfdaA7MxfsCbVxgjge2quhNARGYCVwH1E8YA4CkAVc0QkWQRScC52hgH3O3eVglUerGs7d6R\n4grue2cVS3fmMmVMMmMiD518snC5nDtWV7/pXBnUVDgDnF32jDM8dv3B7GqqoPgQFB9wj7F04OhY\nSyU5MPR2GD7Fuqwa40e8mTASgX31ljOBUQ32WQtcCywWkZFALyAJqAEOA6+JSBqwEnhIVRvMimIA\n1mcW8P03V3CkpJJnb0zj2mFJpKcf9vwEBZmw+m3nTun8vRAW6wxNMPQO6D646WMCg527pmMST08Q\nxpg2T1SPN2TyKZ5Y5HpgsqpOcy/fAYxS1fvr7RONU/U0FFgP9AfuwUlkS4ExqrpMRJ4HClX1F028\nz3RgOkBCQsLwmTNntqi8xcXFREW1gxvXGvhPVhWvb6wkOkR4YGgoyTHOPQmexNOpcCvJu2fROXcl\ngpIXO5j93SeR02U0rsC2NzJte/2Mjsff4gH/i8nf4oHGMU2YMGGlqo7w6GBV9coDOA+YV2/5UeDR\nE+wvwG4gGugG7K637QLgs+bec/jw4dpSCxcubPGxvlBZXaNPfLRBe/3sU735H99oTlH5MdtPGM/+\ndarv3Kz6RLTq73urfvVb1dxdXi3v6dDePqPm+Fs8qv4Xk7/Fo9o4JmCFevi97s0qqeVAHxHpDWQB\nNwO31t/B3ROqVJ02imnAIlUtBApFZJ+I9FPVLTgN4ZswAOQUV3Df26tYtiuXqWN78+il/Qny5Aa8\nw1ucWb42fuhM23nh406DtA2wZ4zxgNcShqpWi8j9wDycbrWvqupGEbnXvf1FIAWYISIKbASm1jvF\nA8Db7h5SO4Ep3ipre7IuM5/vv7mS3JJK/nRTGtcMTWr+oNyd8PXTzr0QwREw7qdw3n2nPiObMaZD\n8ep9GKo6B5jTYN2L9V5/A/Q9zrFrAM/q1TqID1Zm8uiH6+kaFcoHPzif1MRmekEVZDpzE69+yxl/\n6bz7YMyPrIurMaZF7E7vdkBV+Vv6Dv4wbwvnnRXPC7cNo/PxpktVhZxtnLPtZVj8hbNuxFRnGI5O\n3Vqv0MYYv2MJo41zuZTffraZV/+zi2uGJvL09YMbDxhYUwV7/gtb5zqTzuftIpEAGHaHU/1kd1cb\nY04DSxhtWFWNi/95fx0frs5iyphkfnH5AAIC3OMpleY6U4pu+Ry2fwUVBc5w4L3HwXn3sTQ3jvMm\nX+/bAIwxfsUSRhtVVlnDfe+sYkHGIf7fxX25b8I5SP4e2PSxcyWxdylojTMUx4AroO+lcNZ4CHX6\nV1ekp/uy+MYYP2QJow0qKKti2ozlrNiTx2+vTuX2UWfC0r/Dl78EVxUkpMLYHzuTCPUY1ibn/jXG\n+B9LGG3MocJy7nz1W3YcLuavtwzj8nNCYeZtsOUz6HcZTH4K4nr5upjGmA7IEkYbsudICXf881ty\niit47e6RjA3bBf+Y4gzqN/kp5yY7mxPCGOMjljDaiE3Zhdz56rfUuFy8M20kQ/a9CV/9GqITYeq8\n5memM8YYL7OE0QZ8uyuXqTOWExUaxOw7+nHW4u/Dti9gwFVw5V+ceSWMMcbHLGH42Dc7jnD3a9+S\nGBfOrMlK1/cnQ+kRuPyPzg13VgVljGkjLGH4UGF5FQ/PXkPP2FA+TltKxPu/h7jeMG3+8eehMMYY\nH7GE4UO//XQT1UWH+TD5LSKWLIbU6+GK52z0WGNMm2QJw0cWZhzi0xXbWRD/Jzod2g1X/BmG3WlV\nUMaYNssShg8UlFXx8w9W83rUCySUboVbZkLfS3xdLGOMOSFLGD7wm0828lD53xkZuBKueN6ShTGm\nXbAxJVrZgoyD9Fj7F24KXAjj/geG3+3rIhljjEcsYbSigtIqlrz3PA8Hv0/N4FtgwmO+LpIxxnjM\nEkYrmjXzVR6r/jtFieMIvOov1sBtjGlXPEoYIvIvEblcRCzBtNCy/yzgtj2/IDfqHDrd+Q4EBvu6\nSMYYc1I8TQB/A24FtonIUyLSz4tl8juF+3dwzpdTKA6IJnbaR3afhTGmXfIoYajqfFW9DRgG7Abm\ni8h/RWSKiNi/yidSmkvZa1cTqFUUXjeTkLgevi6RMca0iMdVTCISD9wNTANWA8/jJJAvT3DMZBHZ\nIiLbReSRJrbHiciHIrJORL4VkdQG2wNFZLWIfOppOduUqjLy/3kdsRX7mTv4OfqkjvB1iYwxpsU8\nbcP4EFgMRABXqOqVqjpLVR8Aoo5zTCDwAnApMAC4RUQGNNjtMWCNqg4G7sRJQvU9BGz2NJg2xVVD\n5XtTiT6ymmeifsK1V9n82saY9s3TK4w/q+oAVf0/Vd1ff4OqHu/f5pHAdlXdqaqVwEzgqgb7DAAW\nuM+TASSLSAKAiCQBlwOveFjGtkMV5j1GyNbP+N/q27n6th8SEmT9BYwx7Zun32IDRCS2dsFdlfTD\nZo5JBPbVW850r6tvLXCt+5wjgV5Aknvbc8D/AC4Py9h2/Oc5WPYir1RfStT4BxnYw+azMMa0f6Kq\nze8kskZVhzRYt1pVh57gmOuByao6zb18BzBKVe+vt080TjXUUGA90B+4BydpXKaqPxSR8cD/U9Xv\nHud9pgPTARISEobPnDmz2XiaUlxcTFRUk7VrJ6Xb/vn03/IXPtMxPB18H784P4KggNa/3+J0xdOW\n+FtM/hYP+F9M/hYPNI5pwoQJK09QU3QsVW32gfNlLvWWA4GNzRxzHjCv3vKjwKMn2F9wemBFA/+H\nc0WyGzgAlAJvNVfO4cOHa0stXLiwxcfWyZij+mScZv1lsp7zs3/rf7fnnPo5W+i0xNPG+FtM/haP\nqv/F5G/xqDaOCVihHuQBVfW4SmoeMEtEJorIROBdYG4zxywH+ohIbxEJAW4GPq6/g4jEureB0/tq\nkaoWquqjqpqkqsnu4xao6u0eltU39i6D9+5Gu6cxtexBUpLiGX1WZ1+XyhhjThtPR6v9H5xqnx+4\nl7+kmcZoVa0Wkftxkk0g8KqqbhSRe93bXwRSgBkiosBGYOrJh9AGHNoM79wI0YksGP4Cm9/bxV9v\nPQuxoT+MMX6k2YTh7h77hjo37r14MidX1TnAnAbrXqz3+hugbzPnSAfST+Z9W1VBJrx1HQSFoXf8\ni7++m0XPzuFMHtjN1yUzxpjTqtkqKVWtAXrVqzoytUpz4c1roaIYbv+AFQXRrN6bz7SxZxEUaN1o\njTH+xdMqqZ3Af0TkY6CkdqWqPuuVUrUHlSVONVTebrjjQ+iWyj9mrCA2IpgbRiQ1e7gxxrQ3niaM\nHe5HAGAj59VUwXt3Q9ZKuPFNSB7D9kPFzN98kAcvPIeIEJvI0Bjjfzz6ZlPVX3m7IO2GKnz8AGz7\nwpleNcW5PeSVxTsJDQrgzvOTfVs+Y4zxEo8Shoh0xekpNRAIq12vqhd6qVxt1/wnYO27MOHxuulV\nDxWV869VWVw/IokuUaG+LZ8xxniJpy2zbwMZQG/gVzg31C33Upnarg0fwH+eh3PvgXH/r271jP/u\npsrl4p4LzvJh4Ywxxrs8TRjxqvpPoEpVv1bV7wEd7+pizTsQ1xsu/X3d9KolFdW8tXQvlwzoRu8u\nkT4uoDHGeI+nCaPK/bzfPVXrUKBj3cZcXgA7v4aUKyAgsG71rOX7KCirYvp37OrCGOPfPO3O81sR\niQF+AvxlL58aAAAgAElEQVQFZ7ynH3utVG3Rti/BVQX9j46BWF3j4p9LdnFuchzDzozzYeGMMcb7\nPO0lVTvjXQEwwXvFacMyPoXIMyDp3LpVn63fT1Z+GU9eOdCHBTPGmNbhaS+p14BG46C72zL8X1W5\nc4Ux6HoIcGrxVJWXFu3k7K6RTOx/ho8LaIwx3udplVT9ObXDgGuA7NNfnDZq1yKoLIb+V9St+u+O\nI2zMLuSpawcR4IP5LowxprV5WiX1Qf1lEXkXWOKVErVFGZ9ASCfofUHdqn8s2kmXqFCuHtpwEkFj\njPFPLR0hrw/QMephXDWw5XPoezEEOTflbd5fyKKth5kyJpmw4MBmTmCMMf7B0zaMIo5twzgA/Mwr\nJWpr9n0LJYeh/+V1q15etJOIkEBuH9XLhwUzxpjW5WmVVMcdcDDjUwgMgXMmAZCdX8bHa7O547xe\nxEQE+7hwxhjTejyqkhKRa9z3YdQux4rI1d4rVhuh6iSM3t+BsGgAXvvPLhSYOra3b8tmjDGtzNM2\njCdUtaB2QVXzgSe8U6Q25NAmZ74L94i0heVVvPvtPi4f1J2kuAjfls0YY1qZpwmjqf38f9KHzZ8C\nAv0uA+CjNdkUV1QzfZwNA2KM6Xg8TRgrRORZETnb/XgWWOnNgrUJGZ9Az1EQ5XQI23GomKjQIAb2\niPZxwYwxpvV5mjAeACqBWcBMoBy4z1uFahPy9sCB9cf0jsrKLyMxNhwRu1HPGNPxeNpLqgR45GRP\nLiKTgeeBQOAVVX2qwfY44FXgbJwk9D1V3SAiPYE3gASc7rwvqerzJ/v+pyTjM+e5XsLIzi+jR2zY\ncQ4wxhj/5mkvqS9FJLbecpyIzGvmmEDgBeBSYABwi4gMaLDbY8AaVR0M3ImTXACqgZ+o6gBgNHBf\nE8d6V8ZncMYAiD+7blVWfhmJceGtWgxjjGkrPK2S6uLuGQWAqubR/J3eI4HtqrpTVStxqrKuarDP\nAGCB+5wZQLKIJKjqflVd5V5fBGwGWm8MjpIc2PvfY4YyL6moJr+0ih6xljCMMR2TpwnDJSJn1i6I\nSDJNjF7bQCKwr95yJo2/9NcC17rPORLoBSTV38H9XkOBZR6W9dRtnQvqalQdBZBoCcMY00F52jX2\n58ASEfkaEOACYPppeP+ngOdFZA2wHlgN1NRuFJEo4APgR6pa2NQJRGR6bVkSEhJIT09vUUGKi4vr\njk1d/zpRoV1ZuiUPtjrr1h2uBuDQrgzS87e16D1aU/14/IW/xeRv8YD/xeRv8cApxqSqHj1wqqAe\nBy4HrgfGNbP/ecC8esuPAo+eYH8BdgPR7uVgYB7wsKdlHD58uLbUwoULnRflRaq/7qo653+O2f7W\n0t3a62efalZeaYvfozXVxeNH/C0mf4tH1f9i8rd4VBvHBKxQD79jPR18cBrwEE510RqchuhvgAtP\ncNhyoI+I9AaygJuBWxucNxYoVaeNYxqwSFULxem3+k9gs6o+60kZT5sdX0FNxTHtF+BUSQUGCAnR\n1kvKGNMxedqG8RBwLrBHVSfgtCnkn+gAVa0G7se5StgMzFbVjSJyr4jc694tBdggIltwelM95F4/\nBrgDuFBE1rgfl51MYC22+VMI7wxnnnfM6qy8MrpFhxFokyUZYzooT9swylW1XEQQkVBVzRCRfs0d\npKpzgDkN1r1Y7/U3QN8mjluCU0XVumqqYOs8Z+yowGN/NNn55dal1hjToXmaMDLd1Uf/Br4UkTxg\nj/eK5SO7F0NFwTG9o2pl5ZcxsndnHxTKGGPaBk/v9L7G/fJJEVkIxABzvVYqX8n4DIIj4Oxjm2aq\na1wcKCy3LrXGmA7tpEecVdWvvVEQn1OXkzDOvhCCj00Mh4oqqHGp3bRnjOnQWjqnt9/pVLQdivZD\nyhWNtmW5b9qzcaSMMR2ZJQy3LjlLQQKhz8WNttXe5Z1kjd7GmA7MEoZbl5xlkDwWIho3bGfm1V5h\nWMIwxnRcljAADm8lsjSz0c16tbLzy4iLCCYixP8nGTTGmOOxhAGQ8anz3L/pewOz8svs6sIY0+FZ\nwgDI+JTCTudATFKTm7PdM+0ZY0xHZgmjshRKj5DTZXSTm1WVrDy7wjDGGEsYIRHw4Br29by6yc2F\nZdWUVNZYDyljTIdnCQNABA0IbnLT0XswLGEYYzo2SxjNyLKZ9owxBrCE0axsu8IwxhjAEkazsvLL\nCAkKID4yxNdFMcYYn7KE0Ywsd5faAJs4yRjTwVnCaEZ2fpkNOmiMMVjCaFZWnt20Z4wxYAnjhCqq\nazhUVGEN3sYYgyWMEzpQUA5Yl1pjjAFLGCdk92AYY8xRXk0YIjJZRLaIyHYReaSJ7XEi8qGIrBOR\nb0Uk1dNjW0OWex6MRBsWxBhjvJcwRCQQeAG4FBgA3CIiAxrs9hiwRlUHA3cCz5/EsV6Xne9USXWL\nsV5SxhjjzSuMkcB2Vd2pqpXATOCqBvsMABYAqGoGkCwiCR4e63VZ+aV07RRKaFBga7+1Mca0Od5M\nGInAvnrLme519a0FrgUQkZFALyDJw2O9Lju/3NovjDHGzddzjj4FPC8ia4D1wGqg5mROICLTgekA\nCQkJpKent6ggxcXFjY7dnl1Kz+iAFp/Tl5qKp73zt5j8LR7wv5j8LR44tZi8mTCygJ71lpPc6+qo\naiEwBUBEBNgF7ATCmzu23jleAl4CGDFihI4fP75FhU1PT6f+sapK3vy5XNm3F+PHp7TonL7UMB5/\n4G8x+Vs84H8x+Vs8cGoxebNKajnQR0R6i0gIcDPwcf0dRCTWvQ1gGrDInUSaPdbbjpRUUlHtooc1\neBtjDODFKwxVrRaR+4F5QCDwqqpuFJF73dtfBFKAGSKiwEZg6omO9VZZm3K0S21Ea76tMca0WV5t\nw1DVOcCcButerPf6G6Cvp8e2pqPzYNgVhjHGgN3pfVy1d3knxdoVhjHGgCWM48rKLyMyJJDocF93\nJDPGmLbBEsZxZOWV0SM2HKfzljHGGEsYx5FdUGZjSBljTD2WMI4jO7/c5sEwxph6LGE0obSymtyS\nShsWxBhj6rGE0YTaUWotYRhjzFGWMJpQN3GStWEYY0wdSxhNOHrTniUMY4ypZQmjCVl5ZQQGCAmd\nQn1dFGOMaTMsYTQhO7+MbtFhBAXaj8cYY2rZN2ITMvPLrMHbGGMasITRhOz8Mht00BhjGrCE0UCN\nSzlQYDftGWNMQ5YwGjhUVE61S61LrTHGNGAJowHrUmuMMU2zhNFAZl7tPBiWMIwxpj5LGA3UDgti\nVxjGGHMsSxgNZOWXEhsRTGSoTZxkjDH12bdiA9n55fSIsasL0/5VVVWRmZlJeXl5q71nTEwMmzdv\nbrX38zZ/iicsLIykpKRTOocljAay8so4M97m8TbtX2ZmJp06dSI5ObnVZo4sKiqiU6dOrfJercFf\n4lFVjhw5QmZm5imdx6tVUiIyWUS2iMh2EXmkie0xIvKJiKwVkY0iMqXeth+7120QkXdFpFXupMu2\nu7yNnygvLyc+Pt6mGTaICPHx8ad8tem1hCEigcALwKXAAOAWERnQYLf7gE2qmgaMB/4oIiEikgg8\nCIxQ1VQgELjZW2WtVVheRVFFtd3lbfyGJQtT63T8LnjzCmMksF1Vd6pqJTATuKrBPgp0EieSKCAX\nqHZvCwLCRSQIiACyvVhWwKmOAkiMtSopY05Vfn4+f/vb31p07GWXXUZ+fv4J9/nlL3/J/PnzW3R+\n0zLeTBiJwL56y5nudfX9FUjBSQbrgYdU1aWqWcAzwF5gP1Cgql94saxA/Zv27ArDmFN1ooRRXV3d\n5Ppac+bMITY29oT7/PrXv+aiiy5qcfl8obm42zpfN3pfAqwBLgTOBr4UkcU4VVBXAb2BfOA9Ebld\nVd9qeAIRmQ5MB0hISCA9Pb1FBSkuLmbpnnUA7Nm8moKd7bvHcXFxcYt/Fm2Vv8Xk7XhiYmIoKiry\n2vmbUlNTU/eeP/nJT9ixYweDBw9mwoQJXHLJJfz2t78lNjaWrVu3snr1am655RaysrIoLy/nBz/4\nAVOmOM2YqampfP311xQXF3Pddddx3nnnsWzZMrp3787MmTMJDw/n3nvvZfLkyVx99dWkpqZyyy23\nMHfuXKqqqnjjjTfo27cvOTk5TJ06lf379zNy5EgWLlzIokWLiI+PP6bcP/7xj1m1ahVlZWVcddVV\n/PznPwdg+fLlPProo5SWlhISEsInn3xCRERE3dVNQEAAd911F/fee29dmePj41m1ahWPP/44c+bM\n4X//93/ZtWsXu3fvJikpiSeffJLp06dTWloKwDPPPMOoUaMA+NOf/sSsWbMICAhg0qRJ3HXXXdx1\n110sXrwYgO3btzNlypS65ZNVXl5+Sr933kwYWUDPestJ7nX1TQGeUlUFtovILqA/0AvYpaqHAUTk\nX8D5QKOEoaovAS8BjBgxQsePH9+iwqanpxMZmEDI1t1cOWkCAQHtu+43PT2dlv4s2ip/i8nb8Wze\nvLmuh8+vPtnIpuzC03r+AT2ieeKKgcesq9+r6I9//CNbtmxh3TrnH7H09HTWrl3Lhg0b6N27NwBv\nvPEGnTt3pqysjHPPPZfbbrutrqE+KioKgB07djBr1iyGDBnCjTfeyBdffMHtt99OcHAw4eHhdOrU\nCREhMTGRNWvW8Le//Y2///3vvPLKKzz66KNMmjSJRx99lLlz5/LGG28QFRXVqOfT008/TefOnamp\nqWHixIns2rWL/v37M3XqVGbPns25555LYWEhERERvPzyy2RnZ7Nu3TqCgoLIzc2tK0PtuSMjIwkM\nDKRTp06Ehoaybds2lixZQnh4OKWlpSxYsICwsDC2bdvGLbfcwooVK/j888+ZO3cuy5cvJyIigtzc\nXDp37kxcXBw7duxgyJAhvPfee0ydOrXFPbfCwsKIiopq8e+dN/+NXg70EZHeIhKC02j9cYN99gIT\nAUQkAegH7HSvHy0iEe72jYmA1ztDZ+U5w5q392RhTFs1cuTIumQB8Oc//5m0tDRGjx7Nvn372LZt\nW6NjevfuzZAhQwAYPnw4u3fvbvLc1157baN9lixZws03O/1lJk+eTFxcXJPHzp49m2HDhjF06FA2\nbtzIpk2b2LJlCwkJCZx77rkAREdHExQUxPz58/n+979PUJDz/3bnzp2bjfvKK68kPNzpfVlVVcU9\n99zDoEGDuOGGG9i0aRMA8+fPZ8qUKURERBxz3mnTpvHaa69RU1PDrFmzuPXWW5t9P2/x2hWGqlaL\nyP3APJwqpldVdaOI3Ove/iLwG+B1EVkPCPAzVc0BckTkfWAVTiP4atxXEd7kzINhXWqN/2l4JeAr\nkZGRda/T09OZP38+33zzDREREYwfP77Jbp+hoUenSg4MDKSsrKzJc9fuFxgYeFJtBbt27eKZZ55h\n+fLlxMXFcffdd7eo+2lQUBAulwug0fH14/7Tn/5EQkICa9euxeVyERZ24jbT6667jl/96ldceOGF\nDB8+vFF1WmvyakW9qs5R1b6qeraq/s697kV3skBVs1X1YlUdpKqp9dsoVPUJVe3vXn+HqlZ4s6wA\nWXYPhjGnTadOnU7YhlJQUEBcXBwRERFkZGSwdOnS016GMWPGMHv2bAC++OIL8vLyGu1TWFhIZGQk\nMTExHDx4kM8//xyAfv36cfDgQZYvXw441W3V1dVMmjSJf/zjH3VJKTc3F4Dk5GRWrlwJwAcffHDc\nMhUUFNC9e3cCAgJ48803qampAWDSpEm89tprdW0btecNCwvjkksuOaaNx1fad8vuaVTtUg4VVdgV\nhjGnSXx8PGPGjCE1NZWf/vSnjbZPnjyZ6upqUlJSeOSRRxg9evRpL8MTTzzBF198QWpqKu+99x7d\nunVrVP+flpbG0KFD6d+/P7feeitjxowBICQkhNdee40HHniAtLQ0Jk2aRHl5OdOmTePMM89k8ODB\npKWl8c4779S910MPPcSIESMIDAw8bpl++MMfMmPGDNLS0sjIyKi7+pg8eTJXXnklI0aMYMiQITzz\nzDN1x9x2220EBARw8cUXn+4f0clRVb95DB8+XFtq9mdfaa+ffaqzvt3b4nO0JQsXLvR1EU47f4vJ\n2/Fs2rTJq+dvSmFhYau/54mUl5drVVWVqqr+97//1bS0tJM6vq3E84c//EEff/zxUz7Ppk2bGv3e\nASvUw+9YX3erbTNyyhTAZtozxo/s3buXG2+8EZfLRUhICC+//LKvi3TSrrnmGnbs2MGCBQt8XRRL\nGLVyy53GKquSMsZ/9OnTh9WrV/u6GKfkww8/9HUR6lgbhlvtFUb3GLvL2xhjmmIJw+1IudIlKpSw\n4OM3VhljTEdmCcPtSJnL2i+MMeYELGG4HSlXEm3QQWOMOS5LGDhdi3PL1G7aM8bHasePys7O5vrr\nr29yn/Hjx7NixYoTnue5556ruwEOPBsu3TTPEgaQW1JJpct6SBnTVvTo0YP333+/xcc3TBieDJfe\nlqhq3TAjbYklDCA73xn3xa4wjDl9HnnkEV544YW65SeffJJnnnmG4uJiJk6cyLBhwxg0aBAfffRR\no2N3795NamoqAGVlZdx8882kpKRwzTXXHDOW1A9+8ANGjBjBwIEDeeKJJwBnQMPs7GwmTJjAhAkT\nAGfYjpycHACeffZZUlNTSU1N5bnnnqt7v5SUFO655x4GDhzIxRdf3OSYVZ988gmjRo1i6NChXHTR\nRRw8eBBwhqqfMmUKgwYNYvDgwXVDg8ydO5dhw4aRlpbGxIkTj/k51EpNTWX37t3s3r2bfv36ceed\nd5Kamsq+ffuajA+cYdfPP/980tLSGDlyJEVFRYwbN441a9bU7TN27FjWrl3r8eflCbsPA8jKd/4T\nsSsM47c+fwQOrD+95+w2CC596ribb7rpJn70ox9x3333Ac6IsPPmzSMsLIwPP/yQ6OhocnJyGD16\nNFdeeeVxpxD9+9//TkREBJs3b2bdunUMGzasbtvvfve7Y4YlX7duHQ8++CDPPvssCxcupEuXLsec\na+XKlbz22mssW7YMVWXUqFF85zvfIS4ujm3btvHuu+/y8ssvc+ONN/LBBx9w1VXHThI6duxYli5d\niojwyiuv8PTTT/PHP/6R3/zmN8TExLB+vfMzzsvL4/Dhw9xzzz0sWrSI3r17140NdSLbtm1jxowZ\ndcOkNBVf//79uemmm5g1a1bdsOvh4eFMnTqV119/neeee46tW7dSXl5OWlpas+95MuwKA8iyKwxj\nTruhQ4dy6NAhsrOzWbt2LXFxcfTs2RNV5bHHHmPw4MFcdNFFZGVl1f2n3pRFixZx++23AzB48GAG\nDx5ct62pYclPZMmSJVxzzTVERkYSFRXFtddeWzcZkSfDqGdmZnLJJZcwaNAg/vCHP7Bx40bAGZq8\nNjECxMXFsXTpUsaNG1c3nLsnw6D36tXrmDG1jjfsevfu3RsNu37DDTfw6aefUlVVxauvvsrdd9/d\n7PudLLvCwJkHIzQQYiOCfV0UY7zjBFcC3nTDDTfw/vvvc+DAAW666SYA3n77bQ4fPszKlSsJDg4m\nOTm5RcOJn65hyWt5Moz6Aw88wMMPP8yVV15Jeno6Tz755Em/T/1h0OHYodDrD4N+svFFREQwadIk\nPvroI2bPnl03cu7pZFcYOPNgxIfJcS+JjTEtc9NNNzFz5kzef/99brjhBsAZ3vuMM84gODiYhQsX\nsmfPnhOeY9y4cXUjwm7YsKFuBr/jDUsOxx9a/YILLuDf//43paWllJSU8OGHH3LBBRd4HE9BQQGJ\niYkAzJgxo279pEmTjmmvycvLY/To0SxatIhdu3YBxw6DvmrVKgBWrVpVt72hEw27vn///kbDroMz\n2dKDDz7Iueeee9zJok6FJQyceTDiw+1HYczpNnDgQIqKikhMTKR79+6AM1T3ihUrGDRoEG+88Qb9\n+/c/4Tl+8IMfUFxcTEpKCr/85S8ZPnw4cPxhyQGmT5/O5MmT6xq9aw0bNoy7776bkSNHMmrUKKZN\nm8bQoUM9jufJJ5/khhtuYPjw4ce0jzz++OPk5eWRmppKWloaCxcupGvXrrz00ktce+21pKWl1V1h\nXXfddeTm5jJw4ED++te/0rdv3ybf60TDrs+aNavRsOvgVKVFR0d7bd4McUa39Q8jRozQ5vpnN2X4\nb75kUJyL1++/xAul8g1/m/8a/C+m1pjTOyUlxWvnb0r9Ob39QXuLJzs7m/Hjx5ORkUFAQON/gjdv\n3szBgweP+b0TkZWqOsKT83f4f6tdLmVc36707WxjSBlj2q833niDUaNG8bvf/a7JZHE6dPiEERAg\n/OmmIZzfw9r/jTHt15133sm+ffvq2oq8ocMnDGOMMZ6xhGGMH/OnNkpzak7H74JXE4aITBaRLSKy\nXUQeaWJ7jIh8IiJrRWSjiEypty1WRN4XkQwR2Swi53mzrMb4m7CwMI4cOWJJw6CqHDlyhLCwUxuR\n22sV9yISCLwATAIygeUi8rGq1r8V8z5gk6peISJdgS0i8raqVgLPA3NV9XoRCQEivFVWY/xRUlIS\nmZmZHD58uNXes7y8/JS/lNoSf4onLCyMpKSkZu97ORFvtvSOBLar6k4AEZkJXAXUTxgKdBLnjrko\nIBeoFpEYYBxwN4A7gVR6sazG+J3g4OC6YSlaS3p6+knd19DW+Vs8p8qbVVKJwL56y5nudfX9FUgB\nsoH1wEOq6gJ6A4eB10RktYi8IiKRGGOM8Rmv3bgnItcDk1V1mnv5DmCUqt7fYJ8xwMPA2cCXQBrQ\nF1gKjFHVZSLyPFCoqr9o4n2mA9MBEhIShs+cObNF5S0uLq6bvMUf+Fs84H8x+Vs84H8x+Vs80Dim\nCRMmeHzjHqrqlQdwHjCv3vKjwKMN9vkMuKDe8gKcqqxuwO566y8APmvuPYcPH64ttXDhwhYf2xb5\nWzyq/heTv8Wj6n8x+Vs8qo1jAlaoh9/r3mzDWA70EZHeQBZwM3Brg332AhOBxSKSAPQDdqpqjojs\nE5F+qrrFvc+Jxy0GVq5cmSMiLW3R6QLktPDYtsjf4gH/i8nf4gH/i8nf4oHGMfXy9ECvjiUlIpcB\nzwGBwKuq+jsRuRdAVV8UkR7A60B3QICnVPUt97FDgFeAEGAnMEVV87xY1hXq6WVZO+Bv8YD/xeRv\n8YD/xeRv8cCpxeTV8TBUdQ4wp8G6F+u9zgYuPs6xawC/+qCMMaY9szu9jTHGeMQSxlEv+boAp5m/\nxQP+F5O/xQP+F5O/xQOnEJNfzYdhjDHGe+wKwxhjjEc6fMJoboDE9khEdovIehFZIyInPwWhj4nI\nqyJySEQ21FvXWUS+FJFt7ufTP2GxFx0npidFJMv9Oa1x9ypsF0Skp4gsFJFN7oFDH3Kvb7ef0wli\napefk4iEici39QZ3/ZV7fYs/ow5dJeUeIHEr9QZIBG7RYwdIbHdEZDcwQlXbZf9xERkHFANvqGqq\ne93TQK6qPuVO7HGq+jNflvNkHCemJ4FiVX3Gl2VrCRHpDnRX1VUi0glYCVyNM/5bu/ycThDTjbTD\nz8k9Rl+kqhaLSDCwBHgIuJYWfkYd/QqjboBEdQY4rB0g0fiQqi7CGYiyvquAGe7XM3D+kNuN48TU\nbqnqflVd5X5dBGzGGSuu3X5OJ4ipXXLfyF3sXgx2P5RT+Iw6esLwZIDE9kiB+SKy0j3Wlj9IUNX9\n7tcHgARfFuY0ekBE1rmrrNpN9U19IpIMDAWW4SefU4OYoJ1+TiISKCJrgEPAl6p6Sp9RR08Y/mqs\nqg4BLgXuc1eH+A33+Df+UJf6d+AsYAiwH/ijb4tz8kQkCvgA+JGqFtbf1l4/pyZiarefk6rWuL8L\nkoCRIpLaYPtJfUYdPWFkAT3rLSe517Vrqprlfj4EfIhT9dbeHXTXMdfWNR/ycXlOmaoedP9Bu4CX\naWefk7te/APgbVX9l3t1u/6cmoqpvX9OAKqaDywEJnMKn1FHTxh1AyS6Z/W7GfjYx2U6JSIS6W6w\nwz2HyMXAhhMf1S58DNzlfn0X8JEPy3Ja1P7Rul1DO/qc3A2q/wQ2q+qz9Ta128/peDG1189JRLqK\nSKz7dThO554MTuEz6tC9pKDpARJ9XKRTIiJn4VxVgDNW2DvtLSYReRcYjzOq5kHgCeDfwGzgTGAP\ncKOqtptG5OPENB6nmkOB3cD369Utt2kiMhZYjDPxmcu9+jGcOv92+TmdIKZbaIefk4gMxmnUDsS5\nOJitqr8WkXha+Bl1+IRhjDHGMx29SsoYY4yHLGEYY4zxiCUMY4wxHrGEYYwxxiOWMIwxxnjEEoYx\nPiQi40XkU1+XwxhPWMIwxhjjEUsYxnhARG53zy2wRkT+4R7UrVhE/uSea+ArEenq3neIiCx1D1b3\nYe1gdSJyjojMd89PsEpEznafPkpE3heRDBF5233HMSLylHtuhnUi0q6G1jb+yRKGMc0QkRTgJmCM\neyC3GuA2IBJYoaoDga9x7t4GeAP4maoOxrlruHb928ALqpoGnI8zkB04o6L+CBiAM8jdGPfduNcA\nA93n+a13ozSmeZYwjGneRGA4sNw9VPREnC92FzDLvc9bwFgRiQFiVfVr9/oZwDj3+F6JqvohgKqW\nq2qpe59vVTXTPbjdGiAZKADKgX+KyLVA7b7G+IwlDGOaJ8AMVR3ifvRT1Seb2K+l4+xU1HtdAwSp\najXOqKjvA98F5rbw3MacNpYwjGneV8D1InIG1M2J3Avn7+d69z63AktUtQDIE5EL3OvvAL52z+CW\nKSJXu88RKiIRx3tD95wMMao6B/gxkOaNwIw5GUG+LoAxbZ2qbhKRx4EvRCQAqALuA0pwJqV5HGdO\ngZvch9wFvOhOCDuBKe71/7+dOzZCGAaCAPiXOTDlEboJSnAXLoPeyOXAjpmHYXCyW8BLim7uAy1V\ntSVZzxn3N8fequqZZKqj4Tx+/Cz4mN9q4UtJXmOM+ep7wL9YSQHQomEA0KJhANAiMABoERgAtAgM\nAOUekeIAAAAOSURBVFoEBgAtAgOAlh25QIc5rzQ8fwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x39196f1588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tr_accuracy = hist.history['acc']\n",
    "val_accuracy = hist.history['val_acc']\n",
    "\n",
    "plt.plot(tr_accuracy)\n",
    "plt.plot(val_accuracy)\n",
    "plt.grid()\n",
    "plt.xlabel('epochs')\n",
    "plt.ylabel('accuarcy')\n",
    "plt.legend(['training accuracy', 'validation accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and Loading the model\n",
    "\n",
    "Since the training takes a long time, it is useful to save the results.  See the [keras page](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) for many more useful saving and loading features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save(\"mnist_mod.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now reload the model with the `load_model` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "hidden (Dense)               (None, 100)               78500     \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 79,510\n",
      "Trainable params: 79,510\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "model = load_model(\"mnist_mod.h5\")\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the performance on the test data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 60us/step\n",
      "accuracy = 0.970600\n"
     ]
    }
   ],
   "source": [
    "score, acc = model.evaluate(Xts, yts, verbose=1)\n",
    "print(\"accuracy = %f\" % acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}