analyze-a-b-test-results.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "85cdd68d",
   "metadata": {
    "papermill": {
     "duration": 0.013178,
     "end_time": "2024-11-18T02:41:52.168765",
     "exception": false,
     "start_time": "2024-11-18T02:41:52.155587",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## **Analyze A/B Test Results**\n",
    "\n",
    "A company recently launched a new webpage design aimed at improving user engagement and increasing conversions. To evaluate its effectiveness, the company decided to conduct an A/B test to compare the new webpage with the current version. The goal is to determine whether the new design leads to a statistically significant increase in conversions.\n",
    "In this A/B test, the audience was split randomly into two groups of equal size:\n",
    "+ **Control Group**: Users who interacted with the current webpage.\n",
    "+ **Test Group**: Users who interacted with the new webpage.\n",
    "\n",
    "The company expects an analysis of the results to determine whether the new webpage design should be adopted.\n",
    "\n",
    "### **Data Overview**\n",
    "The dataset used for this analysis contains 69,889 rows, where each row represents a user interaction. Key attributes include:\n",
    "+ **Country**: The user's geographic location.\n",
    "+ **Group**: Indicates whether the user was in the Control or Test group.\n",
    "+ **Converted**: A binary indicator showing whether the user converted (1 for converted, 0 for not converted).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2d26ab5e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:52.195952Z",
     "iopub.status.busy": "2024-11-18T02:41:52.195374Z",
     "iopub.status.idle": "2024-11-18T02:41:53.231807Z",
     "shell.execute_reply": "2024-11-18T02:41:53.230744Z"
    },
    "papermill": {
     "duration": 1.053334,
     "end_time": "2024-11-18T02:41:53.234618",
     "exception": false,
     "start_time": "2024-11-18T02:41:52.181284",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import Necessary packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b8793a2b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.262363Z",
     "iopub.status.busy": "2024-11-18T02:41:53.261728Z",
     "iopub.status.idle": "2024-11-18T02:41:53.347580Z",
     "shell.execute_reply": "2024-11-18T02:41:53.346446Z"
    },
    "papermill": {
     "duration": 0.102136,
     "end_time": "2024-11-18T02:41:53.350108",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.247972",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>group</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UK</td>\n",
       "      <td>control</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>US</td>\n",
       "      <td>treatment</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UK</td>\n",
       "      <td>treatment</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>UK</td>\n",
       "      <td>control</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>UK</td>\n",
       "      <td>treatment</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country      group  converted\n",
       "0      UK    control          0\n",
       "1      US  treatment          1\n",
       "2      UK  treatment          0\n",
       "3      UK    control          0\n",
       "4      UK  treatment          0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#store dataset do df\n",
    "df = pd.read_csv('/kaggle/input/ab-test-results-e-commerce/ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ac6c9ab8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.377279Z",
     "iopub.status.busy": "2024-11-18T02:41:53.376872Z",
     "iopub.status.idle": "2024-11-18T02:41:53.384047Z",
     "shell.execute_reply": "2024-11-18T02:41:53.382944Z"
    },
    "papermill": {
     "duration": 0.023846,
     "end_time": "2024-11-18T02:41:53.386676",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.362830",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69889, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find shape of data\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c4763090",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.414389Z",
     "iopub.status.busy": "2024-11-18T02:41:53.413949Z",
     "iopub.status.idle": "2024-11-18T02:41:53.424123Z",
     "shell.execute_reply": "2024-11-18T02:41:53.423200Z"
    },
    "papermill": {
     "duration": 0.026955,
     "end_time": "2024-11-18T02:41:53.426508",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.399553",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13047832992316388"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Find proportion of users converted\n",
    "(df['converted'] == 1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d96cfc31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.454197Z",
     "iopub.status.busy": "2024-11-18T02:41:53.453785Z",
     "iopub.status.idle": "2024-11-18T02:41:53.479006Z",
     "shell.execute_reply": "2024-11-18T02:41:53.477862Z"
    },
    "papermill": {
     "duration": 0.042085,
     "end_time": "2024-11-18T02:41:53.481553",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.439468",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find if any rows have missing values\n",
    "df.isnull().sum().sum() > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "11700d58",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.509368Z",
     "iopub.status.busy": "2024-11-18T02:41:53.508912Z",
     "iopub.status.idle": "2024-11-18T02:41:53.838141Z",
     "shell.execute_reply": "2024-11-18T02:41:53.836968Z"
    },
    "papermill": {
     "duration": 0.346081,
     "end_time": "2024-11-18T02:41:53.840732",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.494651",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAHMCAYAAADI/py4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEyUlEQVR4nO3deViVdf7/8dcBZVNZVBZxg9RR0dTckKYsFUXFyklHLTMyl9GwUsqMynWmNBtNyy3rm7SRZjM2poWDuE1JaZjl3qJ9ddIDlgJugML9+6PvuX8eQb2PgQf1+biuc12c+37f93mfmxvPy3v5HJthGIYAAABwSR7ubgAAAOBaQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAq6yDRs2yGaz6cMPP3R3K5ZkZ2erf//+qlWrlmw2m+bMmVMu67XZbJoyZYpLyzz00EOKiIgol9dH+XjooYdUvXp1d7cBXBWEJlyXUlJSZLPZ5OPjo59//rnU/DvvvFMtW7Z0Q2fXnnHjxmnNmjVKTk7WO++8o549e5aq+ec//ymbzaY33njjoutJT0+XzWbTK6+8Um69nT59WlOmTNGGDRvKbZ1TpkyRzWYr87Fo0aJye53y9tNPP120b5vNphkzZri7xUtasWKFevXqpdq1a8vLy0vh4eEaMGCA1q1b5+7WJEmHDx/WlClTtH37dne3Ajeq4u4GgIpUWFioGTNm6NVXX3V3K9esdevW6Z577tGTTz550Zr4+HgFBAQoNTVVw4cPL7MmNTVVnp6eGjRokCTpzJkzqlLFtX+CXn/9dZWUlJjPT58+ralTp0r6LQiXp4ULF5Y6ghIdHV2ur1ER7rvvPvXu3bvU9FtuucUN3VyeYRh6+OGHlZKSoltuuUVJSUkKCwvTkSNHtGLFCnXr1k2ff/65br31Vrf2efjwYU2dOlURERFq06aNW3uB+xCacF1r06aNXn/9dSUnJys8PNzd7VxVp06dUrVq1X73enJychQYGHjJGm9vb/Xv319LlizR4cOHS23rgoICrVixQt27d1dISIgkycfHx+Veqlat6vIyV6p///6qXbu2pdry2tbloW3btnrggQfc3YZls2bNUkpKisaOHavZs2fLZrOZ85599lm98847LofryuD06dPy8/NzdxsoZ5yew3XtmWeeUXFx8WVPTThObaSkpJSad+G1N47TN999950eeOABBQQEKDg4WBMnTpRhGDp06JDuuece+fv7KywsTLNmzSrzNYuLi/XMM88oLCxM1apV0913361Dhw6Vqvvyyy/Vs2dPBQQEyM/PT3fccYc+//xzpxpHT7t379b999+voKAg3XbbbZd8z/v379ef//xn1axZU35+furUqZNWr15tznec4jQMQ/PnzzdP81zMAw88oJKSEi1durTUvNWrVysvL0+DBw82p124XU+cOKGxY8cqIiJC3t7eCgkJUffu3bVt2zaz5vxrmn766ScFBwdLkqZOnWr251in3W7X0KFDVa9ePXl7e6tOnTq655579NNPP11yu1yOY7ts3LhRjzzyiEJCQlSvXj1z/oIFC9SiRQt5e3srPDxciYmJys3NdVqH4/Twt99+qzvuuEN+fn5q3LixeZ3bxo0bFR0dLV9fXzVt2lRr1679XT1f6F//+pfi4+MVHh4ub29vNWrUSH/9619VXFxcqvbLL79U7969FRQUpGrVqqlVq1aaO3duqbqff/5Zffv2VfXq1RUcHKwnn3yyzPWd78yZM5o+fbqaNWumv//972XuX0OGDFHHjh3N55fbb6X//zu68HftuJ7w/NO5jt/F7t271aVLF/n5+alu3bqaOXOm03IdOnSQJA0dOtTc1xz/XjjWkZWVpc6dO8vPz0/PPPOMEhISVLt2bZ09e7bU++rRo4eaNm16ye2DyofQhOtaZGSkHnzwQb3++us6fPhwua574MCBKikp0YwZMxQdHa2//e1vmjNnjrp37666devqxRdfVOPGjfXkk09q06ZNpZZ//vnntXr1ak2YMEGPPfaY0tPTFRsbqzNnzpg169atU+fOnZWfn6/JkyfrhRdeUG5urrp27aotW7aUWuef//xnnT59Wi+88IJGjBhx0d6zs7N16623as2aNXrkkUf0/PPPq6CgQHfffbdWrFghSercubPeeecdSVL37t31zjvvmM/L0rlzZ9WrV0+pqaml5qWmpsrPz099+/a96PKjRo3SwoUL1a9fPy1YsEBPPvmkfH19tWfPnjLrg4ODtXDhQknSn/70J7O/e++9V5LUr18/rVixQkOHDtWCBQv02GOP6cSJEzp48OBFezjfsWPH9Msvv5iP48ePO81/5JFHtHv3bk2aNElPP/20pN/Ca2JiosLDwzVr1iz169dPr732mnr06FHqg/P48ePq06ePoqOjNXPmTHl7e2vQoEFatmyZBg0apN69e2vGjBk6deqU+vfvrxMnTljq+/Tp0059Ox7nzp0za1JSUlS9enUlJSVp7ty5ateundP7cEhPT1fnzp21e/duPf7445o1a5a6dOmiVatWOdUVFxcrLi5OtWrV0t///nfdcccdmjVrlhYvXnzJXj/77DMdO3ZM999/vzw9PS/73qzst1fi+PHj6tmzp1q3bq1Zs2apWbNmmjBhgj799FNJUvPmzTVt2jRJ0siRI819rXPnzuY6fv31V/Xq1Utt2rTRnDlz1KVLFw0ZMkS//vqr1qxZ4/R6drtd69atu6aOCOL/GMB1aMmSJYYkY+vWrcaPP/5oVKlSxXjsscfM+XfccYfRokUL8/mBAwcMScaSJUtKrUuSMXnyZPP55MmTDUnGyJEjzWnnzp0z6tWrZ9hsNmPGjBnm9OPHjxu+vr5GQkKCOW39+vWGJKNu3bpGfn6+Of2DDz4wJBlz5841DMMwSkpKjCZNmhhxcXFGSUmJWXf69GkjMjLS6N69e6me7rvvPkvbZ+zYsYYk4z//+Y857cSJE0ZkZKQRERFhFBcXO73/xMRES+sdP368IcnYt2+fOS0vL8/w8fEp1duF2zUgIOCyr5OQkGA0bNjQfH706NFS6zGM37a7JOOll16y1Pf5HNvywofjdR371m233WacO3fOXC4nJ8fw8vIyevTo4bT95s2bZ0gy3nzzTXPaHXfcYUgyUlNTzWl79+41JBkeHh7GF198YU5fs2bNRffN8zn24Ys9MjMzzdrTp0+XWv4vf/mL4efnZxQUFBiG8ds+HRkZaTRs2NA4fvy4U+35+2NCQoIhyZg2bZpTzS233GK0a9fukj3PnTvXkGSsWLHiknUOVvdbx+/owIEDTss7/vbWr19vTnP8Lt5++21zWmFhoREWFmb069fPnLZ169aL/h4c61i0aJHT9OLiYqNevXrGwIEDnabPnj3bsNlsxv79+y29b1QeHGnCde+mm27SkCFDtHjxYh05cqTc1nv+Bc+enp5q3769DMPQsGHDzOmBgYFq2rSp9u/fX2r5Bx98UDVq1DCf9+/fX3Xq1NEnn3wiSdq+fbu+//573X///fr111/NIwanTp1St27dtGnTJqeLoqXfjtZY8cknn6hjx45Op/CqV6+ukSNH6qefftLu3butbYQLOP7nfP7Rpn/84x8qKChwOjVXlsDAQH355ZflckTQ19dXXl5e2rBhQ6kjRFb94x//UHp6uvl47733nOaPGDHC6ejI2rVrVVRUpLFjx8rDw8Opzt/fv9QppOrVq5sXxUtS06ZNFRgYqObNmztdcO74uax9qCwjR4506tvxiIqKMmt8fX3Nn0+cOKFffvlFt99+u06fPq29e/dKkr7++msdOHBAY8eOLXVNW1mn0S7c926//fbL9pyfny9JTn8Hl1JR+2316tWdjvp4eXmpY8eOlre59Nt1fUOHDnWa5uHhocGDB2vlypVORwrfe+893XrrrYqMjLyifuE+hCbcEJ577jmdO3euXG+7btCggdPzgIAA+fj4lLp4OCAgoMwP7iZNmjg9t9lsaty4sXkdxvfffy9JSkhIUHBwsNPjjTfeUGFhofLy8pzWYfUf4f/93/8t83qK5s2bm/OvRKtWrdSyZUu9//775rTU1FTVrl1bcXFxl1x25syZ2rlzp+rXr6+OHTtqypQpLn1onc/b21svvviiPv30U4WGhqpz586aOXOm7Ha75XV07txZsbGx5uOPf/yj0/wLt7Vjm124Xb28vHTTTTeV2qb16tUrFT4CAgJUv379UtMkWQ5/TZo0cerb8fD39zdrdu3apT/96U8KCAiQv7+/goODzdDg2Kd+/PFHSbI0NIePj495fZlDUFDQZXt29GT11GNF7bdl/S6s9H++unXrysvLq9T0Bx98UGfOnDFPH+7bt09ZWVkaMmTIFfUK9yI04YZw00036YEHHrjo0aaLXeB8qQtZy7oG42LXZRiGYbHT/89xFOmll14q88hBenp6qVvizz+C4C4PPPCAvvvuO3311Vey2+1av369BgwYcNk7oAYMGKD9+/fr1VdfVXh4uF566SW1aNHCvK7EVWPHjtV3332n6dOny8fHRxMnTlTz5s319ddfX9H6LvR7t/XF9pXy3IfKkpubqzvuuEPffPONpk2bpo8//ljp6el68cUXJanU0UsrrFyPVJZmzZpJknbs2HFFy1+Mq3/P5bHNL7Y/REVFqV27dnr33XclSe+++668vLw0YMAAy+tG5UFowg3DcbTJ8eFwvqCgIEkqdZfTlf7P1QrHkSQHwzD0ww8/mHeHNWrUSNJv/xsv68hBbGzsFd+C37BhQ+3bt6/UdMepmYYNG17ReqXfxgmy2WxKTU3VsmXLVFxcfNlTcw516tTRI488oo8++kgHDhxQrVq19Pzzz1+0/lJ380m/bcMnnnhC//73v7Vz504VFRVd9G7G38uxzS7crkVFRTpw4MDv2qblacOGDfr111+VkpKixx9/XH369FFsbKz5N+Dg2P927txZYb3cdtttCgoK0vvvv3/ZO+0k6/ttRfw9X25fu5QHH3xQ69at05EjR5Samqr4+PhS2xvXBkITbhiNGjXSAw88oNdee63UaRp/f3/Vrl271F1uCxYsqLB+3n77bafTEh9++KGOHDmiXr16SZLatWunRo0a6e9//7tOnjxZavmjR49e8Wv37t1bW7ZsUWZmpjnt1KlTWrx4sSIiIpyuf3FVgwYNdPvtt2vZsmV69913FRkZedmBCYuLi0udagwJCVF4eLgKCwsvupxjHJwLPxxPnz6tgoICp2mNGjVSjRo1Lrm+3yM2NlZeXl565ZVXnI5Q/M///I/y8vIUHx9fIa/rKsdRlfN7LCoqKrWvt23bVpGRkZozZ06p7VteR738/Pw0YcIE7dmzRxMmTChzve+++655p6jV/dYR+M7/ey4uLr7s3XyX4hiH68JtYYXjPxKPP/649u/fz11z17Brb8Qw4HdwDJa3b98+tWjRwmne8OHDNWPGDA0fPlzt27fXpk2b9N1331VYLzVr1tRtt92moUOHKjs7W3PmzFHjxo3NoQI8PDz0xhtvqFevXmrRooWGDh2qunXr6ueff9b69evl7++vjz/++Ipe++mnn9b777+vXr166bHHHlPNmjX11ltv6cCBA/rHP/7hdCHzlXjggQc0cuRIHT58WM8+++xl60+cOKF69eqpf//+at26tapXr661a9dq69atlzwy5Ovrq6ioKC1btkx/+MMfVLNmTbVs2VLnzp1Tt27dNGDAAEVFRalKlSpasWKFsrOznS6+Lk/BwcFKTk7W1KlT1bNnT919993at2+fFixYoA4dOly1D8pt27aZp4LO16hRI8XExOjWW29VUFCQEhIS9Nhjj8lms+mdd94pFVg8PDy0cOFC3XXXXWrTpo2GDh2qOnXqaO/evdq1a1ep2+iv1Pjx47Vr1y7NmjVL69evV//+/RUWFia73a6PPvpIW7Zs0ebNmyVZ329btGihTp06KTk5WceOHVPNmjW1dOlSp2EXXNWoUSMFBgZq0aJFqlGjhqpVq6bo6GhL1xEGBwerZ8+eWr58uQIDAytNgMYVcNdte0BFOn/IgQs5bpE+f8gBw/jtNuxhw4YZAQEBRo0aNYwBAwYYOTk5Fx1y4OjRo6XWW61atVKvd+HwBo7bnt9//30jOTnZCAkJMXx9fY34+Hjjf//3f0st//XXXxv33nuvUatWLcPb29to2LChMWDAACMjI+OyPV3Kjz/+aPTv398IDAw0fHx8jI4dOxqrVq0qVScXhhxwOHbsmOHt7W1IMnbv3l1mzfnbtbCw0Bg/frzRunVro0aNGka1atWM1q1bGwsWLHBa5sIhBwzDMDZv3my0a9fO8PLyMtf5yy+/GImJiUazZs2MatWqGQEBAUZ0dLTxwQcfXLb3y23LS+1bhvHbEAPNmjUzqlataoSGhhqjR48udcv+hfuEQ8OGDY34+PhS0638Di435MD5w158/vnnRqdOnQxfX18jPDzceOqpp8yhDc6/Hd8wDOOzzz4zunfvbv5eWrVqZbz66qvm/Ivt947taNWHH35o9OjRw6hZs6ZRpUoVo06dOsbAgQONDRs2ONVZ3W9//PFHIzY21vD29jZCQ0ONZ555xkhPTy9zyIGyfhdl7Wv/+te/jKioKKNKlSpOww9cbB3ncwwpcv5QJbj22AyjnI6zAgCAMv3rX/9S3759tWnTJt1+++3ubgdXiNAEAEAF69Onj/bs2aMffvjhd11UDvfimiYAACrI0qVL9e2332r16tWaO3cugekax5EmAAAqiM1mU/Xq1TVw4EAtWrTosuOVoXLjtwcAQAXhuMT1hXGaAAAALCA0AQAAWMDpuXJSUlKiw4cPq0aNGlzoBwDANcIwDJ04cULh4eGXHdiX0FRODh8+XOrbyQEAwLXh0KFDqlev3iVrCE3lpEaNGpJ+2+j+/v5u7gYAAFiRn5+v+vXrm5/jl0JoKieOU3L+/v6EJgAArjFWLq3hQnAAAAALCE0AAAAWuDU0TZkyRTabzenRrFkzc35BQYESExNVq1YtVa9eXf369VN2drbTOg4ePKj4+Hj5+fkpJCRE48eP17lz55xqNmzYoLZt28rb21uNGzdWSkpKqV7mz5+viIgI+fj4KDo6Wlu2bKmQ9wwAAK5Nbj/S1KJFCx05csR8fPbZZ+a8cePG6eOPP9by5cu1ceNGHT58WPfee685v7i4WPHx8SoqKtLmzZv11ltvKSUlRZMmTTJrDhw4oPj4eHXp0kXbt2/X2LFjNXz4cK1Zs8asWbZsmZKSkjR58mRt27ZNrVu3VlxcnHJycq7ORgAAAJWf4UaTJ082WrduXea83Nxco2rVqsby5cvNaXv27DEkGZmZmYZhGMYnn3xieHh4GHa73axZuHCh4e/vbxQWFhqGYRhPPfWU0aJFC6d1Dxw40IiLizOfd+zY0UhMTDSfFxcXG+Hh4cb06dMtv5e8vDxDkpGXl2d5GQAA4F6ufH67/UjT999/r/DwcN10000aPHiwDh48KEnKysrS2bNnFRsba9Y2a9ZMDRo0UGZmpiQpMzNTN998s0JDQ82auLg45efna9euXWbN+etw1DjWUVRUpKysLKcaDw8PxcbGmjVlKSwsVH5+vtMDAABcv9wamqKjo5WSkqK0tDQtXLhQBw4c0O23364TJ07IbrfLy8tLgYGBTsuEhobKbrdLkux2u1Ngcsx3zLtUTX5+vs6cOaNffvlFxcXFZdY41lGW6dOnKyAgwHwwsCUAANc3t47T1KtXL/PnVq1aKTo6Wg0bNtQHH3wgX19fN3Z2ecnJyUpKSjKfOwbHAgAA1ye3n547X2BgoP7whz/ohx9+UFhYmIqKipSbm+tUk52drbCwMElSWFhYqbvpHM8vV+Pv7y9fX1/Vrl1bnp6eZdY41lEWb29vcyBLBrQEAOD6V6lC08mTJ/Xjjz+qTp06ateunapWraqMjAxz/r59+3Tw4EHFxMRIkmJiYrRjxw6nu9zS09Pl7++vqKgos+b8dThqHOvw8vJSu3btnGpKSkqUkZFh1gAAALj17rknnnjC2LBhg3HgwAHj888/N2JjY43atWsbOTk5hmEYxqhRo4wGDRoY69atM7766isjJibGiImJMZc/d+6c0bJlS6NHjx7G9u3bjbS0NCM4ONhITk42a/bv32/4+fkZ48ePN/bs2WPMnz/f8PT0NNLS0syapUuXGt7e3kZKSoqxe/duY+TIkUZgYKDTXXmXw91zAABce1z5/HbrNU3//e9/dd999+nXX39VcHCwbrvtNn3xxRcKDg6WJL388svy8PBQv379VFhYqLi4OC1YsMBc3tPTU6tWrdLo0aMVExOjatWqKSEhQdOmTTNrIiMjtXr1ao0bN05z585VvXr19MYbbyguLs6sGThwoI4ePapJkybJbrerTZs2SktLK3VxOAAAuHHZDMMw3N3E9SA/P18BAQHKy8vj+iYAAK4Rrnx+V6prmgAAACort56ew9UX8fRqd7dw3fhpRry7WwAAXEUcaQIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAgkoTmmbMmCGbzaaxY8ea0woKCpSYmKhatWqpevXq6tevn7Kzs52WO3jwoOLj4+Xn56eQkBCNHz9e586dc6rZsGGD2rZtK29vbzVu3FgpKSmlXn/+/PmKiIiQj4+PoqOjtWXLlop4mwAA4BpVKULT1q1b9dprr6lVq1ZO08eNG6ePP/5Yy5cv18aNG3X48GHde++95vzi4mLFx8erqKhImzdv1ltvvaWUlBRNmjTJrDlw4IDi4+PVpUsXbd++XWPHjtXw4cO1Zs0as2bZsmVKSkrS5MmTtW3bNrVu3VpxcXHKycmp+DcPAACuCTbDMAx3NnDy5Em1bdtWCxYs0N/+9je1adNGc+bMUV5enoKDg5Wamqr+/ftLkvbu3avmzZsrMzNTnTp10qeffqo+ffro8OHDCg0NlSQtWrRIEyZM0NGjR+Xl5aUJEyZo9erV2rlzp/magwYNUm5urtLS0iRJ0dHR6tChg+bNmydJKikpUf369fXoo4/q6aeftvQ+8vPzFRAQoLy8PPn7+5fnJipXEU+vdncL142fZsS7uwUAwO/kyue32480JSYmKj4+XrGxsU7Ts7KydPbsWafpzZo1U4MGDZSZmSlJyszM1M0332wGJkmKi4tTfn6+du3aZdZcuO64uDhzHUVFRcrKynKq8fDwUGxsrFlTlsLCQuXn5zs9AADA9auKO1986dKl2rZtm7Zu3Vpqnt1ul5eXlwIDA52mh4aGym63mzXnBybHfMe8S9Xk5+frzJkzOn78uIqLi8us2bt370V7nz59uqZOnWrtjQIAgGue2440HTp0SI8//rjee+89+fj4uKuNK5acnKy8vDzzcejQIXe3BAAAKpDbQlNWVpZycnLUtm1bValSRVWqVNHGjRv1yiuvqEqVKgoNDVVRUZFyc3OdlsvOzlZYWJgkKSwsrNTddI7nl6vx9/eXr6+vateuLU9PzzJrHOsoi7e3t/z9/Z0eAADg+uW20NStWzft2LFD27dvNx/t27fX4MGDzZ+rVq2qjIwMc5l9+/bp4MGDiomJkSTFxMRox44dTne5paeny9/fX1FRUWbN+etw1DjW4eXlpXbt2jnVlJSUKCMjw6wBAABw2zVNNWrUUMuWLZ2mVatWTbVq1TKnDxs2TElJSapZs6b8/f316KOPKiYmRp06dZIk9ejRQ1FRURoyZIhmzpwpu92u5557TomJifL29pYkjRo1SvPmzdNTTz2lhx9+WOvWrdMHH3yg1av//11kSUlJSkhIUPv27dWxY0fNmTNHp06d0tChQ6/S1gAAAJWdWy8Ev5yXX35ZHh4e6tevnwoLCxUXF6cFCxaY8z09PbVq1SqNHj1aMTExqlatmhISEjRt2jSzJjIyUqtXr9a4ceM0d+5c1atXT2+88Ybi4uLMmoEDB+ro0aOaNGmS7Ha72rRpo7S0tFIXhwMAgBuX28dpul4wTtONh3GaAODad02N0wQAAHAtIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWuByaDh06pP/+97/m8y1btmjs2LFavHhxuTYGAABQmbgcmu6//36tX79ekmS329W9e3dt2bJFzz77rKZNm1buDQIAAFQGLoemnTt3qmPHjpKkDz74QC1bttTmzZv13nvvKSUlpbz7AwAAqBRcDk1nz56Vt7e3JGnt2rW6++67JUnNmjXTkSNHyrc7AACASsLl0NSiRQstWrRI//nPf5Senq6ePXtKkg4fPqxatWqVe4MAAACVgcuh6cUXX9Rrr72mO++8U/fdd59at24tSVq5cqV52g4AAOB6U8XVBe6880798ssvys/PV1BQkDl95MiRqlatWrk2BwAAUFm4fKSpa9euOnHihFNgkqSaNWtq4MCB5dYYAABAZeJyaNqwYYOKiopKTS8oKNB//vOfcmkKAACgsrF8eu7bb781f969e7fsdrv5vLi4WGlpaapbt275dgcAAFBJWA5Nbdq0kc1mk81mU9euXUvN9/X11auvvlquzQEAAFQWlkPTgQMHZBiGbrrpJm3ZskXBwcHmPC8vL4WEhMjT07NCmgQAAHA3y6GpYcOGkqSSkpIKawYAAKCysnQh+MqVK3X27Fnz50s9XLFw4UK1atVK/v7+8vf3V0xMjD799FNzfkFBgRITE1WrVi1Vr15d/fr1U3Z2ttM6Dh48qPj4ePn5+SkkJETjx4/XuXPnnGo2bNigtm3bytvbW40bNy7z617mz5+viIgI+fj4KDo6Wlu2bHHpvQAAgOubpSNNffv2ld1uV0hIiPr27XvROpvNpuLiYssvXq9ePc2YMUNNmjSRYRh66623dM899+jrr79WixYtNG7cOK1evVrLly9XQECAxowZo3vvvVeff/65pN8uQI+Pj1dYWJg2b96sI0eO6MEHH1TVqlX1wgsvSPrttGJ8fLxGjRql9957TxkZGRo+fLjq1KmjuLg4SdKyZcuUlJSkRYsWKTo6WnPmzFFcXJz27dunkJAQy+8HAABcv2yGYRjubuJ8NWvW1EsvvaT+/fsrODhYqamp6t+/vyRp7969at68uTIzM9WpUyd9+umn6tOnjw4fPqzQ0FBJ0qJFizRhwgQdPXpUXl5emjBhglavXq2dO3earzFo0CDl5uYqLS1NkhQdHa0OHTpo3rx5kn47BVm/fn09+uijevrppy31nZ+fr4CAAOXl5cnf3788N0m5inh6tbtbuG78NCPe3S0AAH4nVz6/XR6nqSy5ubm/ex3FxcVaunSpTp06pZiYGGVlZens2bOKjY01a5o1a6YGDRooMzNTkpSZmambb77ZDEySFBcXp/z8fO3atcusOX8djhrHOoqKipSVleVU4+HhodjYWLOmLIWFhcrPz3d6AACA69cVfffcsmXLzOd//vOfVbNmTdWtW1fffPONyw3s2LFD1atXl7e3t0aNGqUVK1YoKipKdrtdXl5eCgwMdKoPDQ01x4iy2+1Ogckx3zHvUjX5+fk6c+aMfvnlFxUXF5dZc/5YVBeaPn26AgICzEf9+vVdfu8AAODa4XJoWrRokRkQ0tPTtXbtWqWlpalXr14aP368yw00bdpU27dv15dffqnRo0crISFBu3fvdnk9V1tycrLy8vLMx6FDh9zdEgAAqEAuf2Gv3W43Q9OqVas0YMAA9ejRQxEREYqOjna5AS8vLzVu3FiS1K5dO23dulVz587VwIEDVVRUpNzcXKejTdnZ2QoLC5MkhYWFlbrLzXF33fk1F95xl52dLX9/f/n6+srT01Oenp5l1jjWURZvb295e3u7/H4BAMC1yeUjTUFBQeZRlbS0NPNaIMMwXLpz7mJKSkpUWFiodu3aqWrVqsrIyDDn7du3TwcPHlRMTIwkKSYmRjt27FBOTo5Zk56eLn9/f0VFRZk156/DUeNYh5eXl9q1a+dUU1JSooyMDLMGAADA5SNN9957r+6//341adJEv/76q3r16iVJ+vrrr80jRlYlJyerV69eatCggU6cOKHU1FRt2LBBa9asUUBAgIYNG6akpCTVrFlT/v7+evTRRxUTE6NOnTpJknr06KGoqCgNGTJEM2fOlN1u13PPPafExETzKNCoUaM0b948PfXUU3r44Ye1bt06ffDBB1q9+v/fRZaUlKSEhAS1b99eHTt21Jw5c3Tq1CkNHTrU1c0DAACuUy6HppdfflkRERE6dOiQZs6cqerVq0uSjhw5okceecSldeXk5OjBBx/UkSNHFBAQoFatWmnNmjXq3r27+VoeHh7q16+fCgsLFRcXpwULFpjLe3p6atWqVRo9erRiYmJUrVo1JSQkaNq0aWZNZGSkVq9erXHjxmnu3LmqV6+e3njjDXOMJkkaOHCgjh49qkmTJslut6tNmzZKS0srdXE4AAC4cVW6cZquVYzTdONhnCYAuPa58vlt6UjTypUr1atXL1WtWvWyX5Vy9913W+8UAADgGuHWr1EBAAC4VlgKTSUlJWX+DAAAcKOwPORA//79lZaWJi6BAgAANyLLoen48eOKj49XgwYNNGnSJO3fv78i+wIAAKhULIemjIwM7d+/X8OGDdO7776rJk2aqGvXrkpNTVVhYWFF9ggAAOB2Lo0I3rBhQ02ZMkX79+9Xenq6wsPDNWLECNWpU0eJiYnKysqqqD4BAADcyuWvUXHo2rWr3n33Xdntdk2fPl1Lly69ou+eAwAAuBa4PCL4+Q4cOKCUlBSlpKQoLy/P/B46AACA643LR5oKCgr07rvvqmvXrmrSpInefvttDRs2TAcOHFBaWlpF9AgAAOB2lo80bdmyRW+++aaWLVumgoIC/elPf1JaWpq6desmm81WkT0CAAC4neXQ1KlTJ7Vu3Vp//etfNXjwYAUFBVVkXwAAAJWK5dD01VdfqW3bthXZCwAAQKVl+ZomAhMAALiRXfGQAwAAADcSQhMAAIAFlkLTypUrdfbs2YruBQAAoNKyFJr+9Kc/KTc3V5Lk6empnJyciuwJAACg0rEUmoKDg/XFF19IkgzDYFwmAABww7E05MCoUaN0zz33yGazyWazKSws7KK1xcXF5dYcAABAZWEpNE2ZMkWDBg3SDz/8oLvvvltLlixRYGBgBbcGAABQeVge3LJZs2Zq1qyZJk+erD//+c/y8/OryL4AAAAqFcuhyWHy5MmSpKNHj2rfvn2SpKZNmyo4OLh8OwMAAKhEXB6n6fTp03r44YcVHh6uzp07q3PnzgoPD9ewYcN0+vTpiugRAADA7VwOTePGjdPGjRu1cuVK5ebmKjc3V//617+0ceNGPfHEExXRIwAAgNu5fHruH//4hz788EPdeeed5rTevXvL19dXAwYM0MKFC8uzPwAAgErhik7PhYaGlpoeEhLC6TkAAHDdcjk0xcTEaPLkySooKDCnnTlzRlOnTlVMTEy5NgcAAFBZuHx6bu7cuYqLi1O9evXUunVrSdI333wjHx8frVmzptwbBAAAqAxcDk0tW7bU999/r/fee0979+6VJN13330aPHiwfH19y71BAACAysDl0CRJfn5+GjFiRHn3AgAAUGm5fE0TAADAjYjQBAAAYAGhCQAAwAJCEwAAgAUuh6abbrpJv/76a6npubm5uummm8qlKQAAgMrG5dD0008/qbi4uNT0wsJC/fzzz+XSFAAAQGVjeciBlStXmj+vWbNGAQEB5vPi4mJlZGQoIiKiXJsDAACoLCyHpr59+0qSbDabEhISnOZVrVpVERERmjVrVrk2BwAAUFlYDk0lJSWSpMjISG3dulW1a9eusKYAAAAqG5dHBD9w4EBF9AEAAFCpXdHXqGRkZCgjI0M5OTnmESiHN998s1waAwAAqExcDk1Tp07VtGnT1L59e9WpU0c2m60i+gIAAKhUXA5NixYtUkpKioYMGVIR/QAAAFRKLo/TVFRUpFtvvbUiegEAAKi0XA5Nw4cPV2pqakX0AgAAUGm5fHquoKBAixcv1tq1a9WqVStVrVrVaf7s2bPLrTkAAIDKwuXQ9O2336pNmzaSpJ07dzrN46JwAABwvXI5NK1fv74i+gAAAKjUXL6mCQAA4Ebk8pGmLl26XPI03Lp1635XQwAAAJWRy6HJcT2Tw9mzZ7V9+3bt3Lmz1Bf5AgAAXC9cDk0vv/xymdOnTJmikydP/u6GAAAAKqNyu6bpgQce4HvnAADAdavcQlNmZqZ8fHzKa3UAAACVisun5+69916n54Zh6MiRI/rqq680ceLEcmsMAACgMnE5NAUEBDg99/DwUNOmTTVt2jT16NGj3BoDAACoTFwOTUuWLKmIPgAAACo1l0OTQ1ZWlvbs2SNJatGihW655ZZyawoAAKCycTk05eTkaNCgQdqwYYMCAwMlSbm5uerSpYuWLl2q4ODg8u4RAADA7Vy+e+7RRx/ViRMntGvXLh07dkzHjh3Tzp07lZ+fr8cee6wiegQAAHA7l0NTWlqaFixYoObNm5vToqKiNH/+fH366acurWv69Onq0KGDatSooZCQEPXt21f79u1zqikoKFBiYqJq1aql6tWrq1+/fsrOznaqOXjwoOLj4+Xn56eQkBCNHz9e586dc6rZsGGD2rZtK29vbzVu3FgpKSml+pk/f74iIiLk4+Oj6OhobdmyxaX3AwAArl8uh6aSkhJVrVq11PSqVauqpKTEpXVt3LhRiYmJ+uKLL5Senq6zZ8+qR48eOnXqlFkzbtw4ffzxx1q+fLk2btyow4cPOw17UFxcrPj4eBUVFWnz5s166623lJKSokmTJpk1Bw4cUHx8vLp06aLt27dr7NixGj58uNasWWPWLFu2TElJSZo8ebK2bdum1q1bKy4uTjk5OS69JwAAcH2yGYZhuLLAPffco9zcXL3//vsKDw+XJP38888aPHiwgoKCtGLFiitu5ujRowoJCdHGjRvVuXNn5eXlKTg4WKmpqerfv78kae/evWrevLkyMzPVqVMnffrpp+rTp48OHz6s0NBQSdKiRYs0YcIEHT16VF5eXpowYYJWr16tnTt3mq81aNAg5ebmKi0tTZIUHR2tDh06aN68eZJ+C4f169fXo48+qqeffvqyvefn5ysgIEB5eXny9/e/4m1Q0SKeXu3uFq4bP82Id3cLAIDfyZXPb5ePNM2bN0/5+fmKiIhQo0aN1KhRI0VGRio/P1+vvvrqFTctSXl5eZKkmjVrSvrtDr2zZ88qNjbWrGnWrJkaNGigzMxMSb+NRH7zzTebgUmS4uLilJ+fr127dpk156/DUeNYR1FRkbKyspxqPDw8FBsba9ZcqLCwUPn5+U4PAABw/XL57rn69etr27ZtWrt2rfbu3StJat68ealQ4qqSkhKNHTtWf/zjH9WyZUtJkt1ul5eXl3mXnkNoaKjsdrtZc35gcsx3zLtUTX5+vs6cOaPjx4+ruLi4zBrHe7zQ9OnTNXXq1Ct7swAA4JpzReM02Ww2de/eXd27dy+3RhITE7Vz50599tln5bbOipScnKykpCTzeX5+vurXr+/GjgAAQEWyfHpu3bp1ioqKKvM0VF5enlq0aKH//Oc/V9TEmDFjtGrVKq1fv1716tUzp4eFhamoqEi5ublO9dnZ2QoLCzNrLrybzvH8cjX+/v7y9fVV7dq15enpWWaNYx0X8vb2lr+/v9MDAABcvyyHpjlz5mjEiBFlhoOAgAD95S9/0ezZs116ccMwNGbMGK1YsULr1q1TZGSk0/x27dqpatWqysjIMKft27dPBw8eVExMjCQpJiZGO3bscLrLLT09Xf7+/oqKijJrzl+Ho8axDi8vL7Vr186ppqSkRBkZGWYNAAC4sVkOTd9884169ux50fk9evRQVlaWSy+emJiod999V6mpqapRo4bsdrvsdrvOnDkj6bcwNmzYMCUlJWn9+vXKysrS0KFDFRMTo06dOpmvGxUVpSFDhuibb77RmjVr9NxzzykxMVHe3t6SpFGjRmn//v166qmntHfvXi1YsEAffPCBxo0bZ/aSlJSk119/XW+99Zb27Nmj0aNH69SpUxo6dKhL7wkAAFyfLF/TlJ2dXeb4TOaKqlTR0aNHXXrxhQsXSpLuvPNOp+lLlizRQw89JEl6+eWX5eHhoX79+qmwsFBxcXFasGCBWevp6alVq1Zp9OjRiomJUbVq1ZSQkKBp06aZNZGRkVq9erXGjRunuXPnql69enrjjTcUFxdn1gwcOFBHjx7VpEmTZLfb1aZNG6WlpZW6OBwAANyYLI/T1KhRI82aNUt9+/Ytc/4///lPPfnkk9q/f3959nfNYJymGw/jNAHAta9Cxmnq3bu3Jk6cqIKCglLzzpw5o8mTJ6tPnz6udwsAAHANsHx67rnnntM///lP/eEPf9CYMWPUtGlTSb+N0D1//nwVFxfr2WefrbBGAQAA3MlyaAoNDdXmzZs1evRoJScny3FWz2azKS4uTvPnz+f6HwAAcN1yaXDLhg0b6pNPPtHx48f1ww8/yDAMNWnSREFBQRXVHwAAQKVwRSOCBwUFqUOHDuXdCwAAQKXl8hf2AgAA3IgITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGDBFQ05AADlie9ELB98HyJQsTjSBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYIFbQ9OmTZt01113KTw8XDabTR999JHTfMMwNGnSJNWpU0e+vr6KjY3V999/71Rz7NgxDR48WP7+/goMDNSwYcN08uRJp5pvv/1Wt99+u3x8fFS/fn3NnDmzVC/Lly9Xs2bN5OPjo5tvvlmffPJJub9fAABw7XJraDp16pRat26t+fPnlzl/5syZeuWVV7Ro0SJ9+eWXqlatmuLi4lRQUGDWDB48WLt27VJ6erpWrVqlTZs2aeTIkeb8/Px89ejRQw0bNlRWVpZeeuklTZkyRYsXLzZrNm/erPvuu0/Dhg3T119/rb59+6pv377auXNnxb15AABwTbEZhmG4uwlJstlsWrFihfr27Svpt6NM4eHheuKJJ/Tkk09KkvLy8hQaGqqUlBQNGjRIe/bsUVRUlLZu3ar27dtLktLS0tS7d2/997//VXh4uBYuXKhnn31WdrtdXl5ekqSnn35aH330kfbu3StJGjhwoE6dOqVVq1aZ/XTq1Elt2rTRokWLLPWfn5+vgIAA5eXlyd/fv7w2S7mLeHq1u1u4bvw0I97dLVw32C/LB/sk4DpXPr8r7TVNBw4ckN1uV2xsrDktICBA0dHRyszMlCRlZmYqMDDQDEySFBsbKw8PD3355ZdmTefOnc3AJElxcXHat2+fjh8/btac/zqOGsfrlKWwsFD5+flODwAAcP2qtKHJbrdLkkJDQ52mh4aGmvPsdrtCQkKc5lepUkU1a9Z0qilrHee/xsVqHPPLMn36dAUEBJiP+vXru/oWAQDANaTShqbKLjk5WXl5eebj0KFD7m4JAABUoEobmsLCwiRJ2dnZTtOzs7PNeWFhYcrJyXGaf+7cOR07dsyppqx1nP8aF6txzC+Lt7e3/P39nR4AAOD6VWlDU2RkpMLCwpSRkWFOy8/P15dffqmYmBhJUkxMjHJzc5WVlWXWrFu3TiUlJYqOjjZrNm3apLNnz5o16enpatq0qYKCgsya81/HUeN4HQAAALeGppMnT2r79u3avn27pN8u/t6+fbsOHjwom82msWPH6m9/+5tWrlypHTt26MEHH1R4eLh5h13z5s3Vs2dPjRgxQlu2bNHnn3+uMWPGaNCgQQoPD5ck3X///fLy8tKwYcO0a9cuLVu2THPnzlVSUpLZx+OPP660tDTNmjVLe/fu1ZQpU/TVV19pzJgxV3uTAACASqqKO1/8q6++UpcuXcznjiCTkJCglJQUPfXUUzp16pRGjhyp3Nxc3XbbbUpLS5OPj4+5zHvvvacxY8aoW7du8vDwUL9+/fTKK6+Y8wMCAvTvf/9biYmJateunWrXrq1JkyY5jeV06623KjU1Vc8995yeeeYZNWnSRB999JFatmx5FbYCAAC4FlSacZqudYzTdONhTJzyw35ZPtgnAdddF+M0AQAAVCaEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAgirubgAAgMom4unV7m7huvHTjHh3t1BuONIEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0XmD9/viIiIuTj46Po6Ght2bLF3S0BAIBKgNB0nmXLlikpKUmTJ0/Wtm3b1Lp1a8XFxSknJ8fdrQEAADcjNJ1n9uzZGjFihIYOHaqoqCgtWrRIfn5+evPNN93dGgAAcLMq7m6gsigqKlJWVpaSk5PNaR4eHoqNjVVmZmap+sLCQhUWFprP8/LyJEn5+fkV3+zvUFJ42t0tXDcq++/6WsJ+WT7YJ8sP+2T5qez7paM/wzAuW0to+j+//PKLiouLFRoa6jQ9NDRUe/fuLVU/ffp0TZ06tdT0+vXrV1iPqFwC5ri7A8AZ+yQqo2tlvzxx4oQCAgIuWUNoukLJyclKSkoyn5eUlOjYsWOqVauWbDabGzu79uXn56t+/fo6dOiQ/P393d0OwD6JSod9svwYhqETJ04oPDz8srWEpv9Tu3ZteXp6Kjs722l6dna2wsLCStV7e3vL29vbaVpgYGBFtnjD8ff35x8DVCrsk6hs2CfLx+WOMDlwIfj/8fLyUrt27ZSRkWFOKykpUUZGhmJiYtzYGQAAqAw40nSepKQkJSQkqH379urYsaPmzJmjU6dOaejQoe5uDQAAuBmh6TwDBw7U0aNHNWnSJNntdrVp00ZpaWmlLg5HxfL29tbkyZNLnf4E3IV9EpUN+6R72Awr99gBAADc4LimCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAcI0rLi52dws3BEITALjoyJEj7m4BkCR99913euqpp1SvXj13t3JDIDShUjp37pxOnjzp7jZwAzr/i7jLcuTIEd15551XpxmgDKdPn9aSJUt0++23KyoqSps2bbrsfovyQWiCW3388cdKSUlxmvb888+revXqCgwMVI8ePXT8+HH3NIcb0pIlS/T888+XOc8RmIKDg69yV4D0xRdfaPjw4apTp45mz56tzMxMrV+/Xl988YXGjx/v7vZuCIQmuNXs2bN16tQp8/nmzZs1adIkTZw4UR988IEOHTqkv/71r27sEDealStX6oUXXtDChQudptvtdnXp0kU1a9ZUWlqam7rDjWjWrFlq0aKF+vfvr6CgIG3atEk7duyQzWZTrVq13N3eDYWvUYFbhYSEaM2aNbrlllsk/XZqZPfu3eaH0ieffKLHH39c33//vTvbxA1m9erV6tevn1JSUjRo0CAzMPn7+2vt2rWqUaOGu1vEDaRKlSqaMGGCpk2bJk9PT3N61apV9c033ygqKsqN3d1YONIEtzpx4oTT/5Q+++wzdevWzXzeokULHT582B2t4QYWHx+vN998U0OHDlVKSoq6du2q6tWr69///jeBCVfdX//6Vy1fvlyRkZGaMGGCdu7c6e6WbliEJrhV3bp1tWfPHknSyZMn9c033+jWW2815//666/y8/NzV3u4gd1///2aNWuWhg0bpmrVqmnt2rUKCAhwd1u4ASUnJ+u7777TO++8I7vdrujoaLVu3VqGYXDN51XG6Tm4VXJysj766CM988wz+uSTT7R582bt37/fPAS9ePFivf322/rss8/c3CluFLfccotsNpv5fPfu3apfv36pI0zbtm272q0Bkn47Qp+amqo333xTWVlZ6tixo/r3788ddFcBoQludebMGf3lL3/Rxx9/rLCwMC1evFi33367Ob9Lly7q2bOnJkyY4MYucSOZMmWKU2i6mMmTJ1+FboBL27lzp/7nf/5H7733nnJyctzdznWP0AQAQCW2bt06jRkzRl988YX8/f2d5uXl5SkmJkavvPKKYmNj3dThjYNrmuBWQUFBqlmzZqlHZGSk4uLilJ6e7u4WcYNhn0RlM2fOHI0YMaJUYJKkgIAAjRo1SvPnz3dDZzcejjTBrd56660yp+fm5iorK0vLli3Thx9+qLvuuusqd4YbFfskKpuGDRsqLS1NzZs3L3P+3r171aNHDx08ePAqd3bjITShUps9e7Y+/PBDbd682d2tAJLYJ3H1+fj4aOfOnWrcuHGZ83/44QfdfPPNOnPmzFXu7MbD6TlUan369NHevXvd3QZgYp/E1Va3bt1Ljs307bffqk6dOlexoxsXoQmVWmFhoby8vNzdBmBin8TV1rt3b02cOFEFBQWl5p05c0aTJ09Wnz593NDZjYfTc6jUxo4dq7179/JdX6g02CdxtWVnZ6tt27by9PTUmDFj1LRpU0m/Xcs0f/58FRcXa9u2bQoNDXVzp9e/Ku5uADe2iw3GlpeXp23btum7777Tpk2brnJXuJGxT6KyCQ0N1ebNmzV69GglJyfLcazDZrMpLi5O8+fPJzBdJRxpglt16dKlzOn+/v5q2rSpRo8ercjIyKvcFW5k7JOozI4fP64ffvhBhmGoSZMmCgoKcndLNxRCEwAAgAVcCA4AAGABoQkAAMACQhMAAIAFhCYAAAALCE0A8Dv99NNPstls2r59u7tbAVCBCE0AcJUUFRW5uwUAvwOhCcA1r6SkRDNnzlTjxo3l7e2tBg0a6Pnnn5ck7dixQ127dpWvr69q1aqlkSNH6uTJk+ayd955p8aOHeu0vr59++qhhx4yn0dEROiFF17Qww8/rBo1aqhBgwZavHixOd8xbtMtt9wim82mO++8U5L00EMPqW/fvnr++ecVHh6upk2batq0aWrZsmWp99CmTRtNnDixnLYIgIpAaAJwzUtOTtaMGTM0ceJE7d69W6mpqQoNDdWpU6cUFxenoKAgbd26VcuXL9fatWs1ZswYl19j1qxZat++vb7++ms98sgjGj16tPbt2ydJ2rJliyRp7dq1OnLkiP75z3+ay2VkZGjfvn1KT0/XqlWr9PDDD2vPnj3aunWrWfP111/r22+/1dChQ3/nlgBQkfgaFQDXtBMnTmju3LmaN2+eEhISJEmNGjXSbbfdptdff10FBQV6++23Va1aNUnSvHnzdNddd+nFF1906asnevfurUceeUSSNGHCBL388stav369mjZtquDgYElSrVq1FBYW5rRctWrV9MYbbzh9yW9cXJyWLFmiDh06SJKWLFmiO+64QzfddNOVbwgAFY4jTQCuaXv27FFhYaG6detW5rzWrVubgUmS/vjHP6qkpMQ8SmRVq1atzJ9tNpvCwsKUk5Nz2eVuvvlmp8AkSSNGjND777+vgoICFRUVKTU1VQ8//LBL/QC4+jjSBOCa5uvr+7uW9/Dw0IXfJnX27NlSdVWrVnV6brPZVFJSctn1nx/YHO666y55e3trxYoV8vLy0tmzZ9W/f38XOwdwtXGkCcA1rUmTJvL19VVGRkapec2bN9c333yjU6dOmdM+//xzeXh4qGnTppKk4OBgHTlyxJxfXFysnTt3utSD40hScXGxpfoqVaooISFBS5Ys0ZIlSzRo0KDfHf4AVDyONAG4pvn4+GjChAl66qmn5OXlpT/+8Y86evSodu3apcGDB2vy5MlKSEjQlClTdPToUT366KMaMmSIeT1T165dlZSUpNWrV6tRo0aaPXu2cnNzXeohJCREvr6+SktLU7169eTj46OAgIBLLjN8+HA1b95c0m9BDkDlx5EmANe8iRMn6oknntCkSZPUvHlzDRw4UDk5OfLz89OaNWt07NgxdejQQf3791e3bt00b948c9mHH35YCQkJevDBB82Lsbt06eLS61epUkWvvPKKXnvtNYWHh+uee+657DJNmjTRrbfeqmbNmik6Otrl9wzg6rMZF57MBwBUOMMw1KRJEz3yyCNKSkpydzsALOD0HABcZUePHtXSpUtlt9sZmwm4hhCaAOAqCwkJUe3atbV48WIFBQW5ux0AFhGaAOAq46oI4NrEheAAAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFvw/ddqXec1UlHAAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot number of visitors from each country\n",
    "df['country'].value_counts().plot(kind='bar')\n",
    "plt.title('Number of Visits From Each Country')\n",
    "plt.ylabel('Count of Visits')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fbbf0b9a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.869482Z",
     "iopub.status.busy": "2024-11-18T02:41:53.869019Z",
     "iopub.status.idle": "2024-11-18T02:41:53.877356Z",
     "shell.execute_reply": "2024-11-18T02:41:53.876336Z"
    },
    "papermill": {
     "duration": 0.025557,
     "end_time": "2024-11-18T02:41:53.879746",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.854189",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13047832992316388"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability of conversion\n",
    "(df['converted'] == 1).sum() / len(df['converted'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f0020a30",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.909272Z",
     "iopub.status.busy": "2024-11-18T02:41:53.908223Z",
     "iopub.status.idle": "2024-11-18T02:41:53.933572Z",
     "shell.execute_reply": "2024-11-18T02:41:53.932377Z"
    },
    "papermill": {
     "duration": 0.042612,
     "end_time": "2024-11-18T02:41:53.936003",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.893391",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1052540515600669"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability of conversion from the control page\n",
    "df.query('group == \"control\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c3edfaa6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:53.965507Z",
     "iopub.status.busy": "2024-11-18T02:41:53.965043Z",
     "iopub.status.idle": "2024-11-18T02:41:53.982358Z",
     "shell.execute_reply": "2024-11-18T02:41:53.981311Z"
    },
    "papermill": {
     "duration": 0.035045,
     "end_time": "2024-11-18T02:41:53.985042",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.949997",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15532078043793132"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability of conversion from the treatment page\n",
    "df.query('group == \"treatment\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1665f624",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.014656Z",
     "iopub.status.busy": "2024-11-18T02:41:54.014214Z",
     "iopub.status.idle": "2024-11-18T02:41:54.037648Z",
     "shell.execute_reply": "2024-11-18T02:41:54.036519Z"
    },
    "papermill": {
     "duration": 0.041212,
     "end_time": "2024-11-18T02:41:54.040223",
     "exception": false,
     "start_time": "2024-11-18T02:41:53.999011",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5038131894861853"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability that an individual used treatment\n",
    "((df['group'] == 'treatment').sum()) / len(df['group'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c4ab7ab9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.070374Z",
     "iopub.status.busy": "2024-11-18T02:41:54.069919Z",
     "iopub.status.idle": "2024-11-18T02:41:54.093748Z",
     "shell.execute_reply": "2024-11-18T02:41:54.092600Z"
    },
    "papermill": {
     "duration": 0.041814,
     "end_time": "2024-11-18T02:41:54.096267",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.054453",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.04990771079855199"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability that user was from canada\n",
    "(df['country'] == 'CA').sum() / len(df['country'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5faadb31",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.127232Z",
     "iopub.status.busy": "2024-11-18T02:41:54.126161Z",
     "iopub.status.idle": "2024-11-18T02:41:54.145135Z",
     "shell.execute_reply": "2024-11-18T02:41:54.144012Z"
    },
    "papermill": {
     "duration": 0.037002,
     "end_time": "2024-11-18T02:41:54.147706",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.110704",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13277379733879222"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability of conversion that a user was in the US\n",
    "df.query('country == \"US\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f5f713",
   "metadata": {
    "papermill": {
     "duration": 0.013796,
     "end_time": "2024-11-18T02:41:54.175785",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.161989",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "`h)` Given that an individual was in the `UK`, what was the probability that they `converted`? $P(\\text{converted} == 1|\\text{country} ==\\text{\"UK\"})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f7f85bce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.206368Z",
     "iopub.status.busy": "2024-11-18T02:41:54.205906Z",
     "iopub.status.idle": "2024-11-18T02:41:54.222637Z",
     "shell.execute_reply": "2024-11-18T02:41:54.221552Z"
    },
    "papermill": {
     "duration": 0.035038,
     "end_time": "2024-11-18T02:41:54.225276",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.190238",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12512107572218106"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find probability of conversion that a user was in the UK\n",
    "df.query('country == \"UK\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "32747f8d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.257383Z",
     "iopub.status.busy": "2024-11-18T02:41:54.256934Z",
     "iopub.status.idle": "2024-11-18T02:41:54.316698Z",
     "shell.execute_reply": "2024-11-18T02:41:54.315549Z"
    },
    "papermill": {
     "duration": 0.079268,
     "end_time": "2024-11-18T02:41:54.319440",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.240172",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>US</th>\n",
       "      <th>UK</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>10.7%</td>\n",
       "      <td>10.2%</td>\n",
       "      <td>9.4%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>treatment</th>\n",
       "      <td>15.8%</td>\n",
       "      <td>14.9%</td>\n",
       "      <td>15.4%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              US     UK     CA\n",
       "group                         \n",
       "control    10.7%  10.2%   9.4%\n",
       "treatment  15.8%  14.9%  15.4%"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create table with conversion rates \n",
    "us_conversion = df.query('country == \"US\"').groupby('group')['converted'].mean()\n",
    "uk_conversion = df.query('country == \"UK\"').groupby('group')['converted'].mean()\n",
    "ca_conversion = df.query('country == \"CA\"').groupby('group')['converted'].mean()\n",
    "country_conversion_rates = pd.DataFrame({    \n",
    "    'US': us_conversion,    'UK': uk_conversion,    \n",
    "    'CA': ca_conversion})\n",
    "country_conversion_rates = country_conversion_rates.map(lambda x: f\"{x*100:.1f}%\")\n",
    "country_conversion_rates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da411a14",
   "metadata": {
    "papermill": {
     "duration": 0.014505,
     "end_time": "2024-11-18T02:41:54.349103",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.334598",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Analysis of conversion rates\n",
    "These probabilities indicate that the Treatment page is consistently associated with higher conversions across all countries. This suggests that the Treatment page has a positive impact on conversion regardless of location. Also, due to the fact that the individual countries have minimal differences in conversion, this suggests that the location itself has little influence on conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4d51c924",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.380866Z",
     "iopub.status.busy": "2024-11-18T02:41:54.380413Z",
     "iopub.status.idle": "2024-11-18T02:41:54.405829Z",
     "shell.execute_reply": "2024-11-18T02:41:54.404708Z"
    },
    "papermill": {
     "duration": 0.044574,
     "end_time": "2024-11-18T02:41:54.408706",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.364132",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#calculate conversion rate under null hypothesis\n",
    "p_control_treatment_null  = df['converted'].mean()\n",
    "\n",
    "#create samples for control group under null hypothesis\n",
    "n_treatment = df.query('group == \"treatment\"').shape[0]\n",
    "\n",
    "#create samples for control group under null hypothesis\n",
    "n_control = df.query('group == \"control\"').shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "81e7ea61",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.441757Z",
     "iopub.status.busy": "2024-11-18T02:41:54.440964Z",
     "iopub.status.idle": "2024-11-18T02:41:54.448686Z",
     "shell.execute_reply": "2024-11-18T02:41:54.447363Z"
    },
    "papermill": {
     "duration": 0.026965,
     "end_time": "2024-11-18T02:41:54.451235",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.424270",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13047832992316388"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_control_treatment_null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "311560be",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.484030Z",
     "iopub.status.busy": "2024-11-18T02:41:54.483181Z",
     "iopub.status.idle": "2024-11-18T02:41:54.490746Z",
     "shell.execute_reply": "2024-11-18T02:41:54.489559Z"
    },
    "papermill": {
     "duration": 0.026468,
     "end_time": "2024-11-18T02:41:54.493217",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.466749",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35211"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_treatment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4a4a7dc0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.526364Z",
     "iopub.status.busy": "2024-11-18T02:41:54.525225Z",
     "iopub.status.idle": "2024-11-18T02:41:54.532970Z",
     "shell.execute_reply": "2024-11-18T02:41:54.531824Z"
    },
    "papermill": {
     "duration": 0.027009,
     "end_time": "2024-11-18T02:41:54.535471",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.508462",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34678"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e66925ac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.569153Z",
     "iopub.status.busy": "2024-11-18T02:41:54.568726Z",
     "iopub.status.idle": "2024-11-18T02:41:54.578889Z",
     "shell.execute_reply": "2024-11-18T02:41:54.577748Z"
    },
    "papermill": {
     "duration": 0.03046,
     "end_time": "2024-11-18T02:41:54.581444",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.550984",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#simulate transactions (null treatment)\n",
    "treatment_converted = np.random.choice([0, 1],size=n_treatment, p=[1 - p_control_treatment_null, p_control_treatment_null])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "601daec9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.614096Z",
     "iopub.status.busy": "2024-11-18T02:41:54.613268Z",
     "iopub.status.idle": "2024-11-18T02:41:54.619903Z",
     "shell.execute_reply": "2024-11-18T02:41:54.618815Z"
    },
    "papermill": {
     "duration": 0.025845,
     "end_time": "2024-11-18T02:41:54.622461",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.596616",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#simulate transactions (null conversion)\n",
    "control_converted = np.random.choice([0, 1], size=n_control, p=[1 - p_control_treatment_null, p_control_treatment_null])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "053b1838",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.657316Z",
     "iopub.status.busy": "2024-11-18T02:41:54.656180Z",
     "iopub.status.idle": "2024-11-18T02:41:54.662390Z",
     "shell.execute_reply": "2024-11-18T02:41:54.661332Z"
    },
    "papermill": {
     "duration": 0.027097,
     "end_time": "2024-11-18T02:41:54.664886",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.637789",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#calculate average conversion rate(treatment)\n",
    "mean_treatment = treatment_converted.mean()\n",
    "\n",
    "#calculate average conversion rate(control)\n",
    "mean_control = control_converted.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a11dc27d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.697174Z",
     "iopub.status.busy": "2024-11-18T02:41:54.696720Z",
     "iopub.status.idle": "2024-11-18T02:41:54.704138Z",
     "shell.execute_reply": "2024-11-18T02:41:54.703138Z"
    },
    "papermill": {
     "duration": 0.026577,
     "end_time": "2024-11-18T02:41:54.706667",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.680090",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.00017753124138805698"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find difference\n",
    "estimate = mean_treatment - mean_control\n",
    "estimate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9afe08ba",
   "metadata": {
    "papermill": {
     "duration": 0.014981,
     "end_time": "2024-11-18T02:41:54.737084",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.722103",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Analysis of conversion rate comparison\n",
    "The conversion rates for the Treatment and the Control groups are nearly identical. The p-value of -.0.002 suggests that there is no statistically significant difference between the two groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "39206e29",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.770104Z",
     "iopub.status.busy": "2024-11-18T02:41:54.769685Z",
     "iopub.status.idle": "2024-11-18T02:41:54.779307Z",
     "shell.execute_reply": "2024-11-18T02:41:54.778310Z"
    },
    "papermill": {
     "duration": 0.029212,
     "end_time": "2024-11-18T02:41:54.781731",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.752519",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "p_diffs = []\n",
    "for _ in range(500):    \n",
    "    # simulate the treatment and control converted arrays    \n",
    "    treatment_conversions = np.random.binomial(n_treatment, p_control_treatment_null)    \n",
    "    control_conversions = np.random.binomial(n_control, p_control_treatment_null)         \n",
    "    \n",
    "    # calculate p_treatment and p_control under the null    \n",
    "    p_treatment = treatment_conversions / n_treatment    \n",
    "    p_control = control_conversions / n_control        \n",
    "    \n",
    "    # calculate the difference between p_treatment_null and p_control_null    \n",
    "    p_diff = p_treatment - p_control        \n",
    "    \n",
    "    # add p_diff to the p_diffs array    \n",
    "    p_diffs.append(p_diff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f8485569",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:54.814588Z",
     "iopub.status.busy": "2024-11-18T02:41:54.813777Z",
     "iopub.status.idle": "2024-11-18T02:41:55.106053Z",
     "shell.execute_reply": "2024-11-18T02:41:55.104734Z"
    },
    "papermill": {
     "duration": 0.311423,
     "end_time": "2024-11-18T02:41:55.108538",
     "exception": false,
     "start_time": "2024-11-18T02:41:54.797115",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiYAAAGdCAYAAAAmK7htAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAApUUlEQVR4nO3de3hUdX7H8c+EDJNEmASCIQESwcrVC2xhAxHbFQxkERWWVFRoi5TFdY0uEnVLnkVJWLdc1iKrDYgtG7U1xYVWFLnEFHdRJNyibLkookK5hEQFk3CRyUB+/WOfzBITQiaZYX4Z36/nmQfnd37nzPc7M5l8PHNOjsMYYwQAAGCBiFAXAAAAUIdgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwRmSoC/i22tpalZWVqWPHjnI4HKEuBwAANIMxRqdOnVK3bt0UEdHy/R7WBZOysjIlJyeHugwAANACR44cUY8ePVq8vnXBpGPHjpL+1Jjb7W5yrtfr1dtvv63Ro0fL6XReifJChl7DE72GJ3oNT/TatOrqaiUnJ/t+j7eUdcGk7usbt9vdrGASExMjt9v9nXiT0Gv4odfwRK/hiV6bp7WHYXDwKwAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1IkNdAAAESs9Za4Oy3UPzxwZluwAaYo8JAACwht/B5NixY/rbv/1bxcfHKzo6WjfeeKN27tzpW26M0VNPPaWkpCRFR0crPT1dBw4cCGjRAAAgPPkVTL7++msNHz5cTqdT69ev1759+/TP//zP6tSpk2/OwoUL9dxzz+mFF17Qtm3bdNVVVykjI0Pnzp0LePEAACC8+HWMyYIFC5ScnKyCggLfWK9evXz/bYzR4sWLNXv2bI0bN06S9Morr6hr165avXq17r333gCVDQAAwpFfweTNN99URkaG7r77bm3atEndu3fXQw89pOnTp0uSDh48qPLycqWnp/vWiY2N1dChQ1VSUtJoMPF4PPJ4PL771dXVkiSv1yuv19tkPXXLLzcvHNBreKLXwHK1M0HZrr8187qGJ3pt3jqt5TDGNPsnOSoqSpKUnZ2tu+++Wzt27NCMGTP0wgsvaMqUKdqyZYuGDx+usrIyJSUl+dabOHGiHA6HXnvttQbbzM3NVV5eXoPxwsJCxcTEtKQnAABwhZ09e1aTJk1SVVWV3G53i7fjVzBp3769hgwZoi1btvjGfvazn2nHjh0qKSlpUTBpbI9JcnKyvvrqq8s25vV6VVxcrFGjRsnpdDa3jTaJXsMTvQbWDblFQdnuntwMv+bzuoYnem1adXW1unTp0upg4tdXOUlJSRowYEC9sf79++u//uu/JEmJiYmSpIqKinrBpKKiQoMGDWp0my6XSy6Xq8G40+ls9pPhz9y2jl7DE70GhueCIyjbbWm9vK7hiV4vPTcQ/DorZ/jw4dq/f3+9sU8++UTXXHONpD8dCJuYmKiNGzf6lldXV2vbtm1KS0sLQLkAACCc+bXHZObMmbr55pv1T//0T5o4caK2b9+uF198US+++KIkyeFw6NFHH9XTTz+t3r17q1evXnryySfVrVs3jR8/Phj1AwCAMOJXMPn+97+v119/XTk5OZo7d6569eqlxYsXa/Lkyb45P//5z3XmzBk98MADqqys1C233KINGzb4DpwFAAC4FL+vlXPHHXfojjvuuORyh8OhuXPnau7cua0qDAAAfPdwrRwAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGpGhLgCAnXrOWhvQ7bnaGS1MDegmAYQh9pgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBpcKwdowwJ9PRsACDX2mAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1/Aomubm5cjgc9W79+vXzLT937pyysrIUHx+vDh06KDMzUxUVFQEvGgAAhCe/95hcf/31On78uO+2efNm37KZM2dqzZo1WrlypTZt2qSysjJNmDAhoAUDAIDwFen3CpGRSkxMbDBeVVWl5cuXq7CwUCNHjpQkFRQUqH///tq6dauGDRvW+moBAEBY8zuYHDhwQN26dVNUVJTS0tI0b948paSkqLS0VF6vV+np6b65/fr1U0pKikpKSi4ZTDwejzwej+9+dXW1JMnr9crr9TZZS93yy80LB/Qanlrbq6udCWQ5QeWK+FOtwXxdg/V8+Fsz7+HwRK/NW6e1HMaYZv8kr1+/XqdPn1bfvn11/Phx5eXl6dixY9qzZ4/WrFmjqVOn1gsZkpSamqoRI0ZowYIFjW4zNzdXeXl5DcYLCwsVExPjZzsAACAUzp49q0mTJqmqqkput7vF2/ErmHxbZWWlrrnmGi1atEjR0dEtCiaN7TFJTk7WV199ddnGvF6viouLNWrUKDmdzpa20SbQa3hqba835BYFoargcEUY/XJIrZ7cGSFPrSPU5fhlT26GX/N5D4cnem1adXW1unTp0upg4vdXOReLi4tTnz599Omnn2rUqFGqqalRZWWl4uLifHMqKioaPSaljsvlksvlajDudDqb/WT4M7eto9fw1NJePRfa1i94SfLUOtpc3S19H/IeDk/0eum5gdCqv2Ny+vRpffbZZ0pKStLgwYPldDq1ceNG3/L9+/fr8OHDSktLa3WhAAAg/Pm1x+Txxx/XnXfeqWuuuUZlZWWaM2eO2rVrp/vuu0+xsbGaNm2asrOz1blzZ7ndbj3yyCNKS0vjjBwAANAsfgWTo0eP6r777tOJEyd09dVX65ZbbtHWrVt19dVXS5KeffZZRUREKDMzUx6PRxkZGVqyZElQCgcAAOHHr2CyYsWKJpdHRUUpPz9f+fn5rSoKAAB8N3GtHAAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGu0KpjMnz9fDodDjz76qG/s3LlzysrKUnx8vDp06KDMzExVVFS0tk4AAPAd0OJgsmPHDi1btkw33XRTvfGZM2dqzZo1WrlypTZt2qSysjJNmDCh1YUCAIDw16Jgcvr0aU2ePFn/+q//qk6dOvnGq6qqtHz5ci1atEgjR47U4MGDVVBQoC1btmjr1q0BKxoAAISnyJaslJWVpbFjxyo9PV1PP/20b7y0tFRer1fp6em+sX79+iklJUUlJSUaNmxYg215PB55PB7f/erqakmS1+uV1+ttso665ZebFw7oNTy1tldXOxPIcoLKFWHq/duW+Pv68B4OT/TavHVay+9gsmLFCn3wwQfasWNHg2Xl5eVq37694uLi6o137dpV5eXljW5v3rx5ysvLazD+9ttvKyYmplk1FRcXN2teOKDX8NTSXhemBriQK+CXQ2pDXYLf1q1b16L1eA+HJ3pt3NmzZwPymH4FkyNHjmjGjBkqLi5WVFRUQArIyclRdna27351dbWSk5M1evRoud3uJtf1er0qLi7WqFGj5HQ6A1KPreg1PLW21xtyi4JQVXC4Iox+OaRWT+6MkKfWEepy/LInN8Ov+byHwxO9Nq3uG4/W8iuYlJaW6osvvtBf/uVf+sYuXLigd999V//yL/+ioqIi1dTUqLKyst5ek4qKCiUmJja6TZfLJZfL1WDc6XQ2+8nwZ25bR6/hqaW9ei60rV/wkuSpdbS5ulv6PuQ9HJ7o9dJzA8GvYHLbbbdp9+7d9camTp2qfv366R//8R+VnJwsp9OpjRs3KjMzU5K0f/9+HT58WGlpaQEpGAAAhC+/gknHjh11ww031Bu76qqrFB8f7xufNm2asrOz1blzZ7ndbj3yyCNKS0tr9MBXAACAi7XorJymPPvss4qIiFBmZqY8Ho8yMjK0ZMmSQD8MAAAIQ60OJn/4wx/q3Y+KilJ+fr7y8/Nbu2kgbPSctbbRcVc7o4WpfzqIta0ddwEAwcC1cgAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWCPgpwsDQLi51FlVl2LL2VaH5o8N2WMDLcUeEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2vloM3x97olzcV1RQAg9NhjAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAa0SGugDAFj1nrQ11CQDwncceEwAAYA2/gsnSpUt10003ye12y+12Ky0tTevXr/ctP3funLKyshQfH68OHTooMzNTFRUVAS8aAACEJ7+CSY8ePTR//nyVlpZq586dGjlypMaNG6e9e/dKkmbOnKk1a9Zo5cqV2rRpk8rKyjRhwoSgFA4AAMKPX8eY3HnnnfXu/+pXv9LSpUu1detW9ejRQ8uXL1dhYaFGjhwpSSooKFD//v21detWDRs2LHBVAwCAsNTig18vXLiglStX6syZM0pLS1Npaam8Xq/S09N9c/r166eUlBSVlJRcMph4PB55PB7f/erqakmS1+uV1+ttsoa65ZebFw7o9c9c7cyVLCeoXBGm3r/hjF6vvCvxecFnU3hqSa+Bel4cxhi/fnJ2796ttLQ0nTt3Th06dFBhYaFuv/12FRYWaurUqfVChiSlpqZqxIgRWrBgQaPby83NVV5eXoPxwsJCxcTE+FMaAAAIkbNnz2rSpEmqqqqS2+1u8Xb83mPSt29f7dq1S1VVVVq1apWmTJmiTZs2tbiAnJwcZWdn++5XV1crOTlZo0ePvmxjXq9XxcXFGjVqlJxOZ4traAvo9c9uyC0KQVXB4Yow+uWQWj25M0KeWkeoywkqer3y9uRmBP0x+GwKTy3pte4bj9byO5i0b99e1113nSRp8ODB2rFjh37zm9/onnvuUU1NjSorKxUXF+ebX1FRocTExEtuz+VyyeVyNRh3Op3NfjL8mdvW0avkuRB+v9Q8tY6w7Ksx9HrlXMnPCj6bwpO/v4sDodV/x6S2tlYej0eDBw+W0+nUxo0bfcv279+vw4cPKy0trbUPAwAAvgP82mOSk5OjMWPGKCUlRadOnVJhYaH+8Ic/qKioSLGxsZo2bZqys7PVuXNnud1uPfLII0pLS+OMHAAA0Cx+BZMvvvhCf//3f6/jx48rNjZWN910k4qKijRq1ChJ0rPPPquIiAhlZmbK4/EoIyNDS5YsCUrhAAAg/PgVTJYvX97k8qioKOXn5ys/P79VRQEAgO8mrpUDAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANbw60/SAwDajp6z1gZlu4fmjw3KdgGJPSYAAMAiBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWMOvYDJv3jx9//vfV8eOHZWQkKDx48dr//799eacO3dOWVlZio+PV4cOHZSZmamKioqAFg0AAMKTX8Fk06ZNysrK0tatW1VcXCyv16vRo0frzJkzvjkzZ87UmjVrtHLlSm3atEllZWWaMGFCwAsHAADhJ9KfyRs2bKh3/6WXXlJCQoJKS0v113/916qqqtLy5ctVWFiokSNHSpIKCgrUv39/bd26VcOGDQtc5QAAIOz4FUy+raqqSpLUuXNnSVJpaam8Xq/S09N9c/r166eUlBSVlJQ0Gkw8Ho88Ho/vfnV1tSTJ6/XK6/U2+fh1yy83LxzQ65+52pkrWU5QuSJMvX/DGb2Gj4t/NvlsCk8t6TVQz4vDGNOin5za2lrdddddqqys1ObNmyVJhYWFmjp1ar2gIUmpqakaMWKEFixY0GA7ubm5ysvLazBeWFiomJiYlpQGAACusLNnz2rSpEmqqqqS2+1u8XZavMckKytLe/bs8YWSlsrJyVF2drbvfnV1tZKTkzV69OjLNub1elVcXKxRo0bJ6XS2qg7b0euf3ZBbFIKqgsMVYfTLIbV6cmeEPLWOUJcTVPQaPvbkZvj+m8+m8NSSXuu+8WitFgWThx9+WG+99Zbeffdd9ejRwzeemJiompoaVVZWKi4uzjdeUVGhxMTERrflcrnkcrkajDudzmY/Gf7MbevoVfJcCL8Pek+tIyz7agy9tn2N/Vzy2RSe/P1dHAh+nZVjjNHDDz+s119/Xe+884569epVb/ngwYPldDq1ceNG39j+/ft1+PBhpaWlBaRgAAAQvvzaY5KVlaXCwkK98cYb6tixo8rLyyVJsbGxio6OVmxsrKZNm6bs7Gx17txZbrdbjzzyiNLS0jgjBwAAXJZfwWTp0qWSpFtvvbXeeEFBge6//35J0rPPPquIiAhlZmbK4/EoIyNDS5YsCUixAAAgvPkVTJpzAk9UVJTy8/OVn5/f4qIAAMB3E9fKAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABr+PUn6YHm6jlrbYvXdbUzWpgq3ZBbFJaXjAcAXBp7TAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDa6VAwDwy8XXwgr0ta0OzR/b6m2gbWOPCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALAGwQQAAFiDYAIAAKxBMAEAANYgmAAAAGsQTAAAgDUIJgAAwBp+B5N3331Xd955p7p16yaHw6HVq1fXW26M0VNPPaWkpCRFR0crPT1dBw4cCFS9AAAgjPkdTM6cOaOBAwcqPz+/0eULFy7Uc889pxdeeEHbtm3TVVddpYyMDJ07d67VxQIAgPAW6e8KY8aM0ZgxYxpdZozR4sWLNXv2bI0bN06S9Morr6hr165avXq17r333tZVCwAAwlpAjzE5ePCgysvLlZ6e7huLjY3V0KFDVVJSEsiHAgAAYcjvPSZNKS8vlyR17dq13njXrl19y77N4/HI4/H47ldXV0uSvF6vvF5vk49Xt/xy88JBW+vV1c60fN0IU+/fcEav4YleW87mz7i29jncGi3pNVDPi8MY0+J3k8Ph0Ouvv67x48dLkrZs2aLhw4errKxMSUlJvnkTJ06Uw+HQa6+91mAbubm5ysvLazBeWFiomJiYlpYGAACuoLNnz2rSpEmqqqqS2+1u8XYCusckMTFRklRRUVEvmFRUVGjQoEGNrpOTk6Ps7Gzf/erqaiUnJ2v06NGXbczr9aq4uFijRo2S0+lsfQMWa2u93pBb1OJ1XRFGvxxSqyd3RshT6whgVfah1/BEry23JzcjAFUFR1v7HG6NlvRa941HawU0mPTq1UuJiYnauHGjL4hUV1dr27Zt+ulPf9roOi6XSy6Xq8G40+ls9pPhz9y2rq306rnQ+g8oT60jINtpC+g1PNGr/9rC51tb+RwOBH9/FweC38Hk9OnT+vTTT333Dx48qF27dqlz585KSUnRo48+qqefflq9e/dWr1699OSTT6pbt26+r3sAAAAuxe9gsnPnTo0YMcJ3v+5rmClTpuill17Sz3/+c505c0YPPPCAKisrdcstt2jDhg2KiooKXNUAACAs+R1Mbr31VjV1vKzD4dDcuXM1d+7cVhWGK6PnrLWhLgEAfIL1mXRo/tigbBeBx7VyAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYI6B/YA0AABsF4mwfVzujhal/+svWF/8xOc74CSz2mAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWiAx1Abi8nrPWytXOaGGqdENukTwXHKEuCQCAoGCPCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANQgmAADAGgQTAABgDYIJAACwBsEEAABYg2ACAACsQTABAADWIJgAAABrEEwAAIA1CCYAAMAaBBMAAGANggkAALBGZKgLuNJ6zlob6hIAALisYP6+OjR/bNC23VrsMQEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYI3v3Fk5AAAEEmd7BlbQ9pjk5+erZ8+eioqK0tChQ7V9+/ZgPRQAAAgTQQkmr732mrKzszVnzhx98MEHGjhwoDIyMvTFF18E4+EAAECYCEowWbRokaZPn66pU6dqwIABeuGFFxQTE6Pf/va3wXg4AAAQJgJ+jElNTY1KS0uVk5PjG4uIiFB6erpKSkoazPd4PPJ4PL77VVVVkqSTJ0/K6/U2+Vher1dnz57ViRMn5HQ6m1Vf5PkzzZpnm8hao7NnaxXpjdCFWkeoywkqeg1P9Bqe6LVtOnHiRJPLW/L79dSpU5IkY0zrijMBduzYMSPJbNmypd74E088YVJTUxvMnzNnjpHEjRs3bty4cQuD25EjR1qVI0J+Vk5OTo6ys7N992tra3Xy5EnFx8fL4Wg6kVZXVys5OVlHjhyR2+0OdqkhRa/hiV7DE72GJ3ptmjFGp06dUrdu3Vr12AEPJl26dFG7du1UUVFRb7yiokKJiYkN5rtcLrlcrnpjcXFxfj2m2+0O+zdJHXoNT/Qanug1PNHrpcXGxrb6MQN+8Gv79u01ePBgbdy40TdWW1urjRs3Ki0tLdAPBwAAwkhQvsrJzs7WlClTNGTIEKWmpmrx4sU6c+aMpk6dGoyHAwAAYSIoweSee+7Rl19+qaeeekrl5eUaNGiQNmzYoK5duwb0cVwul+bMmdPgq6BwRK/hiV7DE72GJ3q9MhzGtPa8HgAAgMDgIn4AAMAaBBMAAGANggkAALAGwQQAAFjDqmBy8uRJTZ48WW63W3FxcZo2bZpOnz7d5Drnzp1TVlaW4uPj1aFDB2VmZjb4426HDx/W2LFjFRMTo4SEBD3xxBM6f/58vTmvvvqqBg4cqJiYGCUlJekf/uEfLnstgdYIZa8ej0e/+MUvdM0118jlcqlnz55BvcBiKHut8/777ysyMlKDBg0KVFuNClWv//3f/61Ro0bp6quvltvtVlpamoqKigLaW35+vnr27KmoqCgNHTpU27dvb3L+ypUr1a9fP0VFRenGG2/UunXr6i03xuipp55SUlKSoqOjlZ6ergMHDtSb05LnMxCudK+HDh3StGnT1KtXL0VHR+sv/uIvNGfOHNXU1ASlv28LxWtbx+PxaNCgQXI4HNq1a1egWmpUqPpcu3athg4dqujoaHXq1Enjx48PZFuXFIp+P/nkE40bN05dunSR2+3WLbfcot///vf+Fd6qP2gfYD/84Q/NwIEDzdatW817771nrrvuOnPfffc1uc6DDz5okpOTzcaNG83OnTvNsGHDzM033+xbfv78eXPDDTeY9PR08+GHH5p169aZLl26mJycHN+czZs3m4iICPOb3/zGfP755+a9994z119/vfnRj34Udr0aY8xdd91lhg4daoqLi83BgwfNli1bzObNm4PSpzGh7dUYY77++mtz7bXXmtGjR5uBAwcGur16QtXrjBkzzIIFC8z27dvNJ598YnJycozT6TQffPBBQPpasWKFad++vfntb39r9u7da6ZPn27i4uJMRUVFo/Pff/99065dO7Nw4UKzb98+M3v2bON0Os3u3bt9c+bPn29iY2PN6tWrzR//+Edz1113mV69eplvvvnGN6clz2db7HX9+vXm/vvvN0VFReazzz4zb7zxhklISDCPPfZYUHsNVb8X+9nPfmbGjBljJJkPP/wwWG2GrM9Vq1aZTp06maVLl5r9+/ebvXv3mtdeey1ofYa63969e5vbb7/d/PGPfzSffPKJeeihh0xMTIw5fvx4s2u3Jpjs27fPSDI7duzwja1fv944HA5z7NixRteprKw0TqfTrFy50jf20UcfGUmmpKTEGGPMunXrTEREhCkvL/fNWbp0qXG73cbj8RhjjPn1r39trr322nrbfu6550z37t0D1t/FQtnr+vXrTWxsrDlx4kQwWmsglL3Wueeee8zs2bPNnDlzghpMbOj1YgMGDDB5eXmtbcsYY0xqaqrJysry3b9w4YLp1q2bmTdvXqPzJ06caMaOHVtvbOjQoeYnP/mJMcaY2tpak5iYaH7961/7lldWVhqXy2X+8z//0xjTsuczEELRa2MWLlxoevXq1ZpWmiWU/a5bt87069fP7N27N+jBJBR9er1e0717d/Nv//ZvgW7nskLR75dffmkkmXfffdc3p7q62kgyxcXFza7dmq9ySkpKFBcXpyFDhvjG0tPTFRERoW3btjW6Tmlpqbxer9LT031j/fr1U0pKikpKSnzbvfHGG+v9cbeMjAxVV1dr7969kqS0tDQdOXJE69atkzFGFRUVWrVqlW6//fZgtBrSXt98800NGTJECxcuVPfu3dWnTx89/vjj+uabb4LRakh7laSCggJ9/vnnmjNnTqBbayDUvV6strZWp06dUufOnVvdV01NjUpLS+vVGBERofT0dF+N31ZSUlJvfl3NdfMPHjyo8vLyenNiY2M1dOjQen37+3y2Vqh6bUxVVVVAXr+mhLLfiooKTZ8+Xf/+7/+umJiYQLbVQKj6/OCDD3Ts2DFFREToe9/7npKSkjRmzBjt2bMn0C3WE6p+4+Pj1bdvX73yyis6c+aMzp8/r2XLlikhIUGDBw9udv3WBJPy8nIlJCTUG4uMjFTnzp1VXl5+yXXat2/f4KJ/Xbt29a1TXl7e4C/O1t2vmzN8+HC9+uqruueee9S+fXslJiYqNjZW+fn5gWit0bpD1evnn3+uzZs3a8+ePXr99de1ePFirVq1Sg899FAgWmu07lD1euDAAc2aNUv/8R//ocjI4F9IO5S9ftszzzyj06dPa+LEiS1ppZ6vvvpKFy5caLSGpvpqan7dv5eb4+/z2Vqh6vXbPv30Uz3//PP6yU9+0qI+mitU/RpjdP/99+vBBx+sFzyDJVR9fv7555Kk3NxczZ49W2+99ZY6deqkW2+9VSdPnmx9Y5cQqn4dDof+53/+Rx9++KE6duyoqKgoLVq0SBs2bFCnTp2aXX/Qg8msWbPkcDiavH388cfBLqNJ+/bt04wZM/TUU0+ptLRUGzZs0KFDh/Tggw/6tZ220Gttba0cDodeffVVpaam6vbbb9eiRYv08ssv+7XXxPZeL1y4oEmTJikvL099+vRp1bZs7/XbCgsLlZeXp9/97ncNfrHDfseOHdMPf/hD3X333Zo+fXqoywmK559/XqdOnVJOTk6oSwmq2tpaSdIvfvELZWZmavDgwSooKJDD4dDKlStDXF3gGWOUlZWlhIQEvffee9q+fbvGjx+vO++8U8ePH2/2doL+v5GPPfaY7r///ibnXHvttUpMTNQXX3xRb/z8+fM6efKkEhMTG10vMTFRNTU1qqysrPd/nBUVFb51EhMTGxyJXHfGQ92cefPmafjw4XriiSckSTfddJOuuuoq/dVf/ZWefvppJSUlhU2vSUlJ6t69e71LU/fv31/GGB09elS9e/cOi15PnTqlnTt36sMPP9TDDz8s6U8fEsYYRUZG6u2339bIkSPDoteLrVixQj/+8Y+1cuXKBrtlW6pLly5q165dgzOFLq7x2xITE5ucX/dvRUVFvZ+viooK35lTLXk+WytUvdYpKyvTiBEjdPPNN+vFF19sbTuXFap+33nnHZWUlDS4DsuQIUM0efJkvfzyy63q69tC1Wfd+IABA3zLXS6Xrr32Wh0+fLh1TTUhlK/rW2+9pa+//lput1uStGTJEhUXF+vll1/WrFmzmtdAs49GCbK6A9127tzpGysqKmrWgYOrVq3yjX388ceNHjh48ZHIy5YtM26325w7d84YY8yECRPMxIkT6217y5YtRlJQDrILZa/Lli0z0dHR5tSpU745q1evNhEREebs2bMB7dOY0PV64cIFs3v37nq3n/70p6Zv375m9+7d5vTp02HTa53CwkITFRVlVq9eHejWTGpqqnn44Yd99y9cuGC6d+/e5IF0d9xxR72xtLS0BgfSPfPMM77lVVVVjR786s/zGQih6NUYY44ePWp69+5t7r33XnP+/PlAttSkUPT7f//3f/V+NouKiowks2rVKnPkyJFAt2iMCU2fdfcvPvi1pqbGJCQkmGXLlgWst8aEot8333zTRERE1Pv9Yowxffr0Mb/61a+aXbs1wcSYP50a+L3vfc9s27bNbN682fTu3bveqYFHjx41ffv2Ndu2bfONPfjggyYlJcW88847ZufOnSYtLc2kpaX5ltedajl69Giza9cus2HDBnP11VfXO9WyoKDAREZGmiVLlpjPPvvMbN682QwZMsSkpqaGXa+nTp0yPXr0MH/zN39j9u7dazZt2mR69+5tfvzjH4ddr98W7LNyjAldr6+++qqJjIw0+fn55vjx475bZWVlQPpasWKFcblc5qWXXjL79u0zDzzwgImLi/OdKfR3f/d3ZtasWb7577//vomMjDTPPPOM+eijj8ycOXMaPfUwLi7OvPHGG+Z///d/zbhx4xo9Xbip5zMYQtHr0aNHzXXXXWduu+02c/To0XqvYbCF6rW92MGDB6/I6cKh6HPGjBmme/fupqioyHz88cdm2rRpJiEhwZw8eTJovYaq3y+//NLEx8ebCRMmmF27dpn9+/ebxx9/3DidTrNr165m125VMDlx4oS57777TIcOHYzb7TZTp06tl7zq3ry///3vfWPffPONeeihh0ynTp1MTEyM+dGPftTgh/nQoUNmzJgxJjo62nTp0sU89thjxuv11pvz3HPPmQEDBpjo6GiTlJRkJk+ebI4ePRqWvX700UcmPT3dREdHmx49epjs7Oyg7C2xodeLXYlgEqpef/CDHxhJDW5TpkwJWG/PP/+8SUlJMe3btzepqalm69at9R7/24/1u9/9zvTp08e0b9/eXH/99Wbt2rX1ltfW1ponn3zSdO3a1bhcLnPbbbeZ/fv315tzueczWK50rwUFBY2+fldqp3YoXtuLXYlgYkxo+qypqTGPPfaYSUhIMB07djTp6elmz549QevxYqHod8eOHWb06NGmc+fOpmPHjmbYsGFm3bp1ftXtMMaY5n3pAwAAEFzWnC4MAABAMAEAANYgmAAAAGsQTAAAgDUIJgAAwBoEEwAAYA2CCQAAsAbBBAAAWINgAgAArEEwAQAA1iCYAAAAaxBMAACANf4fUJG08YElV70AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#create histogram of p_diffs\n",
    "\n",
    "p_diffs = pd.Series(p_diffs)\n",
    "p_diffs.hist(bins=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "70950450",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:55.142092Z",
     "iopub.status.busy": "2024-11-18T02:41:55.141656Z",
     "iopub.status.idle": "2024-11-18T02:41:55.186355Z",
     "shell.execute_reply": "2024-11-18T02:41:55.185134Z"
    },
    "papermill": {
     "duration": 0.064467,
     "end_time": "2024-11-18T02:41:55.188845",
     "exception": false,
     "start_time": "2024-11-18T02:41:55.124378",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate new difference in proportions from df\n",
    "new_treatment = df[df['group'] == 'treatment']['converted'].mean()\n",
    "new_control = df[df['group'] == 'control']['converted'].mean()\n",
    "new_diff = new_treatment - new_control\n",
    "\n",
    "#calculate proportion of p_diff > new difference\n",
    "proportion_greater = np.mean(p_diffs > new_diff)\n",
    "\n",
    "proportion_greater"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d182948",
   "metadata": {
    "papermill": {
     "duration": 0.015849,
     "end_time": "2024-11-18T02:41:55.220783",
     "exception": false,
     "start_time": "2024-11-18T02:41:55.204934",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Part IV - Regression Analysis for A/B Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "58d2bdac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:55.254338Z",
     "iopub.status.busy": "2024-11-18T02:41:55.253866Z",
     "iopub.status.idle": "2024-11-18T02:41:55.278609Z",
     "shell.execute_reply": "2024-11-18T02:41:55.277503Z"
    },
    "papermill": {
     "duration": 0.044575,
     "end_time": "2024-11-18T02:41:55.281280",
     "exception": false,
     "start_time": "2024-11-18T02:41:55.236705",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>group</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UK</td>\n",
       "      <td>control</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>US</td>\n",
       "      <td>treatment</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UK</td>\n",
       "      <td>treatment</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>UK</td>\n",
       "      <td>control</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>UK</td>\n",
       "      <td>treatment</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country      group  converted  intercept  ab_page\n",
       "0      UK    control          0          1        0\n",
       "1      US  treatment          1          1        1\n",
       "2      UK  treatment          0          1        1\n",
       "3      UK    control          0          1        0\n",
       "4      UK  treatment          0          1        1"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#add intercept column\n",
    "df['intercept'] = 1\n",
    "\n",
    "#create new binary column (1=treatment, 0=control) \n",
    "df['ab_page'] = pd.get_dummies(df['group'])['treatment'].astype(int)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a96a2368",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:55.316325Z",
     "iopub.status.busy": "2024-11-18T02:41:55.315436Z",
     "iopub.status.idle": "2024-11-18T02:41:55.323991Z",
     "shell.execute_reply": "2024-11-18T02:41:55.322955Z"
    },
    "papermill": {
     "duration": 0.028799,
     "end_time": "2024-11-18T02:41:55.326483",
     "exception": false,
     "start_time": "2024-11-18T02:41:55.297684",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#create x matrix\n",
    "X = df[['intercept', 'ab_page']]\n",
    "\n",
    "#create y response column\n",
    "y = df['converted']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b6cda8d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:55.360808Z",
     "iopub.status.busy": "2024-11-18T02:41:55.360358Z",
     "iopub.status.idle": "2024-11-18T02:41:58.880700Z",
     "shell.execute_reply": "2024-11-18T02:41:58.879093Z"
    },
    "papermill": {
     "duration": 3.542608,
     "end_time": "2024-11-18T02:41:58.885254",
     "exception": false,
     "start_time": "2024-11-18T02:41:55.342646",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.384516\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "#import statsmodels to fit regression model\n",
    "import statsmodels.api as sm\n",
    "logit_mod = sm.Logit(y, X)\n",
    "logit_res = logit_mod.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c5cd4b92",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:58.956518Z",
     "iopub.status.busy": "2024-11-18T02:41:58.955211Z",
     "iopub.status.idle": "2024-11-18T02:41:59.344153Z",
     "shell.execute_reply": "2024-11-18T02:41:59.342609Z"
    },
    "papermill": {
     "duration": 0.429388,
     "end_time": "2024-11-18T02:41:59.348844",
     "exception": false,
     "start_time": "2024-11-18T02:41:58.919456",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           Logit Regression Results                           \n",
      "==============================================================================\n",
      "Dep. Variable:              converted   No. Observations:                69889\n",
      "Model:                          Logit   Df Residuals:                    69887\n",
      "Method:                           MLE   Df Model:                            1\n",
      "Date:                Mon, 18 Nov 2024   Pseudo R-squ.:                0.007175\n",
      "Time:                        02:41:59   Log-Likelihood:                -26873.\n",
      "converged:                       True   LL-Null:                       -27068.\n",
      "Covariance Type:            nonrobust   LLR p-value:                 1.810e-86\n",
      "==============================================================================\n",
      "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "intercept     -2.1402      0.017   -122.305      0.000      -2.174      -2.106\n",
      "ab_page        0.4467      0.023     19.539      0.000       0.402       0.492\n",
      "==============================================================================\n"
     ]
    }
   ],
   "source": [
    "#get summary\n",
    "print(logit_res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af35cd1b",
   "metadata": {
    "papermill": {
     "duration": 0.033433,
     "end_time": "2024-11-18T02:41:59.417045",
     "exception": false,
     "start_time": "2024-11-18T02:41:59.383612",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Create two new columns as dummy variables for `US` and `UK`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "351a7fb1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:59.469070Z",
     "iopub.status.busy": "2024-11-18T02:41:59.467858Z",
     "iopub.status.idle": "2024-11-18T02:41:59.493608Z",
     "shell.execute_reply": "2024-11-18T02:41:59.492376Z"
    },
    "papermill": {
     "duration": 0.046445,
     "end_time": "2024-11-18T02:41:59.496073",
     "exception": false,
     "start_time": "2024-11-18T02:41:59.449628",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>group</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>US</th>\n",
       "      <th>UK</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UK</td>\n",
       "      <td>control</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>US</td>\n",
       "      <td>treatment</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UK</td>\n",
       "      <td>treatment</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>UK</td>\n",
       "      <td>control</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>UK</td>\n",
       "      <td>treatment</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  country      group  converted  intercept  ab_page  US  UK\n",
       "0      UK    control          0          1        0   0   1\n",
       "1      US  treatment          1          1        1   1   0\n",
       "2      UK  treatment          0          1        1   0   1\n",
       "3      UK    control          0          1        0   0   1\n",
       "4      UK  treatment          0          1        1   0   1"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Create two new columns for dummy variables\n",
    "df[['US', 'UK']] = pd.get_dummies(df['country'])[['US', 'UK']].astype(int)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c19efc43",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:59.532363Z",
     "iopub.status.busy": "2024-11-18T02:41:59.531203Z",
     "iopub.status.idle": "2024-11-18T02:41:59.539297Z",
     "shell.execute_reply": "2024-11-18T02:41:59.538120Z"
    },
    "papermill": {
     "duration": 0.029227,
     "end_time": "2024-11-18T02:41:59.542017",
     "exception": false,
     "start_time": "2024-11-18T02:41:59.512790",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#create x matrix and y response column\n",
    "X = df[['intercept', 'ab_page', 'US', 'UK']]\n",
    "y = df['converted']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "877a0f97",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:59.576942Z",
     "iopub.status.busy": "2024-11-18T02:41:59.576519Z",
     "iopub.status.idle": "2024-11-18T02:41:59.741236Z",
     "shell.execute_reply": "2024-11-18T02:41:59.739845Z"
    },
    "papermill": {
     "duration": 0.186718,
     "end_time": "2024-11-18T02:41:59.745329",
     "exception": false,
     "start_time": "2024-11-18T02:41:59.558611",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.384463\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "#statsmodels to fit regression model\n",
    "logit_mod = sm.Logit(y, X) \n",
    "logit_res = logit_mod.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "cb31e547",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-18T02:41:59.818623Z",
     "iopub.status.busy": "2024-11-18T02:41:59.817868Z",
     "iopub.status.idle": "2024-11-18T02:42:00.226637Z",
     "shell.execute_reply": "2024-11-18T02:42:00.225435Z"
    },
    "papermill": {
     "duration": 0.450305,
     "end_time": "2024-11-18T02:42:00.231062",
     "exception": false,
     "start_time": "2024-11-18T02:41:59.780757",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>  <td> 69889</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>  <td> 69885</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>  <td>     3</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Mon, 18 Nov 2024</td> <th>  Pseudo R-squ.:     </th> <td>0.007312</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>02:42:00</td>     <th>  Log-Likelihood:    </th> <td> -26870.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td> -27068.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th> <td>1.778e-85</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -2.1930</td> <td>    0.053</td> <td>  -41.308</td> <td> 0.000</td> <td>   -2.297</td> <td>   -2.089</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>    0.4466</td> <td>    0.023</td> <td>   19.534</td> <td> 0.000</td> <td>    0.402</td> <td>    0.491</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>    0.0727</td> <td>    0.053</td> <td>    1.372</td> <td> 0.170</td> <td>   -0.031</td> <td>    0.177</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>    0.0067</td> <td>    0.056</td> <td>    0.120</td> <td> 0.905</td> <td>   -0.103</td> <td>    0.117</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}   &    converted     & \\textbf{  No. Observations:  } &    69889    \\\\\n",
       "\\textbf{Model:}           &      Logit       & \\textbf{  Df Residuals:      } &    69885    \\\\\n",
       "\\textbf{Method:}          &       MLE        & \\textbf{  Df Model:          } &        3    \\\\\n",
       "\\textbf{Date:}            & Mon, 18 Nov 2024 & \\textbf{  Pseudo R-squ.:     } &  0.007312   \\\\\n",
       "\\textbf{Time:}            &     02:42:00     & \\textbf{  Log-Likelihood:    } &   -26870.   \\\\\n",
       "\\textbf{converged:}       &       True       & \\textbf{  LL-Null:           } &   -27068.   \\\\\n",
       "\\textbf{Covariance Type:} &    nonrobust     & \\textbf{  LLR p-value:       } & 1.778e-85   \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                   & \\textbf{coef} & \\textbf{std err} & \\textbf{z} & \\textbf{P$> |$z$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{intercept} &      -2.1930  &        0.053     &   -41.308  &         0.000        &       -2.297    &       -2.089     \\\\\n",
       "\\textbf{ab\\_page}  &       0.4466  &        0.023     &    19.534  &         0.000        &        0.402    &        0.491     \\\\\n",
       "\\textbf{US}        &       0.0727  &        0.053     &     1.372  &         0.170        &       -0.031    &        0.177     \\\\\n",
       "\\textbf{UK}        &       0.0067  &        0.056     &     0.120  &         0.905        &       -0.103    &        0.117     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{Logit Regression Results}\n",
       "\\end{center}"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:                69889\n",
       "Model:                          Logit   Df Residuals:                    69885\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Mon, 18 Nov 2024   Pseudo R-squ.:                0.007312\n",
       "Time:                        02:42:00   Log-Likelihood:                -26870.\n",
       "converged:                       True   LL-Null:                       -27068.\n",
       "Covariance Type:            nonrobust   LLR p-value:                 1.778e-85\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -2.1930      0.053    -41.308      0.000      -2.297      -2.089\n",
       "ab_page        0.4466      0.023     19.534      0.000       0.402       0.491\n",
       "US             0.0727      0.053      1.372      0.170      -0.031       0.177\n",
       "UK             0.0067      0.056      0.120      0.905      -0.103       0.117\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get summary\n",
    "logit_res.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38b083dc",
   "metadata": {
    "papermill": {
     "duration": 0.03501,
     "end_time": "2024-11-18T02:42:00.301735",
     "exception": false,
     "start_time": "2024-11-18T02:42:00.266725",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Conclusion/Recommendation to customer\n",
    "The A/B test results show that the Treatment page has a slightly higher conversion rate (12.88%) than the Control page (12.90%), but the difference is negligible with a p-value of -0.002, indicating no statistically significant difference. This suggests that while the Treatment page may have a minor positive effect, location has little impact on conversion rates. Therefore, the Treatment page could be kept, but further optimizations should focus on other areas, such as user experience and testing new variations, to drive more significant improvements."
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "none",
   "dataSources": [
    {
     "datasetId": 6108522,
     "sourceId": 9936320,
     "sourceType": "datasetVersion"
    }
   ],
   "dockerImageVersionId": 30786,
   "isGpuEnabled": false,
   "isInternetEnabled": false,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 12.14676,
   "end_time": "2024-11-18T02:42:01.044763",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-11-18T02:41:48.898003",
   "version": "2.6.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "b34053042c33de92a8f6946323cebca57e40086129f2774c267076f4fb825384"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}