WhatApp DataPlotting.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "import seaborn as sns\n",
    "import operator\n",
    "\n",
    "from datetime import datetime, date, time\n",
    "from dateutil.parser import parse\n",
    "from emoji import UNICODE_EMOJI\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "pd.options.mode.chained_assignment = None\n",
    "\n",
    "#so every plotting made in pandas is done in the style of seaborn.\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Small WhatsApp data analysis notebook\n",
    "\n",
    "\n",
    "This notebook provide the code and functions you can use to make some visualisation and statistics out of the exported discussion. The tools developped work on single discussions as well as group discussions. \n",
    "\n",
    "## HOW TO USE\n",
    "\n",
    "Before starting anything, you will need to export your whatsapp discussion to a .txt file.\n",
    "To do so, go to a whatsapp discussion of your choice, tap on the name of the group/person to access the metadata menu, scroll all the way to the bottom where you should see an \"export discussion\" button. Click on it, be careful to choose __NOT__ to include medias files, then wait. It will produce a .txt file containing the whole discussion. Find a way to retrieve this file on your computer, then move it within the folder containing this notebook.\n",
    "\n",
    "Then make sure to run ALL the cells of this notebook. Take the time to read each cell of text, and finally go to the last cell in which you will type the few commands needed in order to load the data. Once this is done, add as much cells as you want below the last one and call the functions you want to use. You are not obligated to read the code, but please take the time to read the comments at the beginning of each function so you understand how to use them and what purposes they serve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing functions\n",
    "\n",
    "Functions which are used to preprocess the data into a more easily computable for statistics format. Please, don't touch anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ios = False\n",
    "\n",
    "#structure of the message beginning\n",
    "begin_msg_appareance = '[DD.MM.YY HH:MM:SS] '\n",
    "begin_msg_reg = r'\\[[0-9][0-9].[0-9][0-9].[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\\] '\n",
    "regex = r'\\[(\\w.*)\\] (\\w.*?): (.*)'\n",
    "alt_reg = r'\\[(\\w.*)\\] (.*)?'\n",
    "date_format = '%d.%m.%y %H:%M:%S'\n",
    "\n",
    "if not ios:\n",
    "    begin_msg_appareance = 'DD.MM.YY à HH:MM - '\n",
    "    begin_msg_reg = r'[0-9][0-9].[0-9][0-9].[0-9][0-9] à [0-9][0-9]:[0-9][0-9] \\- '\n",
    "    regex = r'([\\w\\.]* à [\\w\\:]*) \\- ([\\w\\+\\ ]*): (.*)'\n",
    "    alt_reg = r'([^-]*) \\- (.*)?'\n",
    "    date_format = '%d.%m.%y à %H:%M'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def flatten_text(lines):\n",
    "    res = []\n",
    "    for l in lines:\n",
    "        #weird unicode character appearing whenever there is an \"image absente\" string in the text, apparently triggers a newline.\n",
    "        msg = l.replace('\\u200e', '')\n",
    "        date_zone = msg[:len(begin_msg_appareance)]\n",
    "        if re.match(begin_msg_reg, date_zone) is None:\n",
    "            if len(res) > 0:\n",
    "                res[-1] = res[-1].strip() + \" \" + msg \n",
    "            \n",
    "        else:\n",
    "            res.append(msg)\n",
    "    return res\n",
    "        \n",
    "\n",
    "def extract_infos(text):\n",
    "    if text is not '': \n",
    "        match = re.match(regex, text)\n",
    "        try:\n",
    "            result = [x for x in match.groups() if x and x!=text]\n",
    "            return result\n",
    "        except AttributeError:\n",
    "            #in group system message, we need to match otherwise as there is no <name>:<msg>, but <name with action> \n",
    "            \n",
    "            match = re.match(alt_reg, text)\n",
    "            try:\n",
    "                result = [x for x in match.groups() if x and x != text]\n",
    "                return [result[0], 'System', '']\n",
    "            except AttributeError:\n",
    "                print('String causing the error : %s'%(text))\n",
    "                return ['','','']\n",
    "            \n",
    "            \n",
    "def wa_data_proc(filename):\n",
    "    '''Parse and clean the whatsapp data (without media files) \n",
    "       into a structured panda dataframe\n",
    "        param : filename => relative file path of the \n",
    "                .txt file containing the messages log\n",
    "    '''\n",
    "    with open(filename, 'r') as f:\n",
    "        mainDf = pd.DataFrame([extract_infos(x) for x in flatten_text(f.readlines())], columns=['date', 'from', 'msg'])\n",
    "        date_format = '%d.%m.%y %H:%M:%S'\n",
    "        mainDf['date'] = mainDf.date.apply(lambda t: datetime.strptime(t, date_format))\n",
    "        mainDf['Year'] = mainDf.date.apply(lambda x: x.year)\n",
    "        mainDf['Month'] = mainDf.date.apply(lambda x: x.month)\n",
    "        mainDf['Day'] = mainDf.date.apply(lambda x: x.day)\n",
    "        return mainDf\n",
    "    \n",
    "def change_name(data, old_name, new_name):\n",
    "    '''Allow to change the name of one contact in the\n",
    "       conversation.\n",
    "    '''\n",
    "    data['from'] = data['from'].apply(lambda name: new_name if (name == old_name) else name )\n",
    "    return data\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting functions\n",
    "\n",
    "Functions which will visualise things like the distribution of message through time, or the most talkative person in a discussion. All of those functions need to have the dataframe of the discussion given as argument (the *data* argument you can see on each function's definition) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "months = ['Jan', 'Fev', 'Mar', 'Avr', 'Mai', 'Jun', 'Jul', 'Aou', 'Sep', 'Oct', 'Nov', 'Dec']\n",
    "\n",
    "def plot_msg_in_years(data, year_begin=None, year_end=None, years_cutout=[], plot_kind='line'):\n",
    "    '''\n",
    "    Probably the most fun plotting function, will plot the distribution of number of message per \n",
    "    month through the year, starting at year \"year_begin\" and ending at year \"year_end\" included. \n",
    "    \n",
    "    \n",
    "    ______\n",
    "    Params\n",
    "    ______\n",
    "    data: panda dataframe correctly parsed from a wahtsapp .txt discussion export.\n",
    "    \n",
    "    year_begin: (int) The first year starting in the plot. Will default to the earliest\n",
    "                year of the discussion if no input given.\n",
    "                \n",
    "    year_end: (int) The last year to plot. Will default to the most recent year of the\n",
    "              discussion if no input given.\n",
    "              \n",
    "    years_cutout: (list(int)) Some discussions span multiple years, and \n",
    "                  it might be hard to correctly visualise only one year of message distribution. \n",
    "                  This list will indicate at which years to split the plots. Default to no cutout.\n",
    "                  If the cutout years were 2012 and 2017, the function will generate a first plot\n",
    "                  displaying the messages distribution from year_begin to 2012, then a plot from \n",
    "                  2012 to 2017, and finally a last plot from 2017 to 2019. If no cutout is provided,\n",
    "                  Only one plot will be produced containing all the years.\n",
    "    \n",
    "    plot_kind: How to plot the message distribution. This parameter will be given directly\n",
    "               as is is to the pandas \"plot\" function's parameter \"kind\". Default to line \n",
    "               (as it is the more easier to see). 'bar', 'scatter' and other value produce \n",
    "               different visual results.\n",
    "    \n",
    "    '''\n",
    "    if year_begin is None:\n",
    "        year_begin = data['Year'].min()\n",
    "    if year_end is None:\n",
    "        year_end = data['Year'].max()\n",
    "        \n",
    "    for y in years_cutout:\n",
    "        if y > year_end or y < year_begin:\n",
    "            raise Exception('The cutout years were not all contained between the starting and ending year.')\n",
    "        \n",
    "    count_by_month_and_year = data.groupby(by=['Year', 'Month']).agg('count').unstack().drop(columns=['msg', 'from', 'Day']).T\n",
    "    count_by_month_and_year.index = count_by_month_and_year.index.droplevel()\n",
    "    year_c_sort = sorted(years_cutout)\n",
    "    for y1, y2 in zip([year_begin] + year_c_sort, year_c_sort + [year_end+1]):\n",
    "        plot_title = 'Message count by month '\n",
    "        plot_title += 'in %d'%(y1) if y1 == y2 else 'from %d to %d'%(y1, y2-1)\n",
    "        #for some reasons, the dataframe gets duplicated on the y axis, so iloc assures we only get 12 data point per year.\n",
    "        count_by_month_and_year.iloc[0:len(months)][[i for i in range(y1, y2)]].plot(title=plot_title,grid=True\\\n",
    "                                                            ,figsize=(15,8), xticks=range(1,13), kind=plot_kind).set_xticklabels(months)\n",
    "\n",
    "def get_most_msg_in_day(data):\n",
    "    '''\n",
    "    Return the day durign which the most message was exchanged. Does not discriminate according to \n",
    "    who send the messages.\n",
    "    '''\n",
    "    res = data.groupby(by=['Year', 'Month', 'Day']).agg('count').idxmax()['date']\n",
    "    print('Day with the most messages sent : %s'%str(res))\n",
    "    #returning a dict with the year, month and day of this special day, could be useful for some people/computations.\n",
    "    return {'year':res[0],'month':res[1],'day':res[2]}\n",
    "    \n",
    "    \n",
    "def plot_msg_in_month(data, year, month, plot_kind='line'):\n",
    "    '''\n",
    "    Plot the distribution of message per day through a month given as argument. \n",
    "    Both the year and the month need to be int. \n",
    "    Can be usefule to have a more fine-grained analysis of the message distribution\n",
    "    than the one provided by plot_msg_in_years\n",
    "    '''\n",
    "    \n",
    "    if year > data['Year'].max() or year < data['Year'].min():\n",
    "        raise Exception('please provide a year argument for which there was some discussion')\n",
    "    if month > 12 or month < 1:\n",
    "        raise Exception('please provide a month argument that is contained between 1 and 12')\n",
    "    title = \"Message number distribution per day on the Month of %s %d\"%(months[month-1], year)\n",
    "    data[(data.Year == year) & (data.Month == month)].groupby('Day').agg('count').plot(kind=plot_kind, figsize=(15,8), legend=False, title=title)\n",
    "    \n",
    "def plot_str_occurence(data, string, year_begin=None, year_end=None, years_cutout=[], plot_kind='line'):\n",
    "    '''\n",
    "    Will plot the occurrence of the string given as first argument through time in the same fashion\n",
    "    as plot_msg_in_years (most of the args of this function are directly passed to plot_msg_in_years btw)\n",
    "    '''\n",
    "    count_str_occurence = lambda msg: msg.lower().count(string.lower())\n",
    "    data['str_occ_count'] = data.msg.apply(count_str_occurence)\n",
    "    print('The following plot is about the distribution of the following string\\'s occurence: %s'%(string))\n",
    "    plot_msg_in_years(data[data.str_occ_count > 0], year_begin, year_end, years_cutout, plot_kind)\n",
    "\n",
    "\n",
    "def plot_most_talking_person(data, plot_kind='bar', order_by_number_of_messages=False, plot_by_year=False):\n",
    "    '''\n",
    "    Will plot the number of messages produce by each member of a discussion. The default plotting style\n",
    "    is a bar chart as it is the best one for this job IMO.\n",
    "    '''\n",
    "    \n",
    "    if plot_by_year:\n",
    "        group_by_person = data.groupby(by=['from', 'Year']).agg('count').filter(['from','msg', 'Year']).unstack(fill_value=0)\n",
    "        group_by_person.columns = group_by_person.columns.droplevel(0)\n",
    "        group_by_person['msg'] = group_by_person.sum(axis=1)\n",
    "    else:  \n",
    "        group_by_person = data.groupby(by=['from']).agg('count').filter(['from','msg'])\n",
    "    \n",
    "    if order_by_number_of_messages: \n",
    "        group_by_person = group_by_person.sort_values(by=\"msg\")\n",
    "        \n",
    "    if plot_by_year:\n",
    "        group_by_person = group_by_person.drop(['msg'], axis=1)\n",
    "    \n",
    "    group_by_person.plot( kind=plot_kind, legend=plot_by_year, title='Number of msg by people in the discussion', figsize=(15,8), stacked=True)\n",
    "    \n",
    "def plot_nb_word_by_person(data, plot_kind='bar', order_by_number_of_messages=False,  plot_by_year=False):\n",
    "    '''\n",
    "    Will plot the number of characters produced by each member of a discussion. The default plotting style\n",
    "    is a bar chart as it is the best one for this job IMO.\n",
    "    '''\n",
    "    \n",
    "    data[\"nb_word\"] = data[\"msg\"].apply(lambda m : len(m.split()))\n",
    "\n",
    "    if plot_by_year:\n",
    "        group_by_person = data.filter(['from','nb_word', 'Year']).groupby(by=['from', 'Year']).agg('sum').unstack(fill_value=0)\n",
    "        group_by_person.columns = group_by_person.columns.droplevel(0)\n",
    "        group_by_person['nb_word'] = group_by_person.sum(axis=1)\n",
    "    else:  \n",
    "        group_by_person = data.filter(['from','nb_word']).groupby(by=['from']).agg('sum')\n",
    "    \n",
    "    if order_by_number_of_messages: \n",
    "        group_by_person = group_by_person.sort_values(by=\"nb_word\")\n",
    "        \n",
    "    if plot_by_year:\n",
    "        group_by_person = group_by_person.drop(['nb_word'], axis=1)\n",
    "    \n",
    "    group_by_person.plot(kind=plot_kind, legend=plot_by_year, title='Number of words by people in the discussion', figsize=(15,8),stacked=True)\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Emoji statistics functions\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#small one line function to count the number of emoji in a text extract.\n",
    "count_emoji = lambda text: len([c for c in text if c in UNICODE_EMOJI])\n",
    "\n",
    "def plot_emoji_per_msg_ratio(data):\n",
    "    '''\n",
    "    Plot the average percentage of emoji contained in the messages per person.\n",
    "    '''\n",
    "    data['emoji_count'] = data.msg.apply(count_emoji)\n",
    "    emoji_and_msg_count = data.drop(columns=['date','Year', 'Month', 'Day']).groupby('from')\\\n",
    "    .agg({'msg':'count', 'emoji_count': 'sum'}).reset_index().rename(columns={'msg':'msg_count'})\n",
    "    emoji_and_msg_count['ratio_emoji_per_msg'] = emoji_and_msg_count.apply(lambda r: r['emoji_count']/r['msg_count'], axis=1)\n",
    "    title = 'Average Ratio of emoji in messages'\n",
    "    emoji_and_msg_count[emoji_and_msg_count['ratio_emoji_per_msg'] > 0].set_index('from').drop(columns=['msg_count', 'emoji_count']).plot(kind='bar', title=title)\n",
    "\n",
    "\n",
    "def plot_msg_with_emoji_ratio(data):\n",
    "    '''\n",
    "    Plot the ratio of message containing emoji\n",
    "    '''\n",
    "    data['emoji_count'] = data.msg.apply(lambda x: 1 if count_emoji(x) > 0 else 0)\n",
    "    emoji_and_msg_count = data.drop(columns=['date','Year', 'Month', 'Day']).groupby('from')\\\n",
    "    .agg({'msg':'count', 'emoji_count': 'sum'}).reset_index().rename(columns={'msg':'msg_count'})\n",
    "    title = 'Ratio of message containing emoji'\n",
    "    emoji_and_msg_count['msg_with_emoji_ratio'] = emoji_and_msg_count.apply(lambda r: r['emoji_count']/r['msg_count'], axis=1)\n",
    "    emoji_and_msg_count[emoji_and_msg_count['msg_with_emoji_ratio'] > 0].set_index('from').drop(columns=['msg_count', 'emoji_count']).plot(kind='bar', title=title)\n",
    "\n",
    "def emoji_and_numb(data):\n",
    "    '''\n",
    "    Return a dataframe that list for each person in the discussion the number of messages sent overall,\n",
    "    And the number of emojis used in absolute.\n",
    "    '''\n",
    "    data['emoji_count'] = data.msg.apply(count_emoji)\n",
    "    return data.drop(columns=['date','Year', 'Month', 'Day']).groupby('from').agg({'msg':'count', 'emoji_count': 'sum'}).reset_index().rename(columns={'msg':'msg_count'})\n",
    "\n",
    "def generate_empty_emoji_dict():\n",
    "    #helper function for the one below. \n",
    "    index = UNICODE_EMOJI.keys()\n",
    "    return dict(zip(index, [0] * len(index)))\n",
    "\n",
    "def top_n_emojis_used(data, n):\n",
    "    '''\n",
    "    Generate the ranking of the n most used emoji of each person in a discussion.\n",
    "    Much less clean and safe than the other functions made, I'm sorry, I was too\n",
    "    lazy to figure out the core functionnalities of pandas to use in order to obt\n",
    "    ain the same result with only 2-3 lines of code, but eh, who cares as long\n",
    "    as it works ?\n",
    "    ______\n",
    "    Params\n",
    "    ______\n",
    "    data : panda dataframe correctly parsed from a wahtsapp .txt discussion export.\n",
    "    n : the size of the ranking of the most emoji used.\n",
    "    '''\n",
    "    users = list(data['from'].unique())\n",
    "    dict_of_emoji_use_pp = dict([(u, generate_empty_emoji_dict()) for u in users])\n",
    "    for index, row in data.iterrows():\n",
    "        usr = row['from']\n",
    "        for c in row['msg']:\n",
    "            if c in UNICODE_EMOJI:\n",
    "                dict_of_emoji_use_pp[usr][c] += 1\n",
    "\n",
    "    newDict = {}            \n",
    "    for ke,va in dict_of_emoji_use_pp.items():\n",
    "        res = list(va.items()).copy()\n",
    "        res.sort(key=operator.itemgetter(1))\n",
    "        newDict[ke] = res[-n:]\n",
    "\n",
    "    for k, v in newDict.items():\n",
    "        if v[n-1][1] > 0:\n",
    "            ranking = list(reversed(v))\n",
    "            print('Top five emoji of %s:'%(k))\n",
    "            for idx in range(n):\n",
    "                print('\\t%d: %s, count = %d'%(idx+1, ranking[idx][0], ranking[idx][1]))\n",
    "            print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your turn\n",
    "\n",
    "Now simply replace the empty string of the righ hand side of the first assignment below with the filename of your discussion .txt export, and then run the cell, and there you go sunshine, the data is loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "discussion_filename = '' #place the filename within the two airquotes and run the cell.\n",
    "my_dataframe = wa_data_proc(discussion_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now you can call any of the function from the previous cells here with your \"my_dataframe\" \n",
    "#as the argument correspoinding to \"data\", try it !\n",
    "\n",
    "plot_msg_in_years(my_dataframe)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}