{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import trim_mean\n",
    "from statsmodels import robust\n",
    "import wquantiles\n",
    "import statistics \n",
    "import seaborn as sns\n",
    "import matplotlib.pylab as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "AIRLINE_STATS_CSV =pd.read_csv(\"data/airline_stats.csv\")\n",
    "KC_TAX_CSV = pd.read_csv('data/kc_tax.csv.gz')\n",
    "LC_LOANS_CSV = pd.read_csv('data/lc_loans.csv')\n",
    "AIRPORT_DELAYS_CSV = pd.read_csv('data/dfw_airline.csv')\n",
    "SP500_DATA_CSV = pd.read_csv('data/sp500_data.csv.gz')\n",
    "SP500_SECTORS_CSV = pd.read_csv('data/sp500_sectors.csv')\n",
    "STATE_CSV = pd.read_csv('data/state.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most basic estimate of location is the mean, or average value. The mean is the\n",
    "sum of all the values divided by the number of values. Consider the following set\n",
    "of numbers: {3 5 1 2}. The mean is (3 + 5 + 1 + 2) / 4 = 11 / 4 = 2.75.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trimmed Mean \n",
    "A variation of the mean is a trimmed mean, which you calculate by dropping a\n",
    "fixed number of sorted values at each end and then taking an average of the\n",
    "remaining values.\n",
    "\n",
    "<li>A trimmed mean eliminates the influence of extreme values.\n",
    "<li>Trimmed means are widely used, and in many cases, are\n",
    "preferable to use instead of the ordinary mean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weighted Mean\n",
    "\n",
    "Another type of mean is a weighted mean, which you calculate by multiplying\n",
    "each data Xi value by a weight Wi and dividing their sum by the sum of the\n",
    "weights.\n",
    "#### There are two main motivations for using a weighted mean:\n",
    "\n",
    "<li>Some values are intrinsically more variable than others, and highly variable\n",
    "observations are given a lower weight. For example, if we are taking the\n",
    "average from multiple sensors and one of the sensors is less accurate, then\n",
    "we might downweight the data from that sensor.</li>\n",
    "    \n",
    "    \n",
    "<li>The data collected does not equally represent the different groups that we are\n",
    "interested in measuring. For example, because of the way an online\n",
    "experiment was conducted, we may not have a set of data that accurately\n",
    "reflects all groups in the user base. To correct that, we can give a higher\n",
    "weight to the values from the groups that were underrepresented.</li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Median and Robust Estimates\n",
    "\n",
    "The median is the middle number on a sorted list of the data. If there is an even\n",
    "number of data values, the middle value is one that is not actually in the data set,\n",
    "but rather the average of the two values that divide the sorted data into upper and\n",
    "lower halves. Compared to the mean, which uses all observations, the median\n",
    "depends only on the values in the center of the sorted data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers\n",
    "The median is referred to as a robust estimate of location since it is not influenced\n",
    "by outliers (extreme cases) that could skew the results. An outlier is any value\n",
    "that is very distant from the other values in a data set.\n",
    "\n",
    "The median is not the only robust estimate of location. In fact, a trimmed mean is\n",
    "widely used to avoid the influence of outliers. For example, trimming the bottom\n",
    "and top 10% (a common choice) of the data will provide protection against\n",
    "outliers in all but the smallest data sets. The trimmed mean can be thought of as a\n",
    "compromise between the median and the mean: it is robust to extreme values in\n",
    "the data, but uses more data to calculate the estimate for location."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Location Estimates of Population and Murder Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "state_csv=STATE_CSV.rename(columns={\"Murder.Rate\":\"Murder_Rate\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         State  Population  Murder.Rate Abbreviation\n",
      "0      Alabama     4779736          5.7           AL\n",
      "1       Alaska      710231          5.6           AK\n",
      "2      Arizona     6392017          4.7           AZ\n",
      "3     Arkansas     2915918          5.6           AR\n",
      "4   California    37253956          4.4           CA\n",
      "5     Colorado     5029196          2.8           CO\n",
      "6  Connecticut     3574097          2.4           CT\n",
      "7     Delaware      897934          5.8           DE\n"
     ]
    }
   ],
   "source": [
    "print(STATE_CSV.head(8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6162876.3"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mean \n",
    "state_csv[\"Population\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4783697.125"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trim_mean(state_csv['Population'], 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean is bigger than the trimmed mean, which is bigger than the median.\n",
    "This is because the trimmed mean excludes the largest and smallest five states\n",
    "(trim=0.1 drops 10% from each end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4436369.5"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state_csv['Population'].median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.45"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Weighted Mean \n",
    "def weighted_average_m1(Murder_Rate,Population):\n",
    "    return round(sum([Murder_Rate[i]*Population[i] for i in range(len(Murder_Rate))])/sum(Population),2)\n",
    "\n",
    "weighted_average_m1(state_csv['Murder_Rate'], state_csv['Population'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.066"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Murder_Rate Mean\n",
    "state_csv['Murder_Rate'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Murder_Rate Median\n",
    "state_csv['Murder_Rate'].median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.445833981123393\n"
     ]
    }
   ],
   "source": [
    "# Weighted Mean\n",
    "print(np.average(state_csv['Murder_Rate'], weights=state_csv['Population']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.4\n"
     ]
    }
   ],
   "source": [
    "# Weighted Median\n",
    "print(wquantiles.median(state_csv['Murder_Rate'], weights=state_csv['Population']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.9450000000000003"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Murder_Rate Trimmed Mean \n",
    "trim_mean(state_csv['Murder_Rate'], 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimates of Variability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the heart of statistics lies variability: measuring\n",
    "it, reducing it, distinguishing random from real variability, identifying the various\n",
    "sources of real variability, and making decisions in the presence of it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best-known estimates for variability are the variance and the standard\n",
    "deviation, which are based on squared deviations. The variance is an average of\n",
    "the squared deviations, and the standard deviation is the square root of the\n",
    "variance.\n",
    "\n",
    "Neither the variance, the standard deviation, nor the mean absolute deviation is\n",
    "robust to outliers and extreme values\n",
    "\n",
    "### Variance\n",
    "In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its mean. In other words, it measures how far a set of numbers is spread out from their average value. \n",
    "\n",
    "### Standard Deviation\n",
    "\n",
    "In statistics, the standard deviation is a measure of the amount of variation or dispersion of a set of values.A low standard deviation indicates that the values tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the values are spread out over a wider range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>Population</th>\n",
       "      <th>Murder_Rate</th>\n",
       "      <th>Abbreviation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>4779736</td>\n",
       "      <td>5.7</td>\n",
       "      <td>AL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alaska</td>\n",
       "      <td>710231</td>\n",
       "      <td>5.6</td>\n",
       "      <td>AK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Arizona</td>\n",
       "      <td>6392017</td>\n",
       "      <td>4.7</td>\n",
       "      <td>AZ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Arkansas</td>\n",
       "      <td>2915918</td>\n",
       "      <td>5.6</td>\n",
       "      <td>AR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>California</td>\n",
       "      <td>37253956</td>\n",
       "      <td>4.4</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Colorado</td>\n",
       "      <td>5029196</td>\n",
       "      <td>2.8</td>\n",
       "      <td>CO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Connecticut</td>\n",
       "      <td>3574097</td>\n",
       "      <td>2.4</td>\n",
       "      <td>CT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Delaware</td>\n",
       "      <td>897934</td>\n",
       "      <td>5.8</td>\n",
       "      <td>DE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         State  Population  Murder_Rate Abbreviation\n",
       "0      Alabama     4779736          5.7           AL\n",
       "1       Alaska      710231          5.6           AK\n",
       "2      Arizona     6392017          4.7           AZ\n",
       "3     Arkansas     2915918          5.6           AR\n",
       "4   California    37253956          4.4           CA\n",
       "5     Colorado     5029196          2.8           CO\n",
       "6  Connecticut     3574097          2.4           CT\n",
       "7     Delaware      897934          5.8           DE"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state_csv.head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6848235.347401142"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state_csv['Population'].std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46898327373394.45"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "statistics.variance(state_csv['Population'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 563626.  , 1833004.25, 4436369.5 , 6680312.25])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Quantiles\n",
    "np.percentile(state_csv['Population'],np.arange(0,100,25))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  563626. ,   889558.6,  1353913. ,  2508139.4,  3014731.8,\n",
       "        4436369.5,  5457149.4,  6419552.5,  8940611.8, 12715204.3])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Percentile\n",
    "np.percentile(state_csv['Population'],np.arange(0,100,10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4796417.0\n"
     ]
    }
   ],
   "source": [
    "#IQR (InterQuartrile Range)\n",
    "  \n",
    "from scipy import stats \n",
    "  \n",
    "IQR = stats.iqr(state_csv['Population'], interpolation = 'midpoint') \n",
    "  \n",
    "print(IQR) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4847308.0\n"
     ]
    }
   ],
   "source": [
    "# Interquartile range is calculated as the difference of the 75% and 25% quantile.\n",
    "print(state_csv['Population'].quantile(0.75) - state_csv['Population'].quantile(0.25))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3849876.1459979336\n",
      "3849876.1459979336\n"
     ]
    }
   ],
   "source": [
    "# Median absolute deviation from the median can be calculated with a method in statsmodels\n",
    "print(robust.scale.mad(state_csv['Population']))\n",
    "print(abs(state_csv['Population'] - state_csv['Population'].median()).median() / 0.6744897501960817)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
