{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beta distributions for CTR regularization\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'plotnine'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpandas\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mpd\u001b[39;00m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mplotnine\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mp9\u001b[39;00m\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mstatistics\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mstats\u001b[39;00m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'plotnine'"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import plotnine as p9\n",
    "import statistics as stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some example data\n",
    "\n",
    "This data has impressions and downloads, which we will convert to `ctr`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read-in the downloaded data\n",
    "df = pd.read_csv(\"click_log.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new column for CTR\n",
    "df['ctr'] = df['clicks'] / df['impressions']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the average CTR as a variable\n",
    "mean_ctr = stats.mean(df['ctr'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the variance CTR\n",
    "var_ctr = stats.variance(df['ctr'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate sum(alpha, beta)\n",
    "alpha_beta = mean_ctr * (1 - mean_ctr)  / var_ctr - 1\n",
    "alpha_beta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate alpha\n",
    "alpha = mean_ctr * alpha_beta\n",
    "alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate beta\n",
    "beta = (1-mean_ctr) * alpha_beta\n",
    "beta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calulate expected CTR\n",
    "expected_ctr = alpha / (alpha + beta)\n",
    "expected_ctr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Focus on the query 'duck'\n",
    "\n",
    "To show how this can adjust our CTR metric let's look at for just a single query and handful of documents. You should notice that the adjusted values are all closer to the global average that the were as raw CTR values. The magnitude of this shift is based on the number of observations\n",
    "\n",
    "The concept is shrinking individual resource CTRs towards to the global average CTR. The less data we have on a given resource the more we trust the global average. The more data we have for a resource the more we trust it.\n",
    "\n",
    "This process gives a robust estimates when we only have a few observations, which is often the case for search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ducks = df.loc[df['term'] == 'duck']\n",
    "ducks = ducks.head(100)\n",
    "ducks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# caculate adjusted CTR\n",
    "ducks['adjusted_ctr'] = (alpha + ducks['clicks']) / (alpha + beta + ducks['impressions'])\n",
    "ducks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize it to see the shift\n",
    "\n",
    "ducks_long = pd.melt(ducks[['resource_id', 'ctr', 'adjusted_ctr', 'impressions']], id_vars=['resource_id', 'impressions'], value_vars=['ctr', 'adjusted_ctr'])\n",
    "\n",
    "{\n",
    "    p9.ggplot(ducks_long, p9.aes('value', 'resource_id', color = 'variable', size = 'impressions')) +\n",
    "    p9.geom_vline(xintercept = expected_ctr, linetype='dashed') +\n",
    "    p9.geom_point(alpha = .5) +\n",
    "    p9.scale_x_log10()\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
