An In-Class Demonstration of Bayesian Inference

Sir Ronald Fisher’s venerable experiment “The Lady Tasting Tea” is revisited from a Bayesian perspective. We demonstrate how a similar tasting experiment, conducted in a classroom setting, can familiarize students with several key concepts of Bayesian inference, such as the prior distribution, the posterior distribution, the Bayes factor, and sequential analysis.

Decades later, Dennis Lindley (1993) used an experimental procedure similar to that of Fisher to highlight some limitations of the p-value paradigm. Specifically, the calculation of the p-value depends on the sampling plan, that is, the intention with which the data were collected. Consider the Lindley setup: the lady is offered six pairs of cups, where each pair consists of a cup where the tea was poured first, and a cup where the milk was poured first. She is then asked to judge, for each pair, which cup has had the tea added first. A possible outcome is the sequence RRRRRW, indicating that she was right for the first five pairs, and wrong for the last pair. However, as Lindley demonstrated, the original sampling plan is crucial in calculating the p-value. Was the goal to have the lady taste six pairs of cups -no more, no less -or did she need to continue until she made her first mistake? The observed data are compatible with either sampling plan; yet in the former case, the p-value equals 0.109, whereas in the latter case the p-value equals 0.031. The difference lies in the inclusion of more extreme cases. In the ''test six cups'' plan, the only more extreme outcome is RRRRRR (i.e., the binomial sampling distribution), whereas for the ''test until error'' plan the more extreme outcomes include sequences such as RRRRRRW and RRRRRRRW (i.e., the negative binomial sampling distribution). It seems undesirable that the p-value depends on hypothetical outcomes that are in turn determined by the sampling plan. Harold Jeffreys summarized: ''What the use of p implies, therefore, is that a hypothesis that may be true may be rejected because it has not predicted observable results that have not occurred. This seems a remarkable procedure'' (Jeffreys, 1961, p. 385; see also Berger & Wolpert, 1988).
In this article we revisit Fisher's experimental paradigm to demonstrate several key concepts of Bayesian inference, specifically the prior distribution, the posterior distribution, the Bayes factor, and sequential analysis. Furthermore, we highlight the advantages of Bayesian inference, such as its straightforward interpretation, the ability to monitor the result in realtime, and the irrelevance of the sampling plan. For concreteness, we analyze the outcome of a tasting experiment that featured 57 staff members and students of the Psychology Department at the University of Amsterdam; these participants were asked to distinguish between the alcoholic and the non-alcoholic version of the Weihenstephaner Hefeweissbier, a German wheat beer. We describe how classroom tasting experiments can acquaint students with Bayesian inference, noting that beer can be substituted with anything else suitable (e.g., red and green M&M's, Coca Cola and Pepsi, decaf and regular coffee). We analyze and present the results in the open-source statistical software JASP (JASP Team, 2019).

The Tasting Experiment
On a Friday afternoon, May 12th 2017, an informal beer tasting experiment took place at the Psychology Department of the University of Amsterdam. The experimental team consisted of three members: one to introduce the participants to the experiment and administer the test, one to pour the drinks, and one to process the data. Participants tasted two small cups filled with Weihenstephaner Hefeweissbier, one with alcohol and one without, and indicated which one contained alcohol. Participants were also asked to rate the confidence in their answer (measured on a scale from 1 to 100, with 1 being completely clueless and 100 being absolutely sure), and to rate the two beers in tastiness (measured on a scale from 1 to 100, with 1 being the worst beer ever and 100 being the best beer ever). The experiment was double-blind, such that the person administering the test and interacting with the participants did not know which of the two cups contained alcohol. For ease of reference, each cup van Doorn et al.
was labeled with a random integer between 1 and 500, and each integer corresponded either to the alcoholic or non-alcoholic beer. A coin was flipped to decide which beer was tasted first. The setup was piloted with nine participants; subsequently, we tested as many people as possible within an hour, and also recorded which of the two beers was tasted first. On average, testing took approximately 30 seconds per participant, yielding a total of 57 participants. Of the 57 participants, 42 (73.7%) correctly identified the beer that contained alcohol; in other words, there were s ¼ 42 successes and f ¼ 15 failures. 1

Theoretical Analysis
In order to assess statistically whether and to what extent participants were able to discriminate between alcoholic and non-alcoholic beer we apply the binomial model, where the rate parameter governs the probability of a correct response for each of the participants. Chance performance corresponds to ¼ 1=2. Above-chance performance corresponds to values of higher than 1=2, with ¼ 1 indicating perfect performance.
In the Bayesian framework, we start by specifying a prior distribution. The prior distribution quantifies our beliefs about the parameter of interest before seeing the data. For convenience, we may specify a beta distribution: a probability distribution on the domain ½0, 1 governed by two shape parameters, a and b. Setting a ¼ b ¼ 1 yields a uniform distribution, and implies that all values of rate are equally likely a priori. Setting a > b assigns more prior probability mass to values of higher than 1=2, whereas setting a < b assigns more mass to values of lower than 1=2. 2 The beta prior distribution is then updated to a posterior distribution using Bayes' rule, such that values of that predicted the data well receive a boost in credibility, whereas values of that predicted the data poorly suffer a decline (Rouder & Morey 2017;Wagenmakers et al., 2016): The right-most term is the predictive updating factor that quantifies the change from prior to posterior beliefs brought about by the data. This predictive updating factor indicates how well each value of predicted the data, relative to the average prediction across all values of . When a specific value of predicted the data better than average, the posterior density at that point will be higher than the prior density.
We used the binomial likelihood to assess the quality of each value's prediction (i.e., the likelihood of observing s successes and f failures, given a specific value of ). Because we used the binomial likelihood and a beta prior distribution, the updated posterior distribution will also be a beta distribution -a property known as conjugacy (Gelman et al., 2003).
The obtained posterior distribution can be used for both parameter estimation and hypothesis testing. For parameter estimation, either a point estimate or an interval estimate can be obtained. Commonly used point estimates include the posterior median and posterior mean. Interval estimation can be done with a so-called credible interval, which is an interval that contains x% of the posterior mass 3 and can be interpreted as follows: there is an x% probability that the true parameter lies in this interval. For example, if we obtain a 95% credible interval of ½0:6, 0:9 for , we can be 95% sure that the true value of lies between 0.6 and 0.9.
The posterior distribution can also be used for hypothesis testing, where the traditional goal is to examine specific values of . For instance, we can test the hypothesis H 0 : ¼ 1=2 (i.e., chance performance) by comparing its predictive adequacy to that of an alternative hypothesis H 1 : 6 ¼ 1=2. In other words, H 0 represents the idealized position of a skeptic who believes that the data can be accounted for purely by chance. This ''chance only'' model is pitted against an alternative that allows to take on values different from 1=2.
As before, hypotheses that predict the data well receive a boost in credibility, whereas hypotheses that predict the data poorly suffer a decline. In the Bayesian framework, hypothesis testing is traditionally achieved through the Bayes factor (Etz & Wagenmakers, 2017;Kass & Raftery, 1995). 4 The Bayes factor can be seen as a weighing of one hypothesis' predictive quality relative to that of another. The following equation illustrates this principle, and is very similar to equation (1): It is important to note here that the Bayes factor is a relative metric of the hypotheses' predictive quality. For instance, if the Bayes factor equals 5, this means that the data are 5 times as likely under H 1 than under H 0 . The relative nature of the Bayes factor stands in stark contrast with the frequentist paradigm, where only the null hypothesis is under consideration.
The computation of the Bayes factor is usually not straightforward; however, when the two hypotheses are nested, a convenient computational shortcut can be used, known as the Savage-Dickey density ratio (Dickey & Lientz, 1970;Wagenmakers et al., 2010). The shortcut entails that the Bayes factor equals the ratio of the prior density and the posterior density at the test value 0 . For instance, in the current study, 0 ¼ 1=2 so we have the following ratio: We stress that the mathematical details are not critical for students' understanding of the Bayesian procedures. The following section shows how the example and the associated graphs suffice to clarify the key Bayesian concepts at an intuitive level.

Bayesian Inference with JASP
When the statistical explanation does not resonate with students, a practical demonstration of the analysis might. This can be done with the statistical software JASP, which offers a van Doorn et al.
graphical user interface for conducting Bayesian (and frequentist) analyses. In order to analyze the collected data, the Bayesian binomial test can be used, which can be found under the menu labeled ''Frequencies''. Several settings are available for the binomial test, allowing students to explore different analysis choices. Figure 1 presents a screenshot of the options panel in JASP. For this analysis, we specify a test value of 1=2 (i.e., chance performance), and a ¼ b ¼ 1 for the prior distribution of under H 1 . Note that in a sensitivity or robustness analysis, other values for a and b may be explored to assess their impact on the posterior distribution.
The null hypothesis postulates that participants performed at chance level, whereas the alternative hypothesis postulates that this is not the case. For instance, in the case of twosided hypothesis testing, the hypotheses are specified as follows However, since we wish to test whether or not participants' discriminating ability exceeds chance, we can specify the alternative hypothesis to allow only values of greater than 1=2 (note the '+' in the subscript): where I indicates truncation of the beta distribution to the interval ½1=2, 1. Figure 2 illustrates the results of the binomial test. The left panel shows the prior and the posterior distribution of for the two-sided alternative hypothesis, along with the median and credible interval of the posterior distribution. The posterior median equals 0.731 and the 95% credible interval ranges from 0.610 to 0.833, indicating a substantial deviation of from 1=2. For each value of , the change from prior distribution to posterior distribution is quantified by predictive adequacy: for those values of that predict the data better than average, the posterior density exceeds the prior density (see equation (1)). The left panel shows inference for the two-sided alternative hypothesis (i.e., H 1 : 6 ¼ 1=2) compared to the null hypothesis (i.e., H 0 : ¼ 1=2). The resulting Bayes factor is 122.65 in favor of the alternative hypothesis, that is, the observed data are about 123 times more likely to occur under H 1 than under H 0 .
The right panel shows inference for the one-sided positive hypothesis (i.e., H þ : ! 1=2) compared to the null hypothesis: the resulting Bayes factor is 225.26 in favor of the alternative hypothesis. Note that the posterior distribution itself has hardly changed: the posterior median still equals 0.731 and the 95% credible interval ranges from 0.610 to 0.833. Because virtually all posterior mass was already to the right of 1=2 in the two-sided case, the posterior distribution was virtually unaffected by changing from H 1 to H þ . However, in the right panel, H þ only predicts values greater than 1=2, which is reflected in the prior distribution: all prior mass is now located in the interval ð1=2, 1Þ, and as a result, the prior mass in the interval ð1=2, 1Þ has doubled. Since the posterior density at the point of testing is the same in both panels, but the prior density is doubled in the right panel, the Bayes factor for the directed hypothesis doubles as well.
The experimental procedure also highlights one of the main strengths of Bayesian inference: real-time monitoring of the incoming data. As the data accumulate, the analysis can be continuously updated to include the latest results. In other words, the results may be updated after every participant, or analyzed all at once, without affecting the resulting inference. To illustrate this, we can use Equation 1 to compute the posterior distribution for the first nine participants of the experiment for which s ¼ 6 and f ¼ 3. Specifying the same beta prior distribution as before, namely a truncated beta distribution with shape parameters a ¼ b ¼ 1, and combining this with the data, yields a truncated beta posterior distribution with shape parameters a ¼ 6 þ 1 ¼ 7 and b ¼ 3 þ 1 ¼ 4. 5 The resulting posterior distribution is presented in the left panel of Figure 3. Now, we can take the remaining 48 participants and conduct the Bayesian binomial test. Because we already have knowledge about the population's rate parameter , namely the results of the first nine participants, we can incorporate this in the analysis through the prior distribution, following Lindley's maxim ''today's posterior is tomorrow's prior'' (Lindley, 1972).
In this case, we can specify a truncated beta prior distribution with a ¼ 7 and b ¼ 4, and update this with the data of the remaining 48 participants using Equation 1. Out of the 48 participants, 36 were correct, and 12 were incorrect. Updating the prior distribution with this data yields a posterior distribution with shape parameters a ¼ 7 þ 36 ¼ 43 and b ¼ 4 þ 12 ¼ 16, which is exactly the same posterior distribution obtained when analyzing the full data set at once. This two-step procedure is illustrated in Figure 3. The left panel shows the prior distribution (i.e., the truncated beta distribution with a ¼ 1, b ¼ 1) and the posterior distribution for the first nine participants. The right panel shows the inference for the remaining 48 participants, while incorporating the knowledge gained from the first nine participants in the prior distribution by specifying a truncated beta distribution with a ¼ 7, b ¼ 4.
The ability to monitor the data in real-time and update the inference accordingly prevents wasteful data collection: if there is sufficient evidence to discredit either hypothesis with 50 observations, why collect another 10? Wasteful testing is a serious issue, and monitoring the evidence is important in fields such as medicine, biology, and industry. The Bayesian framework for planning experiments is discussed in more detail by Rouder (2014), Scho¨nbrodt & Wagenmakers (2018), and Scho¨nbrodt et al. (2017). Figure 4 shows the evolution of the Bayes factor as more data are collected. Initially the evidence is inconclusive, but after 30 participants the evidence increasingly supports H 1 .

Concluding Remarks
This article has outlined a teaching tool for familiarizing students with the basics of Bayesian inference. The educational advantage of the Bayesian binomial test is that both the Figure 3. Sequential updating of the Bayesian binomial test. The left panel shows results from a onesided Bayesian binomial test for the first n ¼ 9 participants (s ¼ 6, f ¼ 3). The shape parameters of the truncated beta prior were set to a ¼ 1 and b ¼ 1. The right panel shows results from a one-sided binomial test for the remaining 48 participants. Here, the specified prior is the posterior distribution from the left panel: a truncated beta distribution with a þ s ¼ 7 and b þ f ¼ 4. The resulting posterior distribution is identical to the posterior distribution in Figure 2(b). In order to obtain the total Bayes factor in Figure 2 likelihood function and the parameterization of the prior and posterior distributions are intuitive and straightforward. The tasting experiment allows students to analyze their own data, collected on the fly, making the inferential process more concrete and relevant. Table 1 summarizes the concepts that are introduced during the tasting experiment, as well as how these concepts can be practically demonstrated. The experiment is aimed at introducing college-level students to these concepts. We have positive experiences using it as a teaching tool in both introductory workshops and undergraduate courses in Bayesian inference. Interpreting posterior plots (e.g., Figure 2) 5. Analysis of sensitivity of results to choice of prior distribution Changing the parameters of the beta prior distribution and observing the corresponding changes in the posterior distribution and the Bayes factor 6. Bayesian one-sided testing Specifying different alternative hypotheses 7. Principle of parsimony in Bayesian inference Comparing two-sided results with onesided results; comparing H 0 with H 1 Figure 4. Sequential analysis, showing the evolution of the Bayes factor as n, the number of observed participants, increases. After an initial period of inconclusiveness, the Bayes factor strongly favors H 1 . Figure from JASP.