Probability Theory for Scientists and Engineers
Here&#x27;s my non-standard, nutshell, IMHO advice in using probability theory:<p>(1) Random Variables.  Go outside.  Observe a number.  Then that is the value of a <i>random variable</i>.  To have a random variable, that the number be <i>random</i> in the sense of unpredictable is not needed.  For the phrase and&#x2F;or criterion &quot;truly random&quot;, mostly f&#x27;get about it, but we return to that for the subject of random number generation below.  So, net, your data, all your data, are the values of random variables.<p>(2) Distributions.  Sure, each random variable has a distribution.  And there is the Gaussian, uniform, binomial, exponential, Poisson, etc. distributions.<p>Sometimes in practice can use some assumptions to conclude that a random variable has such a known distribution; this is commonly the case for exercises about flipping coins, rolling dice, shuffling cards.<p>For another example, suppose customers are arriving at your Web site.  Well maybe the number of arrivals since noon have stationary (over time) independent increments -- maybe you can confirm this just intuitively.  Then, presto, bingo, the arrivals are a Poisson process, and the times between arrivals are independent, identically distributed exponential random variables -- see E. Cinlar, <i>Introduction to Stochastic Processes</i>. Further, since might be willing to assume that the arrivals are from many users acting independently, the <i>renewal</i> theorem says that the arrivals will be approximately Poisson, more accurately for more users -- see W. Feller&#x27;s second volume.<p>Sometimes the central limit theorem can be used to justify a Gaussian assumption.<p>Still, net, in practice, mostly we don&#x27;t and can&#x27;t know the distribution.  To have much detail on a distribution of one variable takes a lot of data; the joint distribution on several variables takes much more data; the amount of data needed explodes exponentially with the number of joint variables.  So, net, don&#x27;t expect to know or find the distribution.<p>Often you will be able to estimate mean and variance, etc. but not the whole distribution.  So, usually need to proceed without knowing distributions.  In simple terms:  Distributions -- they exist?  Yup.  We can find them?  Nope!<p>(3) Independence.  Probability theory is, sure, part of math, but, really, the hugely important, unique feature is the concept of independence.<p>One of the main techniques in applied math is divide and conquer.  Well, where you can make an independence assumption lets you so divide.<p>Independence?  A simple criterion for practice is, suppose you are given random variables X and Y.  You are even given their probability distributions (but NOT their <i>joint</i> probability distribution).  Then X and Y are independent if and only if knowing the value of one of them tells you nothing more than you already know about the value of the other one.<p>The hope here is that often in practice you can check this criterion just intuitively from what you know about the real situation.  E.g., does a butterfly flapping its wings in Tokyo tell you more about weather tomorrow in NYC?  My intuitive guess is that this is a case of independence which means that for predicting weather of NYC tomorrow, we can just f&#x27;get about that butterfly.<p>(4) Conditioning.  For random variables X and Y, can have the conditional expectation of Y given X, E[Y|X].  Such conditioning is the main way X tells you about Y.  Then there is a function f(X) = E[Y|X], and f(X) is the best non-linear least squares estimate of Y.  Note that E[E[Y|X]] = E[Y] which means that E[Y|X] is an <i>unbiased</i> estimate of Y.<p>(5) Correlation.  If you don&#x27;t have independence, then likely use the Pearson correlation -- it works like the cosine of an angle.  If random variables X and Y are independent, then their Pearson correlation coefficient is 0 -- proof is an easy exercise just from the basic definition and properties of independence.<p>(6) The Classic Limit Theorems. Pay close attention to the central limit theorem (CLT) and the weak and strong laws of large numbers (LLN).  The CLT is the main reason we get a Gaussian distribution, and the LLN is the main reason we take averages.<p>(7) Random Number Generation.  A sequence of random numbers are to look, for some practical purposes, like a sequence of random variables that are all independent and have uniform distribution on [0,1].  Are they &quot;truly random&quot;?  Maybe not.  But if they are, then they are independent and identically distributed (i.i.d.) on [0,1] -- and that&#x27;s all there is to it, and don&#x27;t have to struggle to say or understand more.