Is it possible for the zero conditional mean assumption to fail? I have a questions about the so-called "zero conditional mean" assumption often made in the context of regression analysis. I am struggling to see how it could be violated, or rather where it is violated.
$ E[\hat \beta] = E[\left(\mathbf X' \mathbf X\right)^{-1}\mathbf X' \mathbf Y] = \beta + E[\left(\mathbf X' \mathbf X\right)^{-1}\mathbf X' \mathbf \varepsilon] $
Above, I give the expectation of the OLS estimate in matrix form. My problem is that I do not see how it is possible for $\ E[X' \varepsilon]$ to not equal zero. Now before you all give me examples from the literature on endogeneity of when there is this bias, the reason I have trouble with this is because whenever we formulate our population regression function as a conditional expectation, by definition the $E[\ X' \varepsilon] =0 $. In other words, in  any population regression function defined as a conditional expectation, we have that the errors are uncorrelated with our regressors. This is because their correlation is defined to be zero. To put it formally (and by the Law of Iterated Expectations):
$ \ E[Y|X]=B_{0}+B_{1}X_{i} \Rightarrow Y_{i}=E[Y|X]+\varepsilon _{i}\Rightarrow E[Y|X_{i}]=E[Y|X]+E[\varepsilon|X ]\Rightarrow E[\varepsilon_{i}|X]=0\ $
But this raises the question: what is the correlation term in the OLS estimates referring to, and how can it be that the assumption does not hold?
EDIT: Some of the comments have suggested that I am confusing errors with residuals. However, this is not obviously the case. My point is as follows. When running a regression, we use OLS to tell us about: $ \ E[Y|X]$. However, for any way in which I specify $ \ E[Y|X]$ it seems as though I am mathematically committing myself to $\ E[\varepsilon_{i}|X]=0\ $. Could someone perhaps specify a population regression function (as a conditional expectation) where this is not the case?
For example, if we consider $\ E[Y|X] = B_{0}+B_{1}X_{i}+B_{2}X_{i} $ and we suppose that $ E[\varepsilon_{i}|X] = g(x) $. Then, by iterated expectations we get that:
$\ Y_{i}=B_{0}+B_{1}X_{i}+B_{2}X_{i}+\varepsilon_{i}  $
$\ E[Y|X]=E[B_{0}+B_{1}X_{i}+B_{2}X_{i}|X] + E[\varepsilon|X] $
$\ E[Y|X]=E[Y|X] + E[\varepsilon|X] $
$\ E[Y|X]=E[Y|X] +g(x) $
$\ g(x)=0$
If $\ g(x) $ does not equal zero, then we have a mathematical contradiction, and therefore I have no idea how in the OLS estimates, we can have anything other than $ \  E[X' \varepsilon=0]\ $.
EDIT 2:
It seems I have found a potential solution to my problem:
CEF: $\ E[Y|X]=B_{0}+B_{1}X_{i}+B_{2}X_{i} $
Regression $\ Y_{i}= B_{0}+B_{1}X_{i}+B_{2}X_{i} + \varepsilon_{i} $
The idea is that $\ E[X|\varepsilon]=0 $ must hold for the CEF, but it may not hold for the regression function. Is this on the right track?

The zero conditional mean assumption for the error term is equivalent to assuming that we have specified the correct form for the regression function in our model.  The reason you are unable to see a situation where this would not hold is that your entire analysis is predicated on correct specification of the initial model.  In particular, your moment results for the OLS estimator are predicated on the initial model form in the regression being correct.

In order to show when the zero conditional mean assumption does not hold, let's generalise by noting that we specify some assumed form for the regression model, but we might get this wrong ---i.e., we will look at model misspecification error.  Remember that the error terms in the regression model are deviations of the response value from its true expected value under the assumed form for the true regression function.  We take some assumed form for the regression function $u_\boldsymbol{\beta}$ and we assume that there is some "true" parameter value $\boldsymbol{\beta}$ such that $u_\boldsymbol{\beta}(\mathbf{x}_i) = \mathbb{E}(Y_i|\mathbf{x}_i)$ for all $\mathbf{x}_i$.  We then define the error terms as deviations $\varepsilon_i \equiv Y_i - u_\boldsymbol{\beta}(\mathbf{x}_i)$, which gives us the assumed ​model form:
$$Y_i = u_\boldsymbol{\beta}(\mathbf{x}_i) + \varepsilon_i.$$
Now, our stipulated function $u_\boldsymbol{\beta}$ is designed to model the expected response as a function of the regressors.  But of course, there is no guarantee of this; maybe we specified the wrong kind of function (e.g., we used a linear regression function when some kind of nonlinear model is needed).
Now, suppose we let $u_*(\mathbf{x}_i) \equiv \mathbb{E}(Y_i|\mathbf{x}_i)$ denote the true regression function (i.e., the true conditional expectation of the response variable).  Perhaps our model is correctly specified (i.e., there exists a parameter value $\boldsymbol{\beta}$ giving $u_* = u_\boldsymbol{\beta}$) and perhaps it is not.  Irrespective of this, the true conditional expected value of the error terms in the model will be:
$$\begin{align}
\mathbb{E}(\varepsilon_i|\mathbf{x}_i) 
&= \mathbb{E}(Y_i - u_\boldsymbol{\beta}(\mathbf{x}_i)|\mathbf{x}_i) \\[6pt]
&= \mathbb{E}(Y_i|\mathbf{x}_i) - u_\boldsymbol{\beta}(\mathbf{x}_i) \\[6pt]
&= u_*(\mathbf{x}_i) - u_\boldsymbol{\beta}(\mathbf{x}_i). \\[6pt]
\end{align}$$
From this result, we can see that if the form for our regression model is correctly specified (i.e., we have $u_* = u_\boldsymbol{\beta}$ under the true parameter value) then the conditional expected value of the error term is zero.  Contrarily, if the form of our model is incorrectly specified (i.e., we have $u_* \neq u_\boldsymbol{\beta}$ under every allowable parameter value) then the conditional expectation of the error term is not generally going to be zero.  (It will still be zero at specific points where the regression functions cross).

As you can see, the failure of the conditional expectation assumption reflects the fact that our assumed regression function does not excapsulate the true conditional expectation of the response variable.  This problem will show up when you do the diagnostic plots for your regression analysis --- specifically, it shows up in the residual plot when you see that the residuals are not distributed randomly around the zero line.
One final thing to note here is that you are never going to be able to see this problem if you work solely within the assumed model form.  If you look at the model under the assumption that $\mathbb{E}(Y_i|\mathbf{x}_i) = u_\boldsymbol{\beta}(\mathbf{x}_i)$ then you get zero conditional expectation for the error term, since the error term is defined as the deviation of the response variable from the assumed regression function.
