Can any probability be expressed as an expectation? In section 6.1 of the notes Stat 3701 Lecture Notes: Bayesian Inference via Markov Chain Monte Carlo (MCMC) by Charles J. Geyer, the author states

Suppose we have a probability or expectation we want to estimate. Probability is a special case of expectation: if $g$ is a zero-or-one valued function, then
$$
E\{ g(X) \} = \Pr\{ g(X) = 1 \}
$$
and any probability can be written this way. So we just consider expectations.

I would assume that $g$ in this context is the function
$$
g(X) =
\begin{cases}
1 & \ \text{if} \ \ X \in A \\
0 & \ \text{if} \ \ X \notin A
\end{cases}
$$
such that
$$
\Pr\{ g(X) = 1 \} = \Pr\{ X \in A \}
$$
where $A$ is some subset of the range (image) of $X$. Is this true for any random variable $X$?

Yes, you are correct. You can use indicator function to define a random variable, say $Y = I_{X\in A}(x)$ and then $Y$ would follow Bernoulli distributon with the "probability of success" $p = \Pr(Y=1) = \Pr(X \in A)$. For Bernoulli distribution, expected value is equal to the probability of success $E[Y] = 1 \times p + 0 \times (1-p) = p$.
