Question: I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with respect to the input distribution. I have seen these between any two continuous or discrete distributions. Can we do it between continuous and discrete or vice versa?

Answer: 
Yes, the KL divergence between continuous and discrete random variables is well defined. If $P$ and $Q$ are distributions on some space $\mathbb{X}$, then both $P$ and $Q$ have densities $f$, $g$ with respect to $\mu = P+Q$ and
$$
D\_{KL}(P,Q) = \int\_{\mathbb{X}} f \log\frac{f}{g}d\mu.
$$

For example, if $\mathbb{X} = [0,1]$, $P$ is Lebesgue's measure and $Q = \delta\_0$ is a point mass at $0$, then $f(x) = 1-\mathbb{1}\_{x=0}$, $g(x) = \mathbb{1}\_{x=0}$ and $$D\_{KL}(P, Q) = \infty.$$
Not in general. The KL divergence is

$$
D\_{KL}(P \ || \ Q) = \int\_{\mathcal{X}} \log \left(\frac{dP}{dQ}\right)dP
$$

provided that $P$ is absolutely continuous with respect to $Q$ and both $P$ and $Q$ are $\sigma$-finite (i.e. under conditions where $\frac{dP}{dQ}$ is well-defined).

For a 'continuous-to-discrete' KL divergence between measures on some usual space, you have the case where Lebesgue measure is absolutely continuous with respect to counting measure, but counting measure is not $\sigma$-finite.