Is there a term for this concept (Randomly Random)? Let's say I have what appears to be a fair dice that over 10K fair throws it produces very close to 1/6 occurrences of each face value.  But then one day it starts a string of throws where the 3 face occurs 1/4 of the time for a run of a few thousand throws. And then one day it reverts to the expected 1/6 probability distribution for a few more thousand throws.  And then it starts returning 1/5 distribution for 1's for a few thousand throws before once again returning to the expected 1/6 distribution for a fair dice.
Is there a term for this concept or something similar?

This is certainly a scenario that could arise in many cases. However, this is a broad class of phenomena and I don't believe there is a single term for it (your description of this as "randomly random" is not in use, nor is it very precise or accurate).
Essentially, what you have have described empirically could be a number of studied statistical models

*

*Broadly, there is a time series $X_t$ of random variables, each of which has some number of samples drawn independently, and whose parameters may (or may not) change from $t$ to $t+1$ in some unknown way. In your case each $X$ is a multinomial distribution (i.e. a single dice row) whose probabilities $p_k^t$ for $k = 1...6$ are not necessarily identical for each $t$; each $t$ could correspond to a single dice throw (once per second or minute), so that each $X_t$ is sampled from only once; or each $t$ could represent a longer length of time (like a day) with many independently drawn samples. The precise relationship between $p_k^t$ and $p_k^{t+1}$ depends on lot on extra unspecified details of the model. Perhaps there is no relationship, and each $p_k^t$ is sampled independently from some sort of Dirichlet distribution; perhaps this is a Markov model, where $p_k^t$ at some time t has a certain probability of transforming into some new dice roll probabilities $p_k^{t+1}$; or perhaps this this is some sort of regression model where the time $t$ is a covariate and the probabilities $p_k^t$ are some parametrized function of $t$.

*Alternatively, there could be a single random variable $X$, representing the probabilities for a dice roll independent of all previous rolls, but your samples (dice rolls) are not all independent, but instead correlated in time for some unknown reason (i.e. rolling a 4 once makes it slightly more likely for a 4 to be rolled the next time). This models also bears some resemblance to a special case of a Markov model (e.g. there is some latent variable representing the dice rolls current bias, which can shift over time as different rolls occur, but the process is stationary and in the long term the dice roll rates tend to some fixed probabilities).

