What's the difference between two correlated variables and two variables with common latent factor? Is it possible to distinguish two variables that are just correlated with each other but are not part of the same phenomenon, from two variables, that are part of the same phenomenon and share a common superior/higher latent factor? I mean a mathematical distinction and not a theoretical definition of a variable.
Let's say I measure the relationship between job quality and salary received, and I can assume that these are variables that measure other (separate) phenomena, different things; lets assume that they are correlated. On the other hand, I have the job quality measured by the employee's manager and job quality measured by self-report of the employee; let's assume that these two variables are also correlated with each other. Sooo... The first two measure separate phenomena, the next two measure one and the same phenomenon which is simply expressed or measured differently. The first two are simply just/only correlated, the next two have an higher factor or they are two variables expressing the same thing.
Do we have a mathematical, statistical or any other rule that can tell us (only by looking at the variables, their numerical values), do the two variables have an common latent factor OR are "just randomly correlated"? What's the mathematical difference between the factor items and correlated variables? PCA, CFA, SEM techniques can tell these things apart, or is it a completely theoretical problem? If these techniques can discriminate differences, what factors/indices should be considered?

I believe there are two things you need to mentally separate:

*

*Actual correlation, and


*Spurious correlation.

If two things are truly correlated, then either: one of them causes the other, or there is a latent factor causing both, or, under some conditions, they both cause another observed factor.
Either way, the saying "no correlation without causation" is definitive. If two things move together, it's because they have some causal relationship.

However, sometimes we see correlations in our data that are not true correlations. If your data is a random sample, this becomes a question of hypothesis testing: is the correlation I'm seeing likely real, or is it possible that it is a sampling artifact?
In this case you can use bootstrap procedures to compute the sampling error distribution of your correlation measure and see if it's significantly different from zero. Then it might be real.
If you don't have randomised data, you're out of luck. There's no statistical way to do hypothesis testing then, and you have to guess.

To bootstrap the sampling distribution of the correlation measure you're using, do the following:
Let's say your observed data is 25 pairs of values.

*

*Draw, with replacement, a random sample of 25 pairs from the observed data. Compute the correlation measure for this random sample. Call the resulting correlation value the first bbootstrap replication of the correlation.


*Do the first step many times to generate many bootstrap replications of the correlation.


*Check if, say, 95 % of the bootstrap replications are greater than 0. Then you can say with 95 % confidence that the correlation is indeed positive.
