How do we know which compounds have higher solubility? Is there any way we can look at the chemical formula of various compounds, do some calculations or something and figure out which compounds will have higher solubility as compared to the others.
It depends on the solution and the solute. Usually, the rule is "like" dissolves "like". A polar solution will dissolve polar compounds and a non-polar solution will dissolve non-polar compounds.

https://www.youtube.com/watch?v=PVL24HAesnc

As far as I know, to find the solubility you have to determine it experimentally. This can be done by "feel" first. If you have a compound that doesn't have many polar groups on it, chances are it can be dissolved by a non-polar solution. If you know what the structure looks like try to match it with a solvent that looks similar to it.