How did the Kaguya clan come into existence? How was the Kaguya clan related to Kaguya and when was the clan formed?
The when and how the clan came to be are fairly unclear through out the series. But we can make a guess as to when very roughly.
According to this timeline, Gamamaru was born in 984 BK (before nine tails attack). Which was said to be the same generation as Hagoromo, the first son of Kaguya Ōtsutsuki. Meaning that the creation would be in one of the years to follow
The clan was named after the first known user of the Tomogoroshi no haikotsu. As their power, the Shikotsumayaku was a derivative power of it.
This derivative power was obtained through descendants of Kaguya Ōtsutsuki, indirectly making her the creator of the Kaguya clan.