  We investigate, firstly, the effects of the Rashba SOC on the band structrue
of the Kane-Mele model. The competition between the Rashba SOC and the
intrinsic SOC can lead to the rich phenomenology. The Rashba SOC can drive the
indirect and direct energy gap to close successively, but maintain the band
touching between the valence band and the conduction band when the Rashba SOC
is large enough to dominant the competition. We find that these touching points
are located at $K$ and $K^{\prime}$ or/and some $2\pi/3$ rotationally symmetric
points around $K$ and $K^{\prime}$ in the Brillouin zone. The indirect and
direct energy gap closings correspond to the topologically trivial and
non-trivial phase transitions respectively. For the small intrinsic SOC, the
topologically non-trivial transition occurs when the ratio of the Rashba SOC to
the intrinsic SOC is equal to the classical result, i.e. $2\sqrt{3}$. For the
large intrinsic SOC, however, we find that the ratio decreases with the
increasing intrinsic SOC. Secondly, using the slave-rotor mean field method we
investigate the influences of the correlation on the Rashba SOC-driven
topologically trivial and non-trivial transition in both the charge condensate
and Mott regions. The topological Mott insulator with gapped or gapless spin
excitations can arise from the interplay of the Rashba SOC and correlations.

Strongly Correlated Electrons