Question: What does the notation $[G : H]$ mean in group theory? Does it mean the same as quotient group? I have seen the above notation a lot while studying group theory but it still remains unclear to me about the meaning of this notation?

Answer: 
The index $[G:H]$ is the number of left cosets of $H$ in $G$ and it is the number of right cosets of $H$ in $G$. That those numbers are equal is obvious when $G$ is abelian, but it is true for all groups and this does *not* require $G$ to be finite, so the equality is not fully explained by relying on Lagrange’s theorem for finite groups.

For each left $H$-coset $gH = \{gh : h \in H\}$, inverting its elements gives us
$$\{h^{-1}g^{-1} : h \in H\} = \{hg^{-1} : h \in H\} = Hg^{-1},
$$
which is a right $H$-coset, and sending each $gH$ to $Hg^{-1}$ is a well-defined bijection from the left $H$-cosets to the right $H$-cosets, even if the number of cosets is infinite.

For example, $[\mathbf Z: m\mathbf Z] = m$ for positive integers $m$, $[\mathbf R^\times:\{\pm 1\}]$ is infinite, the positive reals $\mathbf R\_{>0}$ have index $2$ in $\mathbf R^\times$, and the group of real $2\times 2$ matrices with positive determinant ${\rm GL}\_2^+(\mathbf R)$ has index $2$ in the group of real invertible $2\times 2$ matrices ${\rm GL}\_2(\mathbf R)$. That a subgroup of index $2$ is a normal subgroup holds for infinite groups, not just finite groups.

The index, when infinite, should properly be considered as a cardinal number, but typically in a first group theory course you may do very little with subgroup indices when they are infinite, so you might just lump all cases of infinite index together as “$[G:H]=\infty$” in the same way infinite sets $S$ are all lumped together as “$\# S = \infty$” unless there is a good reason to use the more careful notion of cardinal numbers.
The notation is about a subgroup $H$ of $G$. It is the number of cosets of $H$ in $G$, called the *index of $H$ in $G$*. For example, in $G=\Bbb Z\_6$, we have a subgroup $H=\{ [0]\_6, [2]\_6, [4]\_6\}$; then $H$ and $[1]\_6+H$ are the only cosets (exercise), so $[G:H]=2$.