Question: What does the notation $[G : H]$ mean in group theory? Does it mean the same as quotient group? I have seen the above notation a lot while studying group theory but it still remains unclear to me about the meaning of this notation?

Answer: 
The index $[G:H]$ is the number of left cosets of $H$ in $G$ and it is the number of right cosets of $H$ in $G$. That those numbers are equal is obvious when $G$ is abelian, but it is true for all groups and this does *not* require $G$ to be finite, so the equality is not fully explained by relying on Lagrange’s theorem for finite groups.

For each left $H$-coset $gH = \{gh : h \in H\}$, inverting its elements gives us
$$\{h^{-1}g^{-1} : h \in H\} = \{hg^{-1} : h \in H\} = Hg^{-1},
$$
which is a right $H$-coset, and sending each $gH$ to $Hg^{-1}$ is a well-defined bijection from the left $H$-cosets to the right $H$-cosets, even if the number of cosets is infinite.

For example, $[\mathbf Z: m\mathbf Z] = m$ for positive integers $m$, $[\mathbf R^\times:\{\pm 1\}]$ is infinite, the positive reals $\mathbf R\_{>0}$ have index $2$ in $\mathbf R^\times$, and the group of real $2\times 2$ matrices with positive determinant ${\rm GL}\_2^+(\mathbf R)$ has index $2$ in the group of real invertible $2\times 2$ matrices ${\rm GL}\_2(\mathbf R)$. That a subgroup of index $2$ is a normal subgroup holds for infinite groups, not just finite groups.

The index, when infinite, should properly be considered as a cardinal number, but typically in a first group theory course you may do very little with subgroup indices when they are infinite, so you might just lump all cases of infinite index together as “$[G:H]=\infty$” in the same way infinite sets $S$ are all lumped together as “$\# S = \infty$” unless there is a good reason to use the more careful notion of cardinal numbers.
$[G:H]$ is very similar to the usual notation of a division. It kind of stands for the number of how many times $H$ fits into $G.$ It is the number of elements of $\left|G/H\right|$ and
\begin{align\*}
G/H=\{g\cdot H\,|\,g\in G\}&=\{1\cdot H, g\_2\cdot H, g\_3\cdot H, g\_4\cdot H,\ldots,g\_{[G:H]}H\}\\
&=H\,\dot\cup\, g\_2H \dot\cup\,\ldots\dot\cup\,g\_{[G:H]}H.
\end{align\*}
The equations are those of sets. If $aH=bH$ then it counts only as one element. It should be noted that $H$ does not have to be a normal subgroup. Subgroup is a sufficient condition to create the equivalence classes $[gH]=\{g\cdot h\,|\,h \in H\}$ that exhaust the group $G.$

$H \trianglelefteq G$ being a normal subgroup is equivalent to $G/H=\{[gH]\,|\,g\in G\} $ carrying a group structure again, or $H$ being the kernel of a group homomorphism $\varphi : G \rightarrow G'.$ However, it is not necessary for $G/H$ to be a group again. It already makes sense to speak of $G/H$ as a set of equivalences classes, modulo $H$ so to say.