Question: Why is $\Delta = b^2 - 4ac$ so special that it deserves a name of its own ? I saw it only once in quadratic formula and nowhere else. 

Does higher polynomials also have discriminant ? if so then is there a general formula for it in terms of coefficients ?

Answer: 
It was mentioned already that the discriminant of a polynomial is zero if and only if the polynomial has multiple roots (i.e. it has some factor of $(x-a)^n$ for $n>1$). 

The discriminant is defined for higher polynomials (there is a formula on wikipedia in terms of the polynomial and its derivative). Perhaps an easier to explain definition is as follows: given a polynomial $$p(x)=\prod\_{i=1}^d(x-\alpha\_i)$$ (recall from the fundamental theorem of algebra than any polynomial with real coefficients [and leading coefficient 1] can be written in this form, with $\alpha\_i$ complex). Then we say $$\text{disc}(p(x))=\prod\_{i<j}(\alpha\_i-\alpha\_j)^2$$ where the product is taken over all pairs $(i,j)$ with $1\le i<j\le d$. (Simple exercise - verify this definition agrees with the discrimiant in the quadratic formula.)

In abstract algebra (particularly in Galois Theory), we can do a bit more with the discriminant than just check that no roots are the same. Consider a polynomial with rational coefficients, $$p(x)=\sum\_{i=0}^d a\_ix^i=a\_dx^d+\dots+a\_1x+a\_0$$ for some $a\_i\in\mathbb{Q}$. In some cases, we can factor such polynomials into linear factors with coefficients in $\mathbb{Q}$: $x^2-2x+1=(x-1)^2$. In other cases, we cannot: $x^2+1=(x+i)(x-i)$. In this case, the polynomial factors into linear polynomials with coefficients in $\mathbb{C}$. Both $\mathbb{C}$ and $\mathbb{Q}$ are sets called fields - essentially they are sets where you can do multiplication, division (except by $0$), addition, and subtraction. There are many other examples of fields, for instance $\mathbb{R}$. Notice that $\mathbb{Q}\subset\mathbb{R}\subset\mathbb{C}$. We could add a lot of other fields into this chain of inclusions. There are also fields between $\mathbb{Q}$ and $\mathbb{C}$ that do not include $\mathbb{R}$ or have $\mathbb{R}$ as a subset. One of the fundamental questions of Galois theory is: "What is the smallest field over which we can write a polynomial as a product of linear factors?" This smallest field is called the splitting field. It turns out that the discriminant of a polynomial can be used to compute some interesting properties of splitting fields of polynomials. 

As a quick note, the splitting field of $x^2+1$ viewed as a polynomial with rational coefficients is the set $\{a+bi:a,b\in\mathbb{Q}\}\subset\mathbb{C}$. This is usually denoted by $\mathbb{Q}(i)$ (or $\mathbb{Q}[i]$ if we are not emphasizing the fact that it is a field). Also, notice that $\mathbb{R}\not\subset\mathbb{Q}(i)$ and $\mathbb{Q}(i)\not\subset\mathbb{R}$.

I am not sure what your background is, so I will leave it at this for now, but feel free to ask if you want me to expand on more things. Also, I am by no means an expert. Hopefully someone with more experience could talk more about what discriminants are useful for.
Higher polynomials also have discrimants which are much more complicated to calculate.

Discriminant gives you an idea about the roots of the polynomial without having to calculate the roots (and above degree 4, you just cannot algebraically).

You can think it like what is the determinant to a matrix is what discrimant is to a polynomial.

See <https://en.wikipedia.org/wiki/Discriminant>

Edit: I couldn't find the general formula for discrimant of an n'th degree polynomial on the Wikipedia page but it exists and is known to mathematicians; it is simply a sum of some combinations of coefficients. Edit 2: Someone provided it.