Question: Suppose, a country 'X''s civil airline wants to fly to another country 'Z'. There are three routes. However, the shortest route is 1000km which is through country 'Y'.

Can the country 'Y' block its airspace for civil airlines of 'X' without giving any reason?

If NO, can 'X' go to an international court for this? If YES, what would the penalty for the country 'Y' be?

Answer: 
It's generally accepted that national sovereignty extends above a nation's land, and entails a right to exclusivity in one's airspace. Thus, overflights require consent from the overflown country, either to the flight in particular or a general consent through a treaty. The main such treaty is IASTA as mentioned by CDJB, but there have been others. For instance, in negotiating the division of Germany after WWII, the Western allies allowed the Soviet Union's sector to surround Berlin on the condition that each of the Western sector be allowed an air corridor to Berlin. It was these corridors that were used in the Berlin airlift. I believe that there are also provisions in international law for aircraft in distress, but I don't know the details.

Of course, ultimately, a country experiencing unauthorized overflights doesn't have much ability to directly prevent them other than shooting the aircraft down. Shooting down civilian aircraft merely because they are violating your airspace falls into the "Things international law technically allows you to do, but is very bad optics" category. As far as I know, there's been only one case of an airliner being shot down while it was violating recognized airspace: Korean Air Lines Flight 007. This incident was strongly condemned in the West, although part of that was that the intrusion was accidental. Shooting down commercial flights intentionally violating airspace would probably still result in serious reprisals.
Since this is Politics SE and not Law SE, the answer is a trivial "if they have the means (i.e., guns) yes; otherwise, no".

The court question is best asked over at Law.