Question: Suppose, a country 'X''s civil airline wants to fly to another country 'Z'. There are three routes. However, the shortest route is 1000km which is through country 'Y'.

Can the country 'Y' block its airspace for civil airlines of 'X' without giving any reason?

If NO, can 'X' go to an international court for this? If YES, what would the penalty for the country 'Y' be?

Answer: 
It depends - if country 'Y' is one of the 134 [parties](https://www.icao.int/secretariat/legal/list%20of%20parties/transit_en.pdf) to the [International Air Services Transit Agreement](https://treaties.un.org/doc/Publication/UNTS/Volume%2084/volume-84-II-252-English.pdf), then the first article of that agreement grants scheduled air flights the privilege to fly across its territory without landing, and to land for non-traffic purposes.

Otherwise the answer is a little more complicated - with regard to scheduled international flights for purposes such as commercial transit of passengers, according to article 6 of the [Convention on International Civil Aviation](https://www.icao.int/publications/Documents/7300_cons.pdf), to which the 193 members of the UN are parties:

> 
> No scheduled international air service may be operated over or into the territory of a contracting State, except with the special permission or other authorization of that State, and in accordance with the terms of such permission or authorization.
> 
> 
> 

If the flight was not of this type, however, article 5 of the convention grants aircraft the right of overflight and to make non-traffic stops without needing to require prior permission - subject to the terms of the convention:

> 
> Each contracting State agrees that all aircraft of the other contracting States, being aircraft not engaged in scheduled international air services shall have the right, subject to the observance of the terms of this Convention, to make flights into or in transit non-stop across its territory and to make stops for non-traffic purposes without the necessity of obtaining prior permission, and subject to the right of the State flown over to require landing. Each contracting State nevertheless reserves the right, for reasons of safety of flight, to require aircraft desiring to proceed over regions which are inaccessible or without adequate air navigation facilities to follow prescribed routes, or to obtain special permission for such flights.
> 
> 
> 

There are a few exemptions - for example Article 7 states that parties have the right to refuse permission to the aircraft of other states to load "passengers, mail and cargo carried for remuneration or hire" if their destination is within the party's territory.

Article 9 of the convention grants contracting states the right to restrict or prohibit flights in the case of emergency so long as the prohibition is applied uniformly "without distinction of nationality". This the provision which was used during, for example, the 9/11 attacks.

So the answer to your question in general terms is yes with regard to scheduled flights, such as commercial airlines, but no in general - if such an infraction took place, the affected state party could bring a dispute to the ICAO Council, which could decide to suspend the infracting state's voting rights in the Council or the ICAO Assembly.
Since this is Politics SE and not Law SE, the answer is a trivial "if they have the means (i.e., guns) yes; otherwise, no".

The court question is best asked over at Law.