Japan administers the uninhabited Senkaku islands, which China also claims under the name Diaoyu islands.
Russian naval ships were also spotted in the area at the same time, Japanese local media said.
In 2012, relations between Japan and China deteriorated after Japan bought the islands from a private owner.
The islands are important because they are close to key shipping lanes, offer rich fishing grounds and lie near potential oil and gas reserves.
Reacting to the incident, Japan's Chief Cabinet Secretary Yoshihide Suga expressed "serious concern" at a news conference, saying it would escalate tensions.
Prime Minister Shinzo Abe has told his government to work closely with the US and other countries to deal with this, Mr Suga said.
How uninhabited islands soured China-Japan ties
"Around 00:50 am (15:30 GMT Wednesday), a Chinese naval vessel entered our nation's contiguous waters surrounding the Senkaku Islands," the Japanese foreign ministry said in a statement.
Japan's Vice-Foreign Minister Akitaka Saiki summoned China's ambassador Cheng Yonghua to lodge a protest.
Mr Saiki told the ambassador that the Senkakus are an inherent part of Japan's territory, in terms of history and international law, NHK reported.
He demanded the vessel swiftly move away from the area. Local media reported the vessel left the area around 03:10 local time.
Mr Cheng said the Chinese frigate was allowed to sail in those waters, said Kyodo news agency citing an unnamed source.
Three Russian military vessels were also seen in the waters around the disputed islands, according to Jiji Press, entering around 21:50 on Wednesday and leaving around 03:05 on Thursday.

Fifa's ethics committee is looking into the "disloyal payment", made nine years after Platini carried out consultation work for Fifa president Sepp Blatter.
Platini, a Fifa vice-president, and Blatter are serving 90-day bans while corruption claims are investigated.
In September, criminal proceedings against Blatter began in Switzerland.
BBC Radio 5 live sports news correspondent Richard Conway says a written contract did exist for Platini's work for Fifa between 1998 and 2002, according to sources. A total of SFr 1.05m (£710,000) is believed to have been paid to the Frenchman.
It is an additional SFr 2m (£1.35m) that was paid in February 2011 that is under scrutiny - and for which no written contract is said to exist.
Both Platini and Blatter are believed to have told investigators the additional payment was an oral agreement for SFr 0.5m to be paid for each of the four years (January 1999 to June 2002) the Uefa president was employed by Fifa. In a statement this month, Platini said "the remuneration was agreed at the time".
It is understood the men claimed an agreement to delay payment was reached at the time due to Fifa's then financial position.
Under Swiss law, an employee cannot seek to have a contract paid up after five years have elapsed irrespective of whether it was written or oral.
Both men have denied any wrongdoing, have appealed against their bans and believe they will be exonerated.
Uefa, which meets in Nyon on Thursday to discuss the issue, has said it has "full confidence" in its president and "stands fully behind him".
However, one Uefa executive committee member, Allan Hansen, has been quoted by Danish newspaper Ekstra Bladet as saying "we can no longer support him" - if no contract exists.
Meanwhile, Fifa has banned Thai football chief and former executive member Worawi Makudi for 90 days for an ethics breach and indicated he could face further punishment.
Japan has lodged a protest with the Chinese ambassador in Tokyo, after a Chinese ship sailed close to contested islands in the East China Sea.
[NEXT_CONCEPT]
Uefa chief Michel Platini produced no documentary evidence for the 2m Swiss francs (£1.35m) payment made to him by Fifa, the BBC has learned.