Hanako, which means "Flower Child", was 69 and the oldest elephant in Japan.
The elephant was a gift from Thailand when it was two years old.
Last year a petition was launched to improve the animal's conditions, but Hanako was deemed too old to move back to Thailand.
A spokesman for Inokashira Park Zoo on the outskirts of Tokyo said Hanako was found lying on her side on Thursday morning and efforts to raise her were not successful. She died in the afternoon.
The war on elephants
Have we fallen out of love with elephant rides?
Hanako was a popular attraction in the zoo, but the elephant, born in the wild, spent most of her life caged and alone in a concrete enclosure with little greenery.
Last year a Canadian blogger wrote an article about the elephant's enclosure, describing it as a "concrete prison".
That prompted the petition, which led the zoo to provide new toys and make other changes.
However, the zoo said she was too old to be moved and an independent US wildlife expert said staying in a sanctuary with other elephants would bewilder Hanako after living so many years alone.

South Wales Deputy Police and Crime Commissioner Sophie Howe said Cardiff has reached "saturation point".
She said licences had been granted that the force "haven't necessarily agreed with".
On Friday, the council said it only granted licences in a "saturation zone" when applicants show there will be no negative impact.
After a summit into recent alleged sex attacks in the city, Ms Howe pointed out there are 299  licensed premises in Cardiff's central square mile.
"It is the highest concentration of licensed premises anywhere in the UK," she said.
The council said police and other responsible authorities were notified of every licence application.
Any objections must relate to the authority's licensing objectives.
These include the prevention of crime and disorder, the prevention of public nuisance, public safety and the protection of children from harm.
"If relevant representations are received, the licensing sub-committee must consider the application and the relevant representations made," the council said.
"The policy of the authority in general is that we will not permit any further premises to be licensed for the sale of alcohol in areas designated in a saturation zone unless the applicant is able to demonstrate that there will be no negative impact on one or more of the licensing objectives."
Thursday's summit was attended by figures from the police, the council, the city's universities and businesses.
It followed three sexual assaults alleged to have taken place near Cardiff University Students' Union during freshers' fortnight.

The Care Quality Commission (CQC) rates the Royal Cornwall Hospital as "requires improvement".
Chief Inspector Professor Sir Mike Richards said it was "disappointing" that "not enough progress" had been made its the 2014 report.
Hospital bosses insist changes have been made and care is getting better.
Read more on this story as it develops throughout the day on our Local Live pages.
The other Cornwall NHS hospitals, St Michael's in Hayle and West Cornwall in Penzance, were rated "good".
The CQC's main concerns about the Royal Cornwall Hospital included:
In recent months the trust has been ranked one of the worst in the country for waits in A&E, bed blocking and staff being able to report mistakes.
Agency spend is exceptionally high; the hospital spent Â£1.43m on agency workers in March.
The Royal Cornwall Hospital also has a higher death rate than would be expected so new chief executive Kathy Byrne has some major issues to deal with.
Prof Richards said: "If the trust is going to pull itself round, it will need a combined determination of the staff, the leadership team and the commissioners."
Kathy Byrne, the RCHT's chief executive said: "We don't feel it is a disappointing report.
"I fully accept the CQC's view that in some important areas we have not made enough progress.
"We have acted on all the CQC's "must do" recommendations and are seeing better results for patients across the board."
An elephant that was the focus of an international campaign to improve its living conditions has died in a zoo in Japan.
[NEXT_CONCEPT]
Claims there are "too many" pubs and bars in central Cardiff have been defended by the city's council.
[NEXT_CONCEPT]
Cornwall's main hospital has been told it must improve, two years after it was given the same instruction by inspectors.