"No matter what I did, I always felt I was fighting a losing battle," he said.
But Burgess said he had not been badly treated by England and refused to blame coach Stuart Lancaster.
The 26-year-old, who joined Bath last October, re-signed for the South Sydney Rabbitohs last week.
He was part of England's squad for the Rugby World Cup, in which they were eliminated at the group stage, and played in the hosts' crucial 28-25 defeat by Wales.
Burgess said criticism directed at him following his performance in that match had contributed to his feeling of unease in rugby union.
"A lot of people outside the England camp had an agenda against both England and, in some circumstances, me," he wrote in the Daily Mail.
"This was an upsetting factor to me, that people who are supposed to love the game are actually tearing it to shreds.
"Some ex-players just kept letting rip. It's almost like they don't want anyone else to do well in the jersey. I didn't want to stay in a sport like that."
Burgess also said he had not found the 15-man code as enjoyable as rugby league.
"I want to spend the rest of my career playing the game that's in my heart," said Burgess, who played 21 games for Bath as a blind-side flanker after switching codes last year.
"What makes me love [rugby league] is the physical battle of it; I never found myself reaching that point [in union].
"My heart just isn't in [rugby union], and if my heart's not in it then [Bath and England] won't get the best performances from me."
He said he was satisfied with his displays at the World Cup and believed coach Lancaster had been unfairly criticised after the team's early exit.
Burgess, whose selection for the World Cup squad ahead of the established Luther Burrell reportedly caused disquiet in the England camp, said his decision to quit union had dismayed his international and club team-mates.
"I wanted to go in and see [the Bath squad], but Stuart Hooper, our captain, said he didn't think I'd be well received there, which was fair enough," he said.

Fintan Treanor, 20, was walking home in the early hours of Sunday, 26 June 2011, when he was struck by a vehicle at Lemgare, Clontibret.
He had been walking home after a night out in Castleblaney.
Police appeals have centred on an Audi A4 1995 to 2001 model missing its front lower air-vent grille.
"Five years have passed and the driver of the car involved has not been located," a gardai (Irish police) spokesperson said.
"Perhaps now after five years some person knows a person who disposed of an Audi A4 around that time.
"In five years, loyalties, friendships and relationships change and perhaps someone that felt unable to come forward before is now in a position to provide the investigation team with the vital information that would locate the driver."
Former England centre Sam Burgess returned to rugby league because his "heart wasn't in" rugby union and because he was also stung by criticism from ex-players.
[NEXT_CONCEPT]
Police have renewed their appeal for information on the fifth anniversary of a fatal hit-and-run collision in County Monaghan.