Two tries from Mark Atkinson and one from Matt Scott helped put them 21-6 ahead, before Simon Labouyrie and Dion Evarard Oulai crossed for the hosts.
Billy Twelvetrees' try made it 29-20 at half-time, before Evrard Ourai's second reduced the deficit to just two points.
But May's effort and a late try from Tom Savage earned their side the win.
Bayonne: Jane; Laveau, Whitelock, Lacroix, Fuster; Meret, Duhalde; Iguiniz, Labouyrie, Choirat, Ducat, Jaulhac, Latimer, Lespinasse, Oulai.
Replacements: Leiataua, Bordenave, Taufa, Huete, Saubusse, Duputs, Maillard, Mamao.
Gloucester: Hook; Halaifonua, Scott, Atkinson, May; Twelvetrees, Heinz (c); Thomas, Hibbard, Doran-Jones; Savage, Latta, Ludlow, Rowan, Morgan.
Replacements: Matu'u, Orr, Afoa, Galarza, Kvesic, Braley, Thorley, Sharples.
Referee: Gary Conway (Ireland)
Assistant referees: Stuart Gaffikin and Richard Kerr (Both Ireland)
For the latest rugby union news follow @bbcrugbyunion on Twitter.

RBS, which has already closed dozens of branches in Scotland since 2015, said 176 full-time posts would be affected.
It added that the move was in response to the growing popularity of mobile and online banking.
The union Unite reacted angrily, claiming RBS was "turning its back on the communities that have been the foundation of their business for generations".
In addition to the Scottish closures, RBS announced it would shut 128 NatWest branches.
A total of about 770 staff across the UK will be affected but hundreds of workers will be redeployed, the bank said.
RBS branches earmarked for closure:
An RBS spokesman said many more customers now preferred using mobile and online services over traditional branch counters.
He said: "We interact with our customers over 20 times more through digital channels than physical ones.
"Half of our personal customers in Scotland are mobile banking users - grown from 500,000 to 900,000 since 2014.
"As customers change the way they bank with us, we must change the way we serve them.
"The role of the branch is fast moving to a centre for advice, away from basic transactions.
"While the branch will still be a core part of our offering to customers, inevitably some branches will have to close."
Reacting to the news, Unite regional officer Lyn Turner said: "RBS is turning its back on the communities that have been the foundation of their business for generations.
"That's bad news for our members who now have to live with the threat of redundancy - and it's bad news for customers and businesses.
"Banks have a duty to the wider community and that is especially the case for banks like RBS that have large taxpayer-owned shareholdings."
The Federation of Small Businesses (FSB) said it was "frustrated to see RBS show so little loyalty to our high streets".
Scottish policy convener Andy Willox added: "Branch closures put pressure on local economies and make it harder for local firms to access banking services.
"While more people might be doing online banking, that's no good for cash-based businesses or for rural firms with poor broadband.
"As the big banks shut down branches in chorus, it unfortunately seems like the banking industry is content to leave some customers behind."
After the closures 151 RBS branches will remain.
The bank remains majority-owned by taxpayers, following its multi-billion government bailout almost a decade ago.

Officers from North Wales and Dyfed-Powys police forces have tweeted photographs of sheep which have been injured or killed in dog attacks.
A rural crime team has published a montage of distressing pictures.
The caption reads: "This needs no words or music. Just time to reflect on the devastating impact an irresponsible dog owner can have."
England winger Jonny May scored a try on his return after 10 months out with a knee injury to help Gloucester beat Bayonne in their Challenge Cup opener.
[NEXT_CONCEPT]
RBS has announced that it is shutting 30 more Scottish branches.
[NEXT_CONCEPT]
Police are urging dog owners to be responsible following attacks on sheep ahead of the lambing season.