Rigondeaux, who has been training in Russia, will "not be arriving in the UK" before the fight, according to Frank Warren's Queensberry Promotions.
A statement added that the 35-year-old Cuban is currently in Miami "for some unexplained reason".
"We are all extremely disappointed by the situation," it read.
Queensberry Promotions says Rigondeaux applied for his UK visa in Moscow on 29 February and that five working days were required for the process to be completed.
It said it was told by the application centre that the visa would be issued by Wednesday at the latest and Rigondeaux "would then arrive in the UK shortly thereafter".
Queensberry Promotions claims that Rigondeaux boarded a flight to Miami on Tuesday.
It added: "We have yet to establish how and why he flew to the USA and, further, how he managed to fly anywhere when the normal practice is, when applying for a visa, that you hand your passport in and get it back when the visa is issued."
WBA and IBF champion Carl Frampton has been ordered to fight Rigondeaux after beating Scott Quigg in February.
WBA bosses said former champion Rigondeaux - a two-time Olympic champion who is unbeaten in 16 professional bouts - must face the Northern Irishman before 27 July.
Briton Jazza Dickens' super-bantamweight fight against Guillermo Rigondeaux in Liverpool on Saturday has been cancelled because of visa issues.