Mr Alexander, the Chief Secretary to the Treasury, made the claim in the Independent and later told BBC Radio 4 it was "accurate" - but would not reveal who made the remark.
He told the newspaper it happened during a 2012 meeting on the Budget.
Culture Secretary Sajid Javid denied the claim, saying: "No-one said it."
"If they think someone said it... they should come clean and say who said it. The reality is no-one said it so the only conclusion is that they must be making this up," he told Radio 4's World at One.
Responding on the same programme, Mr Alexander said: "I've said what I've said. It is an accurate reflection of a comment that was made."
Chris Mason, political correspondent
For months, if not years, the two coalition partners, the Conservatives and Liberal Democrats, have talked up the lists they were each keeping of what they saw as the bonkers ideas they had stopped the other lot doing, or their brilliant schemes the other side had thwarted.
David Cameron called it his "little black book".
Today, senior Lib Dem Danny Alexander appears to be trying to cash in one of his.
His reporting of a private conversation is potent because it plays at the prejudice about the Tories they are most scared of - being seen as the party of the rich.
But his suggestion is hugely weakened by having no proof of the conversation and not being willing to name which Conservative said what he accuses them of. It means the Tories can accuse him of making it up, something he insists he isn't.
The 2012 Budget - the coalition's third - contained an announcement on increasing the personal allowance to Â£9,205 in April 2013.
Mr Alexander told the newspaper David Cameron and George Osborne were guilty of "breathtaking hypocrisy" for claiming credit for this increase, which the Lib Dems say they had to force Tory ministers to accept.
"In March 2012 when the economy was still in the very early stages of recovery we wanted to do a very big increase in the personal allowance to put a lot of money back into folk's pockets," he said.
"The Tories' priority at the time was the top rate of tax.
"I remember one meeting with a group of senior Conservatives and one of them - I'm not going to say who - said 'Listen, you take care of the workers and we'll take care of the bosses'.
"That was said across the table in one of the meetings where we were discussing the forward policy of the government. That really spelt out where their priorities lay."
Pushed on the claim on Radio 4, Mr Alexander insisted he would not reveal who had made the remark but said he was using it to make a wider point about the "very different" attitudes between the Lib Dems and the Tories.
When told Downing Street had dismissed the claims, he said: "They can say what they like.
"I remember what was said and I remember the nervous laughter from the other side of the table when it was said."
Mr Javid said it was "always, absolutely" a Conservative plan to raise the income tax threshold - even though David Cameron said during the leadership debates in 2010 that it couldn't be done.
But Mr Alexander said it was "extraordinary" Mr Javid had claimed it was always Tory policy to raise the threshold when they did not put it in their manifesto.
Lib Dem leader Nick Clegg earlier said of the remark: "Of course I believe Danny - I remember him vividly telling me that."

After a goalless first half, Fleetwood went in front when Bobby Grant's superb long-range effort flew beyond Shrimpers goalkeeper Daniel Bentley.
Adam Barrett's header and Jack Payne's 18-yard strike looked to have sealed a Southend comeback, but McLaughlin's close-range header levelled the scores.
The draw moved the Cod Army one point clear of the bottom four in League One.
But any chance of a late play-off push now looks to be over for Phil Brown's Southend, who drop to 11th, nine points behind sixth-placed Walsall.
Danny Alexander has claimed he heard a senior Tory tell a Lib Dem colleague: "You take care of the workers and we'll take care of the bosses."
[NEXT_CONCEPT]
Conor McLaughlin's late header rescued a valuable point for relegation-threatened Fleetwood at Southend.