Sport Wales said it would not consider the organisation's application for future public funding unless new directors were appointed.
It said it was clear the board was not following its own rules for recruitment, board meetings and other key decisions.
One former director said the ultimatum was "short-sighted" and "vindictive".
David Francis, another former Welsh Amateur Boxing Association (WABA) chairman, said Sport Wales, which hands out funding, told him money would stop if he and his colleagues refused to step down.
He said there had been a "relationship breakdown" with a Sport Wales-backed consultant brought in to address governance issues.
The consultant was appointed after auditors found WABA was "not fit for purpose" in 2013, and "partly implemented" recommendations by 2015.
In minutes from the last meeting of Welsh Boxing's governing body, in February, issues such as "missing money" and the administration of "medical cards" were discussed.
Mr Francis said Sport Wales had "worked hard to get rid" of the board for several months.
"We were requested to step down in February as a board and told that the chair and vice chair cannot stand again," he added.
Mr Francis said Sport Wales said it was in relation to the information given to them by the consultant.
He said he had a "difference of opinion" with the consultant, but Sport Wales took the side of the consultant.
He said the 114 clubs the association worked with were "perfectly happy with what we do".
BBC Wales has seen an email from one of WABA's former directors, John Wheatley, describing the decision as "short-sighted" and "vindictive".
"Sport Wales are supposed to give reason for their actions at the divisional meetings, though I doubt very much if they will actually appear at any meeting personally," he told clubs.
Brian Davies, of Sport Wales, said progress had been made until last October, but since then there were "some clear signs of a lack of good governance, processes not being followed, the organisation's own articles of association not being implemented properly".
"Those are alarm bells in terms of an organisation that really is there to serve its membership," he added.
Mr Davies said Sport Wales could no longer have confidence in the people in charge of WABA.
He said it was not critical of the board's dedication, enthusiasm, and commitment, but added: "We need the right people leading an organisation of this nature if we are going to invest public funds in it."
He rejected the criticisms of the decision, saying: "I would understand disappointment, but 'vindictive', no."
Mr Davies said Sport Wales wanted boxing to improve and had to take steps.
He said the board had held meetings outside its remit and had not followed its own rules.
He added: "They signed up to a code of good governance and we don't think they are implementing that governance".
The Welsh Amateur Boxing Association board has been forced to resign over a "lack of confidence in governance".