Moi Ali, who quit the board amid a row over meetings being held behind closed doors, told Holyrood's public audit committee she felt she was bullied.
MSPs and the police watchdog are looking into transparency at the SPA.
Current board members defended Mr Flanagan, saying there was a "changing atmosphere".
The public audit committee has held a series of sessions on transparency at the authority, with one MSP, Alex Neil, decrying an apparent "secret society" within the board. He previously told Mr Flanagan that "it's not the Kremlin you're running".
In the latest session, witnesses included Derek Penman, HM chief inspector of constabulary in Scotland, Ms Ali, and a group of current board members.
Ms Ali left the board amid a row with Mr Flanagan over whether meetings should be held behind closed doors.
She said Mr Flanagan had told her that expressing disagreement publicly was a resignation matter, and told MSPs that her "removal from the board was straightforward punishment for speaking out".
She said it had been a "really horrendous experience", and said she did not believe it would have happened to her if she was a man.
Asked by Labour MSP Monica Lennon if she felt Mr Flanagan was fit to continue as chairman of the SPA board, she replied: "He's not fit to continue on any public board."
However, board members defended Mr Flanagan and the authority, with Iain Whyte characterising it as "an improving board and an improving organisation".
And George Graham said he didn't recognise the characterisation of the board presented, saying "we are listening and we are ready to adapt our approach".
He said there was a "changing atmosphere" at the SPA since Mr Flanagan took over, saying the board was "much more engaged" with "a clearer sense of purpose".
And he defended having meetings behind closed doors, saying there needed to be "clear space to discuss difficult and complex issues".
He said: "If we got the balance wrong and we need to do more committee work in public then we will do that. But every board needs at least some private space to make decisions and discuss some of the difficult, complex issues policing is going to face."
Mr Neil clashed repeatedly with the sitting board members, describing scrutiny by non-executive directors as "wholly inadequate".
He said he was not left with a lot of confidence that the members were "doing the proper job" and holding the chairman to account.
However, Mr Graham replied: "There are an awful lot of good things that we do. The focus on a singular point, or failure if you want to call it that, I think is a poor characterisation of what we're doing."
SNP member Mr Neil also said he wanted Mr Flanagan to come back to give evidence to the committee again over what he called a "PR disaster".
In the previous session, the chairman said the authority had made "significant steps forward" over the past year.

In 2013/14, York Theatre Royal and Sheffield Theatres reported the number of BME employees as 3%, while Hull Truck Theatre had none.
ACE said that a survey showed that 14% of the English workforce were from BME communities.
Hull Truck Theatre said it was taking action to address the issue.
Last year Sir Peter Bazalgette, chair of Ace, said organisations could have their funding cut if they failed to improve diversity.
Speaking at the launch of the new report, Sir Peter said theatres had to make "their work more reflective of their communities, and representative of 21st Century England".
"The arts can show us how things are and the arts can give us a vision of what they could be," he said. "They can make the case for diversity; they can be the case for diversity. They can bring us together."
Ace said it was providing Â£8.5m of funding to encourage diversity.
The executive director of Hull Truck Theatre, Janthi Mills-Ward, said the theatre had introduced an apprenticeship scheme "which has changed the demographic".
"Hull's demographic is actually 94% white," she said. "So actually, young people, or anybody from Hull from a BME community might not have seen a career in the arts as something they could go in to."
A former Scottish Police Authority board member has claimed its chairman Andrew Flanagan is "not fit to continue on any public board".
[NEXT_CONCEPT]
A number of theatres in Yorkshire employ fewer than 5% black or minority ethnic (BME) workers, according to an Arts Council England (Ace) report.