Media playback is not supported on this device
Wellingborough Town's Martin Potton is alleged to have made the comments in a game refereed by Mary Harmer.
During a 7-0 defeat by Rushden and Diamonds he was overheard saying Harmer was not fit enough to referee a women's match, let alone a men's game.
It is alleged he made further comments about other women in sporting roles.
He told BBC Sport: "I deny all the charges but was found guilty in my absence as I was denied the opportunity of a personal hearing."
Lindsay England, founder of equality group Just A Ball Game, told the Victoria Derbyshire programme that she was at the United Counties League Premier Division match last April and overheard the comments.
She reported him to the Football Association, and others complained as well.
She did not know he was chairman of the non-league club, adding: "Nobody challenged him and nobody laughed and joined in with what he was saying. I think he actually knew what he was saying and he believed that he still had that right to do that."
Potton was banned for the maximum five games by an FA disciplinary commission and fined £75 after being found guilty of using derogatory and insulting language towards an official. He was also ordered to attend a mandatory education lesson.
Sue Ravenlaw, head of the FA's equality and safeguarding team, told BBC Sport: "Any time that discriminatory abuse occurs in football, it's just frankly unacceptable.
"We've been encouraging an environment of collective responsibility from club level, with stewards, safety officers all the way through to ourselves. Whenever discriminatory abuse arises, of any form, people should be confident to report it."
The case was one of a number of incidents which came to light during a survey carried out by Professor Sue Bridgewater of Liverpool University on behalf of Women in Football.
The survey, which aims to identify whether women are achieving their full potential in the football sector, was released to coincide with International Women's Day and is a follow-up to an initial survey carried out in 2014.
More than 500 women - from coaches, match officials, administrators and the media - who work in the football industry responded to the 2016 questionnaire.
The main findings showed:
Heather Rabbatts, FA board member and chair of the FA Inclusion Advisory Board, said: "We still have a considerable way to go before there is a level playing field for women working across the game.
"Surveys like this are a good indicator of where the industry is on some of these critical issues and incidents of bullying, sexism and sexual harassment, must be dealt with whenever and wherever they arise in the game."

They include two former mayors, Leopoldo Lopez and Daniel Ceballos, who are being held on charges of inciting anti-government protests last year.
President Nicolas Maduro has blamed them for the 43 deaths that resulted.
Both men started hunger strikes this week in jail.
The protesters were also calling for the release of the former Caracas mayor, Antonio Ledezma.
Smaller demonstrations took place in other cities.
On Friday, Venezuela blocked two former Latin American presidents from visiting Mr Lopez and Mr Ceballos.
Leopoldo Lopez has been held at the Ramo Verde military prison outside Caracas since he was arrested in February 2014 on charges of inciting the violence that led to the deaths of 43 people and wounded hundreds of Venezuelans who took to the streets.
Mr Lopez filmed a video in jail this week urging demonstrators to go out "peacefully" to demand an end to political persecution and censorship and for a date to be set for upcoming legislative elections.
President Maduro has said his country's judiciary is independent and that those in jail are criminals.
A football chairman was banned from attending five matches and fined after being found guilty of making sexist comments about a female referee.
[NEXT_CONCEPT]
Thousands of Venezuelans have held a peaceful march through the streets of the capital Caracas, calling for the release of opposition leaders who have been in jail for more than a year.