It's difficult if not impossible to say for a fact is some action is prejudiced. Certainly, there are some actions that would be universally viewed as such by a given culture. There are very few people alive today who would say that the US history of slavery wasn't racist. But that wasn't true a few generations ago. It wasn't that long ago that homosexuality was widely regarded as deviant behavior in the US and there is still a significant population that retain that viewpoint. Prejudice and sexism/racism/*ism is, to some extent at least, a judgement, not a fact.
What is your concern with asking the question? Are you worried that if your magic system is viewed as sexist it may impact the appeal of your writing? Are you morally opposed to sexism and wish to avoid it showing up in your work, even inadvertently? (I think, or at least hope, that most of us are morally opposed to being sexist, but that's distinct from being morally opposed to writing sexist characters.) The answer to that question may significantly affect the way you approach the issue.
Writing a book which includes sexist characters or cultures does not necessarily imply anything about the author. I'll give two specific examples, but there are many more out there. First, consider <PERSON> Broken Empire series. The characters and society in that series are profoundly misogynistic. <PERSON> has taken significant criticism for that. See this review. 460
But also see <PERSON>'s response to some of the criticism. You can make up your own mind about the worth of the series. The second example is <PERSON> The Handmaid's Tale, also the inspiration for the Hulu television series. The culture there is also profoundly misogynistic. <PERSON>, of course, intentionally wrote the series that way as a protest against the subjugation of women.
The magic system you described doesn't remotely approach the level of sexism in those two works, of course. But they do suggest two quite different ways to approach sexism and misogyny in literature. And they show that you can use the magic system even if some people will consider it sexist. As others have suggested, discuss the issue and try to get a feel for how other people, particularly female readers, will perceive the system. And if you're still concerned with that perception, incorporate it into the story. Have characters note and react to the unfairness of the way the world works, and the assumptions it leads non-witches to make about witches. Explore the theme and use it to deepen and enrich your story. 805