Write what you want to write: Accept what people want to read into it
This is a balancing act, but ultimately we're in La mort de l'auteur territory here.
Ultimately, readers are going to... read... things into what you have written. Some are going to read "overly" literally and miss your figurative intent. Others are going to reach for innuendo in everything that you, in your own head, meant literally. Subtext will develop. Critiques will form where random things are picked apart and you're left wondering when you even wrote the passage in question, because you don't even recall using that specific phrasing and certainly not with any intent for a particular interpretation of it.
Art has two sides. Any art. Any act of communication. There is the side of the creative experience. And there is the side of the interpretive experience. The more you try to strangle your art into a narrow existence of precise communication, the more sterile it becomes. 487
This works in certain circumstances, but for fictional writing I find it to be generally detrimental.
You need to chose what you're okay with, in terms of what an audience interprets as how they experienced what they read, versus your intent in what you wrote. And you need to understand that if you're not okay with a specific interpretation, maybe you need to not write whatever you believe will lead to it: struggle all you want, get as explicit as can be, and all you've done is call attention to whatever it is in question.
Consider, if you read a passage where an author went out of their way to try to highlight that a banana was merely a fruit, merely meant literally—in a work with other figurative uses for other objects, no less—don't you think you'd find it significant that so much time was spent talking about bananas? Wouldn't it seem to, in its own way, be an act of emphasis? Wouldn't you question whether the author's narrative acts in regards to the banana might not have aspects of deception, in terms of trying to claim it is only meant to be a banana?
You can't win this, and you're the one starting it
People are going to read what they want to read, and their basis for that is what you put in front of them to be read. Writing is communication, and communication is always interpretive between the parties involved. Language alone is interpretive: we used a shared basis for meaning, but even with supposed authorities we still develop different interpretations of individual words, much less the nuances of those words in shifting contexts.
If you're using an item in a scene which has widely known and widely used alternate connotations, puerile or not, your only even marginally safe road if you are concerned is to remove it entirely, substitute something else for it.
One question I would have is, why is there even, explicitly, a banana at all? Why not just "a piece of fruit"? I love detail, but ultimately you're the one in control of what you write with detail and what you don't: and you can't escape the fact that the more detail you give, the more questions of whether there is a reason for that detail will occur in your audience. You put in the assumed effort and care to write it, why wouldn't someone think there's significance to the specific choices in what you wrote?
If you can't beat them, join them?
Sometimes the best foils for defusing something are your own characters. Rather than refuse the interpretation of readers, acknowledge that it will happen by letting your own characters engage in it.
The easiest way to control a narrative, inasmuch as one can, is to own it.
Many people make phallic metaphors out of bananas. They make jokes about it. They tease each other about it. Some people find this exceedingly juvenile. Some court it. Some are oblivious to it.
So rather than run away from this, use it. Your intent might not be to have your use of bananas have any lingering other meanings, but outside of literature, when has that ever stopped anyone from doing so?
Why do you expect it to be different with what you write?
How would your characters, if they were real people, actually act in this situation? You can still speak through them, at which point it's easy enough to have someone pick up on the metaphoric aspects and run with it—teasingly or more circumspectly—and someone else react, perhaps by becoming annoyed that they can't even "eat a banana in peace" (which would undoubtedly get its own reaction, leading in turn possibly to something like an exasperated facepalm and a request to just drop it or something similar). 873