Ever have an Ex you can't get out of your mind (for whatever reason)?
that depends, as a prefix ex can refer to a variety of relationships. 
True enough, let's say an EX, well not lover, but someone who you causally dated
spouse. This often has a derogatory tinge, especially if it refers to unrequited feelings.
Ex (relationship)
In social relationships, an ex (plural is exes) is someone with whom a person was once associated, in a relationship or marriage. As a prefix, "ex-" can refer to a variety of different relationships; for example, one might refer to a music group's "ex-guitarist", or someone's "ex-friend". 
When used alone, "ex" as a noun is assumed to refer to a former sexual or romantic partner, especially a former