Is it safe to remove ~/Library/Mail folders I just traded in Mail.app for Sparrow and want to clear up some space on my hard drive. Is it safe to remove the ~/Library/Mail/V2/IMAP-[mailbox] folders for any mail accounts that I have removed from Mail.app?

EDIT: In response to the two "if by safe I assume" answers. I meant both "won't break Mail if I open it again and re-enable those accounts." and "won't delete the messages on the IMAP server"
By safe, I assume you mean "will not delete message on the server?" That's correct, you can go ahead and delete those - think of the as a "cache" of your emails. As long as you aren't deleting and emptying the trash from within whatever mail client you are using, they will remain on the server.

Do consider a backup before deleting ANYTHING from ~/Library (or any other Libraries on your Mac).

In this specific case, I would explicitly rebuild the mailbox indexes just in case the system will clean them up once you're done deleting things...