I've been thinking about it wrong, the spell doesn't remove or nullify my weight, it only counteracts the effects of gravity on my body by pushing back with a similar amount of force. So even though I'm floating like I would in low gravity I still have the same mass. So when I bump into something I still hit it with the same force and mass.