As per spell combat, your off-hand is entirely occupied by the spell, just like the two-weapon fighting rules. Plus you'd need to drop it if you wanted to do it more than once without wasting move actions, since sheathing a weapon is still a move action even with Quick Draw. That said, gloves of storing (http://www.d20pfsrd.com/magic-items/wondrous-items/wondrous-items/e-g/glove-of-storing) would certainly be a boon in this case.