Ambrose99,

Unfortunately, the 220 plus-year-old US Constitution doesn't provide for removing a President from Office on grounds of lacking lack of confidence in a President or because of his/her incompetence.  While it rejected hereditary monarchy, it did mirror many aspects of an idealized version of the constitutional UK monarchy of the time, albeit one selected by popular (although with a restricted electorate) electorate for a 4-year term.

The "not fit to govern" provision to which you refer would probably be interpreted by the courts and constitutional scholars narrowly as applying to mental, emotional, or physical incapacitating ailment.  Given the central place of the Separation of Powers doctrine in the interpretation of the US Constitution, accepting that Congress has the capacity to remove a President on the basis of a broader interpretation of 'fitness' would be seen as problematic.