This phase II trial is studying how well giving pazopanib works in treating patients with recurrent or metastatic invasive breast cancer. Pazopanib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth and by blocking blood flow to the tumor.